We use a Kanban board view in Airtable to monitor and manage experiments. Each experiment is an Airtable record, which translates to a card on the board. There are 5 channels for progress stages of an experiment life cycle:
Idea – a new potential expriment submitted via a hypothesis builder
Plan – calculate the experiment’s sample size, run time, and priority in the testing plan
Build – set up the experiment in the split testing platform
QC – review the experiment and approve it to go live
Active – the experiment is live
Complete – the experiment has concluded
Moving a card from one channel to another triggers automation for repeatable tasks like sending status notifications, running a report, or updating project information. We use Basecamp for project management, but this process can be applied to almost any project management platform that has API, like Asana, Jira, or Trello. The automation is set up to use API hooks for performing simple, repeatable, standardized processes like creating projects/tasks, adding notes, and assigning tasks. This significantly cuts down on overhead and eliminates a lot of tedious manual configuration drudgery from the project management process.
Updating Our Process to Reduce Data Bloat
The goal of any good project management process is to minimize overhead. One important component of project management is dealing with data bloat. A lot of project data has a shelf life; it’s relevant and useful for only specific period of time. Data bloat accumulates when tasks aren’t acted upon and data goes stale. As data piles up, it becomes harder and harder for users of the system to find relevant information quickly and actually get work done. This wastes time and causes frustration. Worse, information bloat can lower morale if ignored for too long. To help combat data bloat, we’re making a change to our project creation automation.
Previously, we had the automation set to create a new project for an experiment when it was moved into the Planning stage. A lot of projects were getting created. Projects for lower priority experiment ideas were just sitting open and unused, making it harder to find the active projects. Sometimes an experiment idea would be scrapped or modified before it was ready to build. If a project had already been opened for that experiment then it had to be manually updated, creating unnecessary revision work.
Now the automation creates projects when an experiment idea progresses to the Build stage. Building an experiment is where the bulk of actionable tasks begin, so this change should help increase project data freshness. Projects will be created more closely to their “ready for work” state instead of sitting open and unattended. The tasks that go on during the planning stage have been migrated to a separate, repeatable weekly task.
Startup Cincy Week is an annual, week-long series of presentations, roundtable discussions, networking, and job fairs celebrating of the growing entrepreneurial community here in Cincinnati, Ohio. The event brings together entrepreneurs, students, mentors, investors, and teachers from local StartUps, BigCos, and universities. Topics range from starting a business, securing funding, scaling, creativity, technology, design, and more.
This is the second year that Corvus CRO has participated in Startup Cincy Week, and the first year that we’ve presented! Being event volunteers is becoming an annual tradition for us. Here’s a shot of Matt working the registration desk on Tuesday morning:
New for us this year was organizing a panel. Since our main focus here at Corvus CRO is User Experience, we wanted to contribute some design based content. We brought together some of local UX designer friends and colleagues to participate in a roundtable discussion titled “UX for Start-ups: Leveraging experience design to grow your business”. All the presentations were livestreamed and recorded on Cintrifuse’s Facebook page, so here’s a link to our session. Please check it out! A great big thank you to Beau Heubach and Katie Pohlman from Gaslight and Paul Armstrong from Alchemy for joining us.
We’re already looking forward to Startup Cincy Week 2020!
October A/B Testing Happy Hour
16-Bit Barcade was the place to be again in October. Thanks to all who attended last month’s A/B Testing Happy Hour, and a special thank you to Jessica Hall at Tech Elevator for helping promote the meetup. Several students from the current bootcamp session showed up to discuss front end development and share interesting personal anecdotes about what brought them into tech.
If you’re interested in attending future happy hours or chatting with other fine folks that work in the CRO space, then please check out the Cincinnati Split Testers meetup group. November’s A/B Testing Happy Hour will be here before you know it!
October CodePen Meetup
Bryan King, our Director of Design, is one of the co-organizers for CodePen Cincinnati, a local meetup group focused on discussing and teaching the latest in front-end development. For the October meetup, he gave a fantastic presentation on building perfect responsive layouts using HTML and CSS. He performed a love coding exercise demonstrating the power and versatility of flexbox and CSS grid. As always, thank you to Dot Dash for hosting.
September’s A/B Testing Happy Hour had the best event turnout to date! A great big thank you to everyone that attended; hope to see you in October, and many more in the future. There were some wonderfully interesting conversations around the role of data science in testing, integrating and scaling testing programs within a company, and some good ol’ nuts-and-bolts test build process. Moving locations to 16-Bit Barcade really helped with accessibility. Unfortunately there was an issue with the beer taps, so we all had to make do with cans, bottles, and poured drinks. The bourbon drinkers of the group were totally cool with that.
Hosting at 16-bit gave us a fun theming idea to design video game inspired custom banners for each monthly event. September’s banner was based on the start screen for the original Sonic the Hedgehog on Sega Genesis. Voting is open to choose next month’s theme: go vote on Twitter!
If you’re ever in the Cincinnati area on the 2nd Wednesday of the month, come join us; we’d love to meet you in person. Your fist beer is on us! The October meetup has been scheduled. You can always join the Cincinnati Split Testers meetup group to get updates and event notifications.
Variation Switching for Experiments With Delayed Activation
Grilling With the Team
Once a month, we have an informal company dinner hosted by one of our staff. It’s the hosts responsibility to prepare the meal and entertain at their home. Having dinner at someone’s home invites open, non-work related conversation. Connecting over food and drink is a major way that we strengthen our team bonds.
This month it was my turn to host. Here’s the menu:
citrus marinated pork loin
sun-dried tomato orzo salad
Mistakes were made: I ended up burning the asparagus tips, and we didn’t take any pictures. That’s okay though, mistakes are learning opportunities. Next month will be better, for sure.
A/B Testing Happy Hour
Another one of our monthly endeavors is organizing a local A/B Testing Happy Hour every second Wednesday of the month here in our home base of Cincinnati, OH. Last Wednesday we enjoyed some fine conversion optimization discussion coupled with blood orange raddlers and raspberry-lemon hefeweizen on the newly opened rooftop bar at Braxton Brewing. If you’re ever in the area, then please join us!
We use Airtable for collating experiment data from all of it’s various sources: internal, client, testing software, analytics software, project management software, etc. This experiment data forms the backbone of our testing process; it’s used for things like hypothesis construction, test prioritization, project management, triggering automation, and reporting.
Because “Transparency and Honesty” is one of our core values, we want to make sure that clients can view all of their experiment data on demand, at any time. However, creating additional accounts isn’t a viable solution for us. Because Airtable’s pricing is per user, costs would quickly balloon up as program maturity rises and more people get involved in the testing process. Also, a high level of data integrity is critical to the testing process. More users with editing privileges means more potential for loss of data fidelity. For these two reasons, we’ve opted to not set up user accounts for clients.
Airtable offers a couple different ways of sharing data to non-users. Private links can be enabled per view, allowing anyone with a link access to see that view. Maintaining a list of links would be cumbersome, so there’s also the ability to embed a view via an iframe, which is our current solution for data transparency. We have individual portals set up for each client. Airtable views for that client’s data are placed into tabs. Clients can easily view all of their test program data from their portal.
If data transparency is as important to you as it is to us, then we’d love to talk. Is finding the right information difficult? Struggling to make sense of all your testing data sources? Corvus CRO can help. Schedule a free 30 minute consultation with us today.
Basecamp is a fantastic project mangement platform with an unbeatable price point at only $100 a month. We use it to collaborate with several of our clients. Both versions of Basecamp (2 and 3) are really good at handling team communication; it handles messages, notifications, scheduling, to-do lists, document sharing, and most of the other important features of a project management tool.
That being said, Basecamp has a little bit of trouble looking at the “big picture” if you have lots of active projects. Something that’s easily possible with an optimization program, especially if you set up a separate project for experiment like we do. Our solution for broader program visibility is a Kanban board:
Experiment progress is tracked through these activity stages:
Plan – Experiments prioritized through a P.I.E.E. ranking
Build – Experiments being built in the testing platform
QC – Experiments ready for proofing and final approval
Active – Live experiments
Complete – Finished experiments
Every experiment gets it’s own card on the Kanban board. Progressing a card through the Kanban stacks forms the core of our automation triggers. For example, when an card is moved to Complete, a Google Analytics report automatically runs to grab user counts, transaction counts, and revenue for the matching time period. That GA data is used to perform comparative analysis and experiment lifetime value estimation.
The newest addition to the Corvus CRO automation arsenal is Basecamp project creation and management. Every time a card is moved to the Plan stack, a new Basecamp project is created from a template; the template sets up the necessary to-do lists and discussion channels for experiment collaboration. Progression into other stacks adds additional to-dos, checks off tasks, uploads files, and/or sends notifications as necessary. The automation handles building the project management framework, eliminating a significant chunk of configuration tedium, and freeing up people to work on more interesting things.
Organization is key to any successful conversion optimization program. Trapped in configuration hell? Project management software causing more headaches than progress? Can’t get a handle on your split testing process? Corvus CRO can help. Schedule a free 30 minute consultation with us today.
Automatic Detection of Winning and Losing Variations
If the Primary KPI is set for an experiment record in Airtable and a matching goal is found, the Result Data Scraper tool will now automatically detect the best and worst performing variations. Manual selection will be displayed as a backup if the Primary KPI is not set or a matching goal is not found. This makes the scraper a little bit easier to use, automating another bit of data entry if possible. Properly declaring a Primary KPI during the idea stage of an experiment results in less work in the completion stage. An ounce of prevention is worth a pound of cure.
Bug Fix for High Conversion Counts
Discovered and resolved a bug in the result data scraper tool if both the amount of conversions and visitors are both over 1,000. The regex used for stripping non-numeric characters from the conversion and visitor count strings did not have the global flag applied. For high counts on both conversion and visitors (for example: 3,521/119,347 conversions/visitors) only the first comma was being stripped. This was causing an error when attempting to push the visitor count into Airtable, since only numeric data would be accepted by that field. More robust string parsing and sanitization has been put in place to resolve this issue and help future-proof against other potential string parsing and conversion issues.
Automating File Organization through Split Test Project Management Process
File organization is one of those tasks that always seems to fall through the cracks. Everyone has their own personal methods for setting up folders and naming files. Compound that with cloud storage, which is great for file sharing but can easily create more issues when multiple users can manipulate files and folders. A solution intended to foster collaboration ends up creating more friction. Asset management policies are something that should be established and adhered to at an organizational level.
One big problem with asset management policies is that it’s tedious for individual users to adhere to. While not particularly difficult, cutting corners is easier when you’re in the thick of production. File and folder names with “new”, “new new”, “final”, etc. start cropping up. When does a file stop being “new”? 1 week later? 1 month? 6 months?
Corvus CRO is taking one small step towards simplifying asset management by automatically creating a matching asset folder any time a split test moves into the “build” (production) stage. Folders are placed in an “Experiment Assets” parent folder, which is within the client’s root. Having a consistent location makes them easy to locate. The auto-generated folders have the same name as the split test, making it easy to cross-reference with the test itself.
While not comprehensive by any means, this is one small step towards a more robust asset management system. It sets a baseline, makes production tasks just a little bit easier, and opens up opportunity for automating away other tedium in the future. Not everything needs to be a giant leap forward. Lots of little steps can get you to the same place, but the fall isn’t as far if one doesn’t work out.
Organization is key to any successful conversion optimization program. Trapped in a web of asset files? Have a bunch of folder names that don’t make sense? Can’t get a handle on your split testing process? Corvus CRO can help. Schedule a free 30 minute consultation with us today.
Partnering with SessionCam to Easily Identify Friction and Opportunity
Corvus CRO is now a SessionCam solutions partner! SessionCam is a SaaS platform that provides session replay, heatmaps, and conversion funnels.
Behavioral analysis tools are necessary to the success of a conversion optimization program. They provide critically important data visualization around how users actually interact with a website. SessionCam was founded in 2009 and is one of the most mature solutions available on the market today.
The true advantage of SessionCam lies in their built-in process automation coupled with a machine learning engine. Their system identifies and quantifies valuable insight, exposing high value opportunities with minimal effort. Their system can automatically map customer journeys and detect website errors, providing significant time and effort savings; no more manual funnel construction. SessionCam’s opportunities engine automatically identifies anomalies in website traffic and user behavior that need attention, printing them by the associated lost revenue.
SessionCam and Corvus CRO can level up your conversion optimization efforts. Already using SessionCam but need help managing it? Want to get a split testing program started? Unsure which behavioral analysis tool best fits your needs? Corvus CRO can help. Schedule a free 30 minute consultation with us today.