Author Archives: robertjmoore

RJMetrics Raises $6.25M Series A From Trinity Ventures

Posted on by

The past year here at RJMetrics has been an amazing experience. Our vision—changing the way businesses make data-driven decisions—is becoming more of a reality every day.

Today, we’re proud to announce a new partner on our journey: Trinity Ventures. Trinity has led a $6.25M Series A investment in RJMetrics and Karan Mehandru has joined our Board of Directors. We’ve known Trinity and Karan for quite some time, and we couldn’t be more excited about adding their passion, vision, and expertise to the RJMetrics team.

You can read more about the financing on TechCrunch, Technically Philly, and PRWeb, so I’ll spare you the boilerplate here. Instead, I would like to take a minute to talk about what this news means for you, our customers.

Ever since our bootstrapped beginnings, customers have been the lifeblood of RJMetrics. This customer-centric philosophy is wired into our DNA, and it will continue to permeate our work as we put this new investment to use.

In just the past few months, we’ve rolled out a new dashboard user experience, a revamped chart builder, new ways to get your data into and out of our data warehouse, and a world-class customer success team.

This pace of innovation, which was bolstered by a small seed round last year, will now accelerate even further. Stay tuned for game-changing enhancements and, as always, please don’t be shy if you have feature suggestions.

Thank you so much for your continued support as we move into this exciting new chapter.

Twitter Vine Flying Past Competition Despite Low Overall Adoption

Posted on by

On January 24, 2013, Twitter released Vine, a mobile service that lets you capture and share short looping videos.  We set out to learn just how popular Vine has become in its first month of existence and how its performance has stacked up against competitors like Viddy and Socialcam.

We loaded data from Twitter’s API into RJMetrics and here’s what we found:

  • Overall, video creation is still an extremely underdeveloped market.  Only about 4% of highly active users shared a video through Vine or a top competitor during Vine’s first month on the market.  In that same period, 98% of the same group shared at least one photo through a leading photo sharing service.
  • In its first month, Vine steadily gained market penetration to 2.8% of Twitter’s highly active users, blowing past competitors Viddy and Socialcam, which were used by 0.5% and 0.2% of the same population, respectively.
  • Twitter’s built-in tools for photo and video sharing are dominating the competition.  Vine.co and pic.twitter.com are the most popular tools in their respective categories by a wide margin.

Vine Adoption

Vine showed impressively stable adoption growth over the course of its first month.  We were expecting to see a spike in adoption around the time of the announcement followed by a leveling-off period, but instead the percent of new users each day has remained consistent.  This is a good sign for future growth because the rate of adoption does not appear to be slowing as time goes by.

Vine vs. Viddy vs. SocialCam

In Vine’s first month, the percent of highly active users who used Vine was meaningfully higher than the percent who used competitors Viddy and SocialCam.

We were concerned that this might not be an apples-to-apples comparison since many users may have just been “trying out” Vine in this month.  As a check, we looked at the average number of times each of these tools was used during the month.  As it turned out, repeat usage of Vine was actually more likely than the other apps.

Video vs Photo

While Vine’s performance is impressive relative to its competitors, it’s still a tiny player in the universe of media sharing on Twitter.  We looked at the percentage of users that linked to various media sharing services and found that photos represent the vast majority of the links sent out by highly active users.

As you can see in the chart above, native Twitter photo hosting (pic.twitter.com) is the dominant player, followed by Instagram and then a number of less prominent competitors.

When you remove Twitter and Instagram, you can see just how small a player Vine is when it comes to sharing media.

About The Data

We decided to sample from Twitter’s most active users to find early-adopter activity.  Twitter’s API was used to identify and download the twitter streams of about 2,500 randomly-selected “highly active” users, each of whom had tweeted at least 100 times so far in 2013.

The result was 2.3 million tweets that were sent between January 24th and February 24th.  320,000 of these tweets contained links, which we followed through any link shorteners to find their final destinations.

The data was then loaded into RJMetrics, where we generated this analysis with just a few clicks.

Conclusions

Twitter’s efforts to add native photo and video sharing into its service are proving fruitful.  These tools have quickly become the most popular options for end users, causing a major impact on the market for 3rd party apps.

Vine appears to be establishing itself as the de facto tool for short video creation and sharing.  However, the significance of this move will only be felt as its market matures.  Today, Vine is a service only used by a small minority of even the most highly active users.

 

4 Lean Startup Tactics that Worked

Posted on by

At RJMetrics, we pride ourselves on being a lean start-up focused on ecommerce analytics.  As a result, we’re always bringing new team members up to speed on what it means to be lean. My favorite way to educate them is through real-world examples.

Here are four of my favorite examples (from our team and others) of lean startups in action.

1. Faking A Move

To me, being lean is all about minimizing the ratio of resources consumed to insights gained. At RJMetrics, we did this to great effect when we were hiring our first employees. At that time, our company was based in Camden, NJ, and new applicants were few and far between. We suspected that our location was the reason for the talent shortage, but we had no way to prove that without making a really expensive bet and moving the business to nearby Philadelphia.

Just then, my co-founder Jake had a great idea: let’s just say we moved. We published an identical job posting but changed our address to a location in Downtown Philadelphia. The applicants started pouring in. We simultaneously started interviewing candidates and looking for new office space. We made our move literally one day before our first new employee started work.

Our Awesome Philly Office

Our Awesome Philly Office

I love this story for two reasons. First, had job applicants not increased we would have saved ourselves the trouble of moving with effectively zero downside. We then could have searched for more fundamental reasons that we weren’t getting applicants. Second, it saved us tons of time. By getting the answer to our question up-front, we were able to move and recruit in parallel rather than in sequence. I have no doubt that this accelerated our growth trajectory by months.

 

2. Actually Talking to Users

In the age of cookies, cheap storage, and abundant APIs, many companies have developed a strong bias against actually talking to their users.  Instead, they attempt to infer intent and sentiment from user actions.

While data can do a great job of telling us what is happening, it can often fall short on why it is happening.  Relying solely on data can also mask client frustrations.  Instead, from time to time companies should rely on a more time-honored tactic: asking what customers think.

Customer interviews are inexpensive, fast, and remove a lot of the interpretation risk associated with data-only strategies.  No matter how many numbers we crunch, we always learn something new when we ask our customers about their most and least favorite parts of our product.

 

3. Naming a Book

Tim Ferriss has a huge bag of tricks, but there is one that always sticks with me. When Ferriss was writing his book “The Four Hour Work Week,” he and his publishers were considering a number of potential titles. Rather than go with his gut, Ferriss devised a simple and brilliant strategy: buy Google Ads for each of the different potential titles, have them display when people were searching for content related to the book, and see what got people to click the most.

Users who clicked the ads would land on a blank page or “under construction” page.  All Tim needed to know was what moved people to click.

This inexpensive experiment allowed Ferriss to create an ad-hoc focus group consisting of thousands of ad viewers and learn which titles piqued their interest the most. Based on sales of The Four Hour Work Week, it’s quite clear that he made the right choice.   Ferriss shares this story in the video below.

 

4. Interactive Mock-Ups

Philadelphia entrepreneur Chris Cera recently turned me onto a really neat tool he uses called Axure. With Axure, you can to mock up interactive user interfaces without any backend coding. In other words, as long as testers follow pre-determined steps, they can get the impression that they’re interacting with a fully-built product.

Anyone who has done user experience work can appreciate why this is a brilliant and valuable tactic. An interactive mock UI/UX can answer huge up-front questions about how and when potential users would derive value from a product (or experience frustrations). With these questions answered, the development of the backend can be much more focused and deliberate, and there is far less risk associated with the final product’s release.

 

 

The 5 Most Common Data Analysis Mistakes

Posted on by

Data-driven decisions are the backbone of modern online businesses. As too many learn, however, the only thing worse than not using your data is using it incorrectly. Here are five mistakes that we’ve seen companies make (before getting their ecommerce analytics on the right track with RJMetrics).

1. Not Accounting for Age

Aggregate analysis of customer behavior can be extremely misleading. For example, consider a company that has acquired customers through Google Ads for years, but only recently started spending money on Sponsored Tweets. An analysis of “customer lifetime spending by acquisition source” would likely show that Google-acquired customers, on average, have spent more than Twitter-acquired customers. But, this doesn’t mean that Twitter is an inferior source of leads—it’s just a misleading way of slicing the data. Of course the average Google customer has spent more: they have been a customer for a longer amount of time and had more opportunities to make repeat purchases.

This is an extremely basic example, but variations on this oversight lead to a surprising amount of confusion in companies of all sizes. To avoid this problem, use cohort analysis to segment customers by common attributes like when they made their first purchase. This allows for apples-to-apples comparisons of customer cohorts that can then be expanded into more insightful and actionable analyses.

CohortAnalysis

A sample cohort analysis

 

2. Not Planning Next Steps

Before you invest time in running an experiment or conducting an analysis, it’s important to understand how the results will impact your behavior. There will always be countless ways to slice and dice your data, and the best way to avoid analysis paralysis is to focus on metrics that will drive you to act.

To determine if a metric is actionable, simply consider the possible results of your analysis and ask yourself how your strategy or behavior will change based on the results. If it won’t, maybe your time would be better spent elsewhere.

 

3. Ignoring Test Significance

Whether you’re running an A/B test or performing ad hoc analysis on your data, remember that the sample size of your data set is key to the significance of your analysis (i.e., how likely it is that your observation is representative of the total population).

Many tests are not worth running, and you can save lots of time being realistic about when data may or may not hold the answers. Sites like test significance can help you determine how costly a given test might be and what it takes to reach significance.

Online Test Significance Tool

Online Test Significance Tool

 

4. Stopping at the Surface

Imagine that a company is studying which referral sources yield the most valuable customers. If they find that one source is superior, they have discovered a correlation between referral source and Customer Lifetime Value (CLV). However, this does not mean that referral source is causing CLV to be higher for those customers—all it means is that they are linked.

Unseen “lingering” variables could be the true reason for the correlation. Perhaps your online store is not well optimized for a female audience and the best-performing referral source is simply the one that refers you the highest percentage of male shoppers. In this case, blindly shifting all of your spend to that referral source could backfire if their demographics shift.

The smarter move would be to redesign your homepage, which would lift CLV across all channels. The only way to discover this opportunity, however, is to dig deeper within your data. Look at all of the characteristics of your customers – not just the ones tied directly to marketing spend – and you may discover far more meaningful metrics to act upon.

 

5. Modeling Growth Projections By Percentage

I worked in venture capital prior to starting RJMetrics, and I quickly learned that most company financial models (which forecast future growth) are exercises in fantasy. One of the main reasons is that so many models are driven by inputs like “percentage growth per month.” In these models, a slight change in that  parameter can be the difference between a billion-dollar company and a dud.

This blanket “percentage growth” methodology says nothing about what’s driving that growth or what such growth actually means in terms of the number of customers added, where they came from, and how they were monetized.

Rather than an exercise in fantasy, make it a model that demonstrates the fundamental economics of scaling your business. Build a “bottom-up” model by using more granular inputs that are specific to your business model. This will lead to a more productive conversation and help set you apart from the pack.

 

 

RJMetrics Winter 2013 Hackathon Results

Posted on by

After the last RJMetrics hackathon, I didn’t think our team could possibly cram more innovation into a 24-hour period.  They just did.

The Projects

Team members unveiled some amazing projects, including:

Spotlight search in our new dashboard UI to provide users with fast access to charts, dashboards, and trends.

Customizations to Zendesk to make our customer support exchanges more streamlined.

A customized sales video generator to provide a personal touch to our sales prospects.

Shaun Presents His Video Generator

Shaun Presents His Video Generator

BallerBoard, a TV display engine that automatically shows stats from RJMetrics, Twitter, and other services in a format that’s easy on the eyes.

BallerBoard

An End-User Query Browser to allow advanced users to query their data warehouses directly using SQL syntax.

A deployment system for MySQL stored procedures, which will greatly increase the number of analyses we can run natively in MySQL.  Last Hackathon’s Median/Percentile feature will be deployed using this system.

A trend-line overlay system that will allow users to fit regression lines to their data and forecast future data points based on these models.

TrendLine

A system for self-auditing and approving RJMetrics Trend/Metric definitions through our UI.

A new concept and 3D rendering of a new RJMetrics conference booth.

Conf

An improved RJMetrics deployment system.  This is an extension of our new AWESOM-O deployment system and its client application Butters.

Drastic improvements to our physical office environment, including a new reception area and accent walls throughout the office.

Working Capybara integration tests to monitor our UI.

 

The Results

Francis “Buck” Ryan took the crown this time around for his work on trend-line overlays.    This was a suggestion that came directly from our feature request page.  Existing users can keep an eye out for it in the beta tests of our new dashboard UI.

Buck will enjoy the grand prize: $500 cash to be spent all in one night.

I can’t wait for Spring.

 

 

Why Many A/B Tests Aren’t Worth It

Posted on by

Note: This post originally appeared as a guest feature on TechCruch to announce our new Test Significance website.

At RJMetrics, we believe in data-driven decisions and that means we do a lot of testing.  However, one of the most important lessons we’ve learned is this: not all tests are worth running.

In a data-driven organization, it’s very tempting to say things like “let’s settle this argument about changing the button font with an A/B test!”  Yes, you certainly could do that.  And you would likely (eventually) declare a winner.  However, you will also have squandered precious resources in search of the answer to a bike shed question.  Testing is good, but not all tests are.  Conserve your resources.  Stop running stupid tests.

The reason for this comes from how statistical confidence is calculated.  The formulas that govern confidence in hypothesis testing reveal an important truth:

Tests where a larger change is observed require a smaller sample size to reach statistical significance.

(If you’d like to dig into why this is the case, a good place to start is Wikipedia’s articles on hypothesis testing and the binomial distribution.)

In other words, the bigger the impact of your change, the sooner you can be confident that the change is not just statistical noise.  This is intuitive but often ignored.  And the implications for early-stage companies are tremendous.

If your site has millions of visitors per month, this isn’t a big deal.  You have enough traffic to hyper-optimize and test hundreds of small changes per month.  But what if, like most start-ups, you only have a few thousand visitors per month?  In these cases, testing small changes can invoke a form of analysis paralysis that prevents you from acting quickly.

Consider a site that has 10,000 visitors per month and has a 5.0% conversion rate.  The table below shows how long it will take to run a “conclusive” test (95% confidence) based on how much the change impacts conversion rate.

Starting
Conversion
Rate		 New
Conversion
Rate		 Total
Participants
Required		 Test
Duration
Required
5.00% 5.20% 185,926 1.5 years
5.00% 5.40% 47,340 5 months
5.00% 5.60% 21,420 2 months
5.00% 5.80% 12,262 37 days
5.00% 6.00% 7,982 24 days
5.00% 6.20% 5,638 17 days
5.00% 6.40% 4,210 12 days
5.00% 6.60% 3,276 10 days
5.00% 6.80% 2,630 8 days
5.00% 6.90% 2,378 7 days
5.00% 7.00% 2,162 6.5 days
5.00% 7.20% 1,814 5.4 days
5.00% 7.40% 1,548 4.6 days
5.00% 7.60% 1,338 4.0 days
5.00% 7.80% 1,170 3.5 days
5.00% 8.00% 1,034 3.1 days

(Data assumes a Bernoulli Trial experiment with a two-tailed hypothesis test and all traffic being split 50/50 between the test groups.)

As you can see, your visitors are precious assets.  Too many start-ups will run that “button font” test, expecting full well that in a best-case scenario it will only impact conversion by a quarter of a percent.  What they don’t appreciate up-front is that this may block their ability to run certain other tests for a year and a half (assuming they don’t end the test prematurely).

When you can’t run many tests, you should test big bets.  A homepage redesign.  A pricing change.  A new “company voice” throughout your copy.  Not only will these tests potentially have a bigger impact, you’ll have confidence sooner if they do.

I found myself making this argument a lot recently here at RJMetrics, so I developed a tool to calculate the required population size for a significant test.  We’ve shared that tool with the world free of charge at Test Significance.  Just input your current conversion rate and your desired confidence interval and it will generate a table like the one above.

TestSigScreenshot-Better

We hope this tool helps a few companies out there learn the lessons we have about when to test and what to expect in terms of finding a conclusive result.

 

Return of the RJMetrics Hackathon

Posted on by

This Thursday at Noon, we will kick off our second seasonal RJMetrics Hackathon.

Our previous hackathon was an enormous success. It caused major disruption in our development pipeline. Many of that hackathon’s projects are currently being beta tested and will hit our production codebase soon.

It also spawned such live features as:

  • QueryMongo.com, which topped Hacker News and helps dozens of coders (including our team members and customers) every day.
  • A snazzy new sales video, which is being A/B tested on our homepage.
  • And much, much more.

Most importantly, everyone had a great time and couldn’t wait for the next one.

The Prize

Last time, our winners enjoyed a lavish dinner at Del Frisco’s Double Eagle Steakhouse. This time, we’ve upped the ante.

This Hackathon’s prize is inspired by the classic film Brewster’s Millions.

The winning team will be given $500, with the caveat that they must SPEND IT ALL IN ONE NIGHT. You can pick any night you want and spend it however you want, but that money has to be gone by the time the sun comes up. Winners are encouraged to document their shenanigans.

Taking Suggestions

Got a suggestion for something our team should take on at the Hackathon? Let us know by emailing support@rjmetrics.com with your suggestions. They will be passed on to our entire team.

Check back next week for results!

MySQL to MongoDB Query Translator

Posted on by

A few weeks ago, the first-ever RJMetrics hackathon took place at our Philadelphia headquarters. I decided to throw my hat into the ring with a project I’d been thinking about for a while: a MySQL to MongoDB query translator.

This was a unique challenge because MongoDB and MySQL are very different technologies that store data in very different ways. To some, translating between them might seem like a non-sequitur. However, I knew there was a use case because of my personal experience learning MongoDB. I would often think about queries in terms of SQL syntax, and a translator like this would have greatly softened the learning curve.

The final product is available at our Query Mongo site, and I encourage you to give it a try. It’s not perfect, but we hope it will be a helpful learning tool for the many people who have SQL experience and are getting started with MongoDB.

In this blog post, I’ll provide some insights into how this tool works.

Continue reading →

RJMetrics Fall 2012 Hackathon

Posted on by

This past weekend, we had one of the most successful events in our company’s history: 16 team members participated in a 24-hour Hackathon. With a massive trophy and a fancy steak dinner on the line, everybody came to win. 

The Rules

This was our first Hackathon, so we asked our friends at other startups about their own best practices. We settled on the rules below.

Logistics:

  • Everyone in the company is encouraged to participate, but it’s not mandatory for anyone.
  • All normal work except for time sensitive customer support should stop for participants.
  • The event kicks off at noon on Friday and ends with “pencils down” at noon sharp on Saturday. Demos and voting will immediately follow.
  • Hackathon projects can be literally anything related to the company in some way. Code, prose, images, audio, video, data analyses, web pages, physical objects and anything else are OK.
  • Participants can brainstorm ideas and form teams at any time between the announcement and the start of the Hackathon. The definitive teams need to be documented no later than one hour after the start of the Hackathon.
  • Teams can be any size with any mix of company role.
  • Normal testing/feedback/QA/review rules are not mandatory for work done during the Hackathon.
  • Hackathon projects might end up being re-written, changed, becoming a core part of what we do, or being completely discarded after the Hackathon.
  • Open source technology may be used as a component of your project.
Presentations and Voting:
  • Demos take place at the end. Each team gets up to 5 minutes for a demonstration and up to 5 minutes for questions. Presentation order will be randomly drawn.
  • Vote for the team that you feel generated the most value for RJMetrics during the time of the Hackathon. If most of the work was done ahead of time, but the final 5 percent was done during the Hackathon, they get 5 percent of the credit.
  • All Hackathon participants will cast votes in the style of instant runoff voting, but they cannot vote for their own team.
  • The winning team get their names on a trophy and a $300 gift card to Del Friscos. The trophy will circulate among the desks of the winners until the next Hackathon, á la the Stanley Cup.

Ample food was always in supply

The Projects

In the end, eight submissions were presented. Our customers should keep an eye out for some of these making their way into our product in the coming weeks.

Adding medians and percentiles to our dashboards: medians aren’t native functionality in most SQL languages, which required the implementation of special stored procedures to make them work.

Automated Training: a team used Kera.io to build interactive online tutorials in which new users can learn by manipulating their own dashboard environments.

Nate and Xiao hammering out the user community project

User Community: a tightly-integrated online community for user discussions and sharing best practices.

Responsive iPad UI: a new user interface for our dashboards that allows iPad users to have an even easier navigation experience in our product.

More Data Sources: integration with Zapier to allow us to easily connect with third party APIs like Shopify, Salesforce.com and Zendesk.

Sales Video: this sweet video was built using PowToon and VoiceBunny and may soon be appearing on an A/B test on our homepage.

New Logo: an impressive prototype for a new RJMetrics logo.

MySQL to MongoDB Query Translator: a tool to translate a traditional MySQL query into its eqivalent in MongoDB.

Red Bull gives Rohan wings

The Results

Our main conclusion from this event: Hackathons are awesome. The time flew by, we all had tons of fun and the output was extremely impressive. It reaffirmed to everyone that there really are no weak links on this team– everyone was a real competitor.

The crew enjoys the final presentations

The Query Translator and iPad UI teams took home first and second place, and both will be enjoying steaks at Del Frisco’s soon.

As for the rest of the crew, an anonymous follow-up survey revealed that over 90 percent of participants would be “extremely likely” to participate in another Hackathon. At this rate, our next one may be just around the corner.

How to Get On the Front Page of Hacker News

Posted on by

Obligatory Plug (let’s get this out of the way): I’m the co-founder of RJMetrics. Our hosted software helps e-commerce and SaaS businesses make smarter decisions using their data. If you’d like to see what RJMetrics can do for your business, sign up for a free 30 day trial. OK, onto the post…

I recently conducted a comprehensive Hacker News Data Analysis to settle a bet with my co-founder Jake (he won, by the way). As a byproduct of the experiment, I was sitting on a wealth of data about Hacker News submissions and their performance.

One of our team members suggested that we could use this data to learn more about how to maximize our chances of our Hacker News submissions making it to the front page. Rather than hoard the data, we decided to share it with the world in a blog post. Here’s what we found.

Continue reading →