In an effort to improve the effectiveness of our website, we recently conducted our first A/B test. This test allowed us to compare two options for the content that users see after clicking the big orange button on our homepage. Thanks to these tests, our visitor-to-lead conversion rate has almost tripled.

Choosing a Goal

We used Google Website Optimizer (GWO) to run the tests, and our first step was deciding on a goal. We decided that a good goal needs to be achievable, measurable, and a meaningful step of the funnel that ends in RJMetrics making money. We also wanted the goal to be as close to the end of the funnel where we make money as possible.

For most businesses, a sale or other revenue-generating event is an obvious goal to use. In our case, however, prospective customers need to talk to a member of our team before making a purchase. This means our website doesn’t generate revenue directly, leaving us no revenue-based actions to use as goal completions. There are several reasons for this set-up, but the important thing is that it is not going to change for this A/B test.

Another goal we considered was the successful education of our visitors about our product. Educating potential customers about RJMetrics and the problems we solve is certainly an achievable goal that drives new customer acquisition. However, it is difficult to reliably measure. We could track how many people watch our promotional video to completion or how many pages they visit. But these specific events are not always meaningful, and frankly I don’t really care about them. Additionally, they are way over on the wrong side of the funnel.

A third option for our goal was the completion of our “get a free demo” form. It’s trivially easy to measure and provides prospects with a feel for what our software can do (by analyzing Vandelay Industries’ data). It also provides us with contact information for these now somewhat-qualified sales leads. This places it relatively far down the funnel. While it would be nicer to further qualify the leads before considering one a “goal completion,” that would be much more difficult to reliably monitor and would reduce our already-small sample size.

We decided that the demo form completion was the right goal for us to be optimizing for right now.

Conducting the Test

The old version of our landing page is still available at our How It Works page. It looks like this:

The page has a Flash movie that gives an overview of our target customers, the problems they face, and how RJMetrics can help. Below the flash movie, we have some marketing copy that gives additional color for people who would rather read than watch. We did not explicitly talk about a goal when creating the page, but if I had to describe what it does, I would say that this page gives information on RJMetrics and gives prospects a way to try out our free demo.

The winning version of the page is our new landing page. It looks like this:

We removed the movie and marketing copy and increased the size of the form fields. We also added a description of what will happen after filling out form. The goal was to minimize the commitment required and make the reciprocity explicit. This page is unambiguous about its goal: driving demo form completion.

The Results

We did a total of 3,519 trials. The original version yielded a conversion rate of 2.1%, and the winning yielded a conversion rate of 6.1%, providing an improvement of over 190%.

We learned a few valuable lessons from this experiment:
1) We need to set an explicit goal for our website, or at the very least for individual pages.
2) Remove anything that does not explicitly drive our goal forward. In this case, more content meant fewer conversions.
3) We need to be testing a lot more.

cta-email-saas_03

We’d like to acquire you as a reader

Get SaaS metrics in your email on a recurring basis.


  • brian armstrong

    Do you feel like people who come to the landing page already know what RJMetrics is?
    If I didn’t, and saw the second landing page, I would be hesitant to put my email if I’m not sure whether it’s right for me yet. I might be curious enough to enter my email just to find out (hence the improvement), but if I got inside and found out it wasn’t what I wanted, I still wouldn’t become a user of the app.

    Anyway, you probably thought of that but just wanted to point that out. Capturing emails is an intermediary step, not the ultimate goal.

  • Jake Stein

    Brian, I think most people do. There are a lot of other resources on the site for people to learn about us, so they still have options if they don’t know. Those options are just not on the landing page now.
    Completely agree that capturing emails is not the end goal, but it’s the right fit for this experiment. We are absolutely going to be looking at how these leads pan out.

  • Tawheed Kader

    My guess is people end up clicking on the links right above to learn more about the service, and then come back to the trial form to sign up.
    I think what you said about each page having a key goal (a.k.a. focus) is incredibly important here. Your older version had a muddled goal, thereby giving the user muddled content.

    Great post and very insightful!

  • Kevin Granade

    I can understand why you have a one-on-one conversation with a team member as a prerequisite for a sale, from what I can see of your offerings it’s not exactly a “one size fits all” proposition, or even a “one size fits more than one target” proposition. I’d recommend that you take a look at http://www.kalzumeus.com/ which is the blog of an entrepreneur running a single-product (ok two products, but they’re basically the same thing) business. Despite the difference in your products, I think you could learn a lot from his writing on sale-funnel optimization, site design, and A/B testing.
    P.S. Don’t be put off by the current (rather silly) post, he really has some amazingly insightful posts there.

  • Jake Stein

    I have actually read a few of the posts there, and they were a big part of the inspiration for this post. I think there is a lot more to learn there.

  • Stephen

    I would say this confirms the way people are using internet these days. One is no longer seeking with google scanning websides and landing pages to find there answers. Your social network leads you by subject to the page you’re interested in. Visitors have specialized knowlegde of the subject and request for answers, not for introductions.
    Therefor each page having it’s own key goal is very important. The landing page can be any page these days. Good thing to keep in mind. Very interesting and useful article.