Category Archives: Testing

What exactly is AB Testing?

Oh man.. I’m still quite the neophyte when it comes to web/mobile development – there’s still much I need to learn in terms of methodology and tools.

That being said, I kinda wish user experience jargon could be a little more intuitive. haha.. eat your own medicine eh?

AB testing is (simply put) a simple science experiment used to make a decision between two choices.

There’s a control – A

There’s an alternative – B

Ideally, A and B should only differ on ONE variable (the variable you are interested in – this could be placement of a particular button or so on) Common sense dictates that including too many variables in one test just muddles the end results, though i suspect the lack of time and resources just makes people do a short cut.

We choose a test group (of statistical significance) and randomly allocate the subjects into either the control condition or the alternative condition and check for resulting conversion rates.

Conversion rates essentially refers to the number of subjects whom are sufficiently “convinced” in the particular condition to become a “follower”. In more web development/marketing specific terms, it just refers to first time visitors who decide to buy something or follow something.

That’s AB testing in a nutshell. Useful tool for development, but I can imagine¬†the amount of effort required. Am not too sure on a few points though –

1. Seems like AB testing is used when there’s an existing product/interface and one wants to make changes and see if the changes has a positive effect on conversion rates.

Does it make sense to do it on a brand new project — to offer two alternatives and potentially double the development effort?

Or is it used in a incremental manner – compare updated development version with earlier version to see if change is well received. If not, ditch the change and move on to a different feature.

2. In a development phase, how do we get a steady stream of first time visitors? Or do we just have a pool of testers and re-use the testers? (the mere hypothesis would argue against this)

Read more about AB testing here.