A/B Testing Views with Kentico

We planned to run these tests on five different page types, with only two variations: Control (the current site with no changes) and VariantA (a version of the site with changes to layout and styling).

Kentico EMS

Initially we planned to set up these tests using the Kentico EMS (Enterprise Marketing Suite) module but found some limitations. Tests could only be created to run on individual pages, rather than across all pages of a certain page type. This was an issue as we needed to test changes to the layout of the product page type which would have involved adding the test to hundreds of pages in the CMS. Also, as the site’s views are all .cshtml files in the Visual Studio solution we would still need some custom code in order to be able to change layout based on the active test variant. For this reason we decided it would be easiest to implement our own custom A/B testing.

Storing Tests

In order to manage A/B tests running on the site from within Kentico we created a custom table: True.ABTests. This includes fields for the name of the test, the name of the control and variant, the document ID or document type it should run on, and a boolean flag to enable/disable a test. Each row in the custom table represents a different A/B test:

Checking For Active Tests

We created an AbTestService class to check the A/B tests custom table for any active tests for a given document ID or document type. We used this service from a custom action filter (AbTestable) which we decorated relevant controllers/actions with – this checks for any active tests on page load for the current document ID or document type, and sets the test name and variant name on the HttpRequest object for use when rendering the view later on. The data from the True.ABTests custom table is loaded into memory when the website is first started in order to increase performance, preventing database access for each page request.

The first time a user visits a page with an active test they are assigned one of the two variants at random (Control or VariantA). This choice is stored in a cookie to ensure they see the same variant each time they access the page.

Displaying Different Views

In order to render different views for each test variant we created a custom view engine which extends the RazorViewEngine class. By redefining ViewLocationFormats and PartialViewLocationFormats in the constructor we can tell the view engine to look for views at a specific ABTests path which is dynamically built using the current test name and variant name.

Overriding the CreateView and FileExists methods of RazorViewEngine allows us to pass the view path to the following method in order to populate the test and variant names from the HttpRequest object:

Inserting the test and variant name into the view path 

Finally, by inserting this custom view engine at the first position in the list of view engines in Application_Start we can ensure that if a view exists for a test variant it will be used, with the default view engine used as a fallback for all views which are unchanged for a test.

Tracking Results

In order to track the results of each test we utilised our existing Google Analytics setup. By adding a custom dimension to the page view event we were able to see which variant was shown to each user, and from here draw conclusions about the effectiveness of each variant.

Use In Other Projects

By adding this module to our local NuGet server we can use it across our other Kentico sites. Future additions which may be required include adding multivariate tests (e.g. Control, VariantA, VariantB) as well as allowing customisation of what percentage of users see each variant (currently the split is always 50/50).

If you'd like to talk about how to implement A/B tests please do get in touch.