Written in C#, mtelligent is a cohort analysis, AB and multivariant testing framework for ASP.Net

Heavily inspired by the lean startup, I set out to build a framework for ASP.Net to support the kind of validated learning the lean start up methodology advocates. It's still very rough, but I wanted to put it out there to get some feedback before moving further. For now, the working title is mtelligent, because I always loved the domain (my blog runs on it) and it sort of fits: Marketing Intelligence.

The basic goal of the framework is to enable you to run experiments on your users (or segments of your users) to validate the hypotheses you make about your users and really understand the value that changes you make to your application bring. You can run simple experiments that just tweak aspects of your user interface, or complex experiments with tons of variables, completely different business logic, whatever you want.

The framework is made up of two components: A class library that can be integrated into your ASP.Net projects by hooking into your Global ASAX Application class and a Dashboard Application that can be used to define meta data about your experiments, hypotheses, cohorts, goals and sites as well as give you basic information about how your experiments are running. (Ignore Sites for now, they are not being using in reporting yet)

Experiments

Before you can leverage the framework to experiment on your users, you need to define the meta data associated with the experiment. The first step is defining an experiment:

Experiments have both a name and System Name. Once Created the System Name cannot be changed as that is what will be used in code to get details of the experiment from its API. You also need to define what cohort you want to target with this experiment (more on that in the cohort section) and what goals you will use to determine which hypothesis was most successful. You also can define multiple custom variables. Variables enable you to define what are the differences between your hypotheses and allow developers to fully customize the user experience to those variable settings when showing a specific hypothesis to a user.

Hypotheses

After you create your experiment, you can then add hypotheses to it. Each Hypothesis requires you to give it both a name and system name and provide variable values that are specific to that hypothesis. Users will only ever be shown one hypothesis for an experiment ever as a tracking cookie is used to identify the user and details about what segment of the experiment the user see is stored in the backend database.

Once you create all your hypotheses, you can set their target percentage. This value is used to determine how the random distribution of selection of hypotheses will be made for visitors. The sum of the Target percentages should equal 100 across all hypotheses. If you just want an even distribution, click the icon next to the Target Percentage heading. If you no longer want to test a hypothesis, set the target percentage to 0. If you are not targeting the "All Users" cohort, you also need to check the "Is Default" checkbox on one of your hypothesis. This hypothesis will be used when the visitor is outside the target cohort and their results will not be tracked for the experiment.

Goals

Goals are how we know which Hypothesis is most successful. When a user fulfills a goal, we call it a conversion.

They are pretty simple to setup. You only need to define a name and system name and give it a value. The value needs to be numeric as the sum of the conversions x value is the metric that determines which outcome is best.

The code to register a conversion is pretty simple:

ExperimentManager.Current.AddConversion("Conversion System Name");

The framework will track all the conversions you trigger during the pages lifetime and you can add a method to your master page or master layout to render any custom JavaScript or Google Analytics code related to those coversions.

@Html.RenderConversionScripts() for MVC or <%= Mtelligent.Web.ExperimentManager.Current.RenderConversionScripts() %> for WebForms

Cohorts

Cohorts give you the ability to segment your users into different groups and then target those groups with specific functionality. You automatically get two cohorts out of the box: "All Users" and "Authenticated Users." These target exactly what you would expect.

The framework is pretty flexible for defining your own custom cohorts and I've already implemented several customizable cohorts that give you a bunch of ways to segment your users. These include:

• Role Cohort - Target authenticated users by Role
• First Visit Cohort - Target visitors based on the date of their first visit.
• Referrer Cohort - Target visitors by where they came from.
• Landing Url Cohort - Target visitors by what url they arrived through.
• Goal Cohort - Target visitors that already converted on another goal.
• SegmentCohort - Target visitors that are already participating in an experiment and have been selected for a hypothesis (I renamed Experiment Segment to Hypothesis later in my development).
• Attribute Cohort - Add your own variables to visitors and use them to segment your users.

Each of these cohorts, require you to specify some cohort parameter (what role, date range, etc you are trying to use to segment your users). Note that you don't have to specify full urls for the landing url or referrer cohort types as the value will just be used with the indexOf funciton to determine whether it matches any of the visitors actual referrers or landing pages.

Also note we keep track of all of a visitors landing pages and referrers. A landing page is any url they arrive at that either has no referrer or a referrer whose host name doesn't match the current pages host name. Referrers are any referring url whose host name doesn't match the current pages host name.

API

Most of the API is facaded through the ExperimentManager class. It is implemented as a Singleton and exposes the following methods:

Initialize
This method must be called in the constructor of the HttpApplication in your site's Global.asax. It hooks into two events:

• Authenticate Request - Initializes the Visitor either from the cookie or User Name (if they are logged in).
• PreSendRequestHeaders - Spawns a thread to commit any visitor details to the database. Where possible we try to avoid making any database calls until the end of the request in a separate thread to avoid impacting performance.

Also note that we have implemented logic to reconcile users if they have activity before logging in and had a pre-existing visitor record. We also detect when the username changes to ensure every user name is associated with a separate visitor record.

AddConversion
Pass the system name of the goal you wish to register a conversion for and this method handles the rest.

AddVisitorAttribute
If you're using Attribute Cohorts, use this method to save details about a visitor that could be used to segment them later.

RemoveVisitorAttribute
Remove the attribute associated with a user with this method.

GetHypothesis
Pass the experiment name and get back details about the hypothesis the current visitor is assigned. If none are assigned it automatically will pick one based on the target percentages. If the user isn't in the target cohort, they will get back the default hypothesis.

For Testing if you want to ensure you receive a specific hypothesis you can override the normal functionality by appending "?Hypothesis=System Name of Hypothesis" to your querystring.

IsVisitorInCohort
If you're less interested in running experiments and more interested in targeting different user cohorts, use this method to determine if the current visitor is in any of your configured system cohorts.

HtmlHelpers

For convenience, we also implemented a couple of HtmlHelpers to make it easy to get hypotheses details from Razor:

GetHypothesis
Same as the Experiment Manager method, but useful if you need to execute control flow based on the hypothesis name or other parameters.

GetHypothesisVariable
Pass the experiment name and variable name and get back the string variable value associated with the users assigned hypothesis.

IsVisitorInCohort
Same as the Experiment Manager method.

The Dashboard

To keep score, I've implemented a rather simplistic dashboard that shows visitors in each hypotheses and conversions and conversion value for each active experiment.

The more I think about the data being collected, the more opportunities I can see for custom reports that compares cohorts to goals, experiments to other experiments and other metrics.

What's Next

I haven't done a ton of testing and there is still plenty of work to do. I don’t even have a package installer available yet. Hopefully this is a solid foundation to build upon.

The solution is pretty straightforward, with the library code in the Mtelligent project, and the Dashboard code in the Mtelligent.Dashboard project. The backend database is SQL Server and have everything in a SQL Database project. So theoretically, you just need to deploy the database project, update your connection strings as needed and you'll be good to go.

If you're integrating into your own project, you just need to add the reference to the Mtelligent assembly, copy the connection string and custom config section to your web config from the Dashboard (or test project), configure the Mtelligent Http Module and start coding to the experiments you configured when running the dashboard pointing to the same database.