growthbook / growthbook Goto Github PK

Currently hard-coded as:

• 0 <= risk <= 0.25% = green
• 0.25% < risk < 1.25% = yellow
• 1.25% <= risk < 100% = red

Add ability to override these thresholds on a per-metric basis. Some metrics are more important than others and will have a smaller risk tolerance.

Also allow changing the default thresholds via environment variables

Currently, all organizations, datasources, metrics, dimensions, apikeys, webhooks, and settings are defined within MongoDB.

For self-hosted installations, we should support defining these (or a subset of these) via a configuration file. Thinking of an initial yml structure like this:

organization:
  name: My Company
datasources:
  warehouse:
    name: Main Warehouse
    type: postgres
    host: localhost
    port: 5432
    user: root
    password: ${POSTGRES_PASSWORD}
    ...
metrics:
  signups:
    name: Sign Ups
    type: binomial
    datasource: warehouse
    sql: SELECT user_id, timestamp from signups
    ...
dimensions:
  country:
    name: Country
    datasource: warehouse
    sql: SELECT user_id, country as value from users

When the config file is present, the following should happen:

• Load the config file on back-end startup and throw error immediately if the structure is invalid
• The model classes for organizations, metrics, datasources, and dimensions will return data from config instead of mongo and throw an error when trying to edit or create
• Signing up for an account in the UI will automatically add the person to the organization in the config
• The front-end hides all CTAs to create or edit datasources, metrics, and dimensions
• Any blank-slate page for datasources, metrics, or dimensions (e.g. onboarding) should show config instructions instead of a create button

Ability to mark two or more experiments as mutually exclusive, meaning a single user will never be included in more than one of them. We can do this with a new optional experiment property namespace.

The namespace property is a tuple with 3 parts: id, range_start, and range_end. For example, [pricing, 0, 0.5].

Users are assigned a value of 0 to 1 based on a hash of their id and the namespace id.

The range defines which part of the namespace should be included in the experiment.

The following two experiments would be mutually exclusive.

{
  "key": "test1",
  "variations": [0, 1],
  "namespace": ["pricing", 0, 0.5]
}

{
  "key": "test2",
  "variations": [0, 1],
  "namespace": ["pricing", 0.5, 1]
}

The benefits of this approach are the simplicity and that it doesn't require any API calls or persistence. That means there is minimal impact to complexity, size, and performance.

The biggest downside is that you can't start an experiment at 100% and then decide halfway through that you want to add a new mutually exclusive experiment. You need to plan up front to leave space. This is a little restrictive, but that's not necessarily a bad thing since this way it's much harder to shoot yourself in the foot and cause statistical problems down the road.

Add an option to persist experiments and results back to a SQL data source automatically.

New data source settings

• sync_results (boolean, default false)
• table_prefix (can include a dot to write to another schema in postgres/redshift e.g. growthbook.)

New cronjob:

• Gather all the data from MongoDB
• Connect to data source
• Run DELETE TABLE IF EXISTS... queries
• Run CREATE TABLE ... queries
• Run INSERT INTO ... queries

Proposed schema:

• metrics
  • metric_id
  • name
  • type
• metrics_tags
  • metric_id
  • tag
• experiments
  • experiment_id
  • name
  • tracking_key
  • project
  • owner
  • hypothesis
  • num_variations
  • status (draft, running, stopped)
  • archived (boolean)
  • result (win, lose, dnf, inconclusive)
  • winner
  • created_at
  • started_at
  • ended_at
  • synced_at
  • latest_phase_id
• experiments_tags
  • experiment_id
  • tag
• experiments_metrics
  • experiment_id
  • metric_id
  • type (goal or guardrail)
• phases
  • phase_id
  • experiment_id
  • type (ramp, main, or holdout)
  • coverage
  • traffic_split (e.g. 50/50 or 40/40/20)
  • status (finished, running)
  • started_at
  • ended_at
• results
  • phase_id
  • metric_id
  • variation
  • users
  • count
  • mean
  • stddev
  • conversion_rate
  • percent_improvement
  • risk_of_choosing
  • chance_to_beat_control

Hi there,

It would be great if you could add the option of enforcing an SSL connection on the SQL connectors.

Otherwise, we cannot use it, since our Redshift enforces the SSL to be enabled for security reasons.

Could you think about using ClickHouse instead of MongoDB?
Please look to http://clickhouse.yandex/docs/en/
ClickHouse is a blazing fast Columnar database, which contains scale-out cluster data models

1. Customize the time period (currently 3 months)
2. Customize the x-axis grouping (currently grouping by day)
3. Add shaded standard deviations when possible
4. Ability to select a user segment to filter the graph

For self-hosted deployments, allow an admin to reset a user's password from the team settings page. This is especially useful when an email SMTP server is not configured. Currently, the only way to reset passwords in that case is to look for a reset link in the server logs.

Hi there. First of all, this project is awesome. Kudos to you all!

I have a question: is there a way through the UI to setup the DataSourceSettings that are used so that I don't need to change the default SQL queries?

After reading the source code it seems there is an option to change the column names instead of rewriting the SQL statement, but I can't find out how.

Thanks

Client libraries can accept a cached list of experiment overrides. Currently, the only way to keep that up to date is to periodically pull from the Growth Book API. There should be an option to push this data instead with a webhook.

Use https://github.com/agenda/agenda as a simple MongoDB job queue

• Add a new webhook property to organizations that stores an endpoint and signing secret
• UI in settings to add a webhook
• Define a webhook job with an organizationId property
• Make the job unique so only a single one per organization exists at a time
• Create the job when an experiment is created or updated and the organization has a webhook defined
• If a job fails, retry it a few times with exponential back-off
• After 3 failures, email the organization owner and stop retrying

Running yarn on a clean clone of the repo fails with the following error after fetching all packages:

error https://registry.yarnpkg.com/rehype-add-classes/-/rehype-add-classes-1.0.0.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "EPERM: operation not permitted, open '/Users/sdll/Library/Caches/Yarn/v6/npm-rehype-add-classes-1.0.0-3d8b37b13ce06e2eb8681b33ccf1c2995611a33b-integrity/node_modules/rehype-add-classes/test.js'"

Trying out yarn install --network-concurrency 1 and yarn cache clean and rm -rf ./yarn.lock && yarn did not help, all suggested here:

yarnpkg/yarn#6312

This issue suggests that the error is related to the prebuild script of rehype-add-classes:

yarnpkg/yarn#7212 (comment)

Environment

• node --version: v14.17.5
• yarn --version: 1.22.11
• system_profiler SPSoftwareDataType
  • System Version: macOS 11.5.1 (20G80)
  • Kernel Version: Darwin 20.6.0

Matomo is an Open Source competitor to Google Analytics. https://matomo.org/

It would be awesome to have an integration to it because Matomo is GDPR Friendly like GrpwthBook if self-hosted and so no opt-in is needed.

Documentation: https://developer.matomo.org/guides/reporting-introduction

Currently, dimensions are defined with standalone SQL queries. SELECT user_id, country as value FROM users. This works fine for user-level properties, but does not work for things like "browser" since the same user may have different values over time.

We should add a new type of experiment-level dimension which is defined by extra columns on the Experiments data source query.

SELECT
  user_id,
  anonymous_id,
  received_at as timestamp,
  experiment_id,
  variation_id,
  context_page_path as url,
  context_user_agent as user_agent,
  context_campaign_source as source -- this is an extra column that could be a dimension
FROM
  experiment_viewed

When saving a data source, we just run a SELECT 1 query right now to test the connection. We could instead run the experiment query with a LIMIT 1 clause and inspect the result and save extra column names in Mongo

Currently, each metric has its own SQL query, but this can be inefficient when there are multiple metrics all reading from the same table. For example, Purchased, Revenue per User, Average Order Value, and Avg Order Size could all theoretically be powered by the same SQL query.

This requires two changes:

1. Allow specifying the value column name in metric query settings (currently hardcoded as value). This will be easy.
2. If two metrics for an experiment have the exact same SQL query and conversion windows, only run the query once and split out the raw results in a post-processing step. This will be hard.

With these two changes, you could support the 4 metrics above all with the following query:

SELECT
  revenue,
  items,
  user_id,
  timestamp
FROM
  orders

The only difference would be the settings:

• Purchased: type = binomial
• Revenue per User: type = revenue, valueColumn = revenue, convertedOnly = false
• Average Order Value: type = revenue, valueColumn = revenue, convertedOnly = true
• Avg Order Size: type = count, valueColumn = items, convertedOnly = true

Currently, mixpanel only filters events down to the day. So if you start an experiment at noon, it will include all events up to 12 hours before the experiment started.

Looking at results by date is an important debugging tool when things go wrong. You can see potential novelty or primacy effects, unusual traffic spikes, etc.

We should add "date" to the experiment results dimension drop-down. When selected, it will show results per day based on when the user was first exposed to the experiment. Might need two separate dimensions, one for cumulative results and one for each individual day. May also want different granularity - day, every 6 hours, hourly.

We can also add a graph to results when these dimensions are selected for number of users and each metric. Users would just show one line per variation and may even be useful for all dimensions not just date. Each metric graph would show value for each variation with standard deviation shading, similar to what we have on metric detail pages.

We currently support two kinds of variation ids - array index and string keys. With the recent change in #97 we no longer need both options and can just use string keys for everyone. This data source option is hard to explain in the UI and docs, so removing it will be good.

TODO:

• Remove from data source form
• Remove from data source info page
• Simplify logic in front-end wherever variationIdFormat was being used
• Simplify logic in back end wherever it was being used
• Testing all edge cases
• Update docs

Metric conversion windows currently limit conversions to the first X hours after being exposed to an experiment. That means, while an experiment is running, some users in the results have had the full X hours to convert and some have had less. This can cause early misleading results, especially if your treatment changes how fast someone converts. For example, it might look like a huge win at first and then flatten out over time. Ignoring the partial data has its own problems - you have to wait X days to see the first results coming in so it can be hard to spot early bugs.

We can expose this trade-off in the UI with a new toggle for experiments to view either partial (default) or complete data. Viewing partial data includes all users, even those who have not had the full conversion window to convert. Viewing complete data only includes users who have had the full conversion window.

We can also add an organization setting to default to a specific value for that toggle.

Option to display the full stats (risk, chance to beat control, uplift) when drilling down by dimension. This can potentially make people come to incorrect conclusions since we don't do any multiple testing corrections. Maybe if number of unique dimension values is below a certain threshold (10?) we default to showing full stats and otherwise we default to limited info with the ability to toggle on full stats if desired.

Add support for removing metrics through an archive functionality. These archived metrics cannot be added to new tests. Care should be taken when editing past experiments so as to not break the metric selector.

• Ability to import a config.yml file and have it create everything in the database
• Setting in config.yml to allow using the db in addition to the yml file for metrics and/or dimensions

The variation id mismatch warning has too many false positives. It should not show in the following cases:

1. The experiment just started and barely has any users yet (e.g. control has 2 users, variation has 0)
2. There's an unknown variation returned from the database that barely has any users (e.g. control has 1000, variation has 1000, something_random has 2). Happens fairly often at scale with front-end event tracking like Segment.
3. Traffic split is extremely uneven (e.g. experiment defines 3 variations, but the current phase has the traffic split 50/50/0)

Also, improve language, especially when using numeric ids. Currently just seeing a list of 0s and 1s is confusing.

From the metric overview or details, add a way to copy a metric to a new metric for users to edit. This supports the cases where users want to add a metric with a similar query without having to re-enter all the details.

Activation metrics are used when the experiment exposure event is logged before the user actually sees the treatment. These delayed treatments are common sources of bugs - for example, an engineer thinking their code only affects a modal, but it actually changes the underlying page too.

We should add a new built-in dimension called Activation Status that is available when an experiment has an activation metric. When selected, there will be two dimension values - Activated and Not Activated. You would expect to see no significant difference between the variations in the "Not Activated" dimension value and if you do it's an indication of a bug.

Currently, self-hosted deployments work the same way as the Cloud, supporting multi-tenant use cases. This means someone could sign up and create a new organization, which is not usually desired.

Changes on self-hosted deployments:

• Disallow creating new organizations once one already exists
• If someone registers a new user account and they have not been invited to the organization, show an error message

Change the list of metrics to be grouped by tags:

• tag1
  • metric1
  • metric3
• tag2
  • metric1
  • metric4
• untagged
  • metric5

Currently, anything above 30 or so metrics is really hard to manage. This extra layer can scale to a thousand metrics. Plus, we can later add support for namespaced tags (e.g. ecommerce.funnel) to further increase nesting and scalability.

Changes:

• New Metric list page
• New <MetricSelector> component with a 2-step process - select the tag, then select the metric

_{Sent from PPHub}

When an API key exists, purge the cdn whenever the contents of the experiment API endpoint changes. Support CloudFront purging via the aws SDK and fastly-style purges via HTTP purge requests.

• Allow customizing the position on screen
• Make it prettier
• Persist the forced variations across page loads (and give a way to "unforce" variations and go back to the default)
• Visual indicator of which experiments are being forced, which ones the current user is excluded from, etc.
• Allow changing the user attributes and seeing how it affects which experiments the person is included in
• Allow viewing attributes of the experiments (weights, coverage, targeting, etc.)

Allow organizations to "claim" email domains. When this happens, anyone who creates an account with a matching email address, will be automatically added to the organization without requiring an invite. Initially, only enable this feature for self-hosted deployments.

Add an optional Visual Editor like those used by Google Optimize, Optimizely, and others.

Features / Requirements:

• Can be enabled/disabled on settings
• Allow point & click editing of HTML (using https://github.com/growthbook/ab-designer)
• Allow adding custom javascript for variations
• Expose a public javascript endpoint in the API that returns a compiled bundle of all experiments (with cache headers)
• Make sure experiment code is valid javascript before saving
• Exclude visual experiments from overrides API endpoint

On the data source detail page, we show the experiment and pageview queries, but there is no easy way to run them, verify they are correct, and debug any errors.

A few things to keep in mind when doing this:

• We should add a small limit to the queries to avoid fetching a lot of data
• There should be no where clause or sort for best performance
• We should make sure to obfuscate the user_id column in the query just in case someone uses email or other PII for it. For example, SELECT MD5(user_id) as user_id. Maybe this obfuscation can be opt out per datasource.
• We can show errors if any required columns are missing
• We could detect any additional columns and suggest adding them as dimensions
• Companies that have never run an A/B test before may not have the experiments table yet or it may be empty and that's ok.
• The pageview query does not apply to all companies

Add a delete button to the "History" tab of experiments.

When deleting a phase

1. Remove from the phases array and save the experiment
2. Bulk delete all snapshots for the phase
3. Bulk update all snapshots for later phases and decrement the phase index by one
4. Add an audit entry

If there's only one phase left and the experiment is running, revert the status back to draft.

Flutter is a fast-growing ecosystem. I find that there will be lots of merit to help grow Growthbook if we have a Flutter Client Side Library!

role was left out of the snowflake integration.

Hello Growthbook Crew 👋🏼

We're really interested in trying Growthbook to replace our experiments framework and I've been trying to get a test experiment working.

I've completed the data source setup + a few metrics which appear to be working fine however when I run an experiment I get a blank output:

I had a look at the underlying queries and they seem to be working fine:

Query 1

Query 2

Any idea what could be going on here?

We currently allow viewing the raw SQL queries on experiment results. We should also be able to generate a Jupyter Notebook on the fly that recreates the experiment results. This will let data teams easily do further analysis on an experiment when required.

For configuration, each data source will need to define a Python code cell with a runQuery function. For example:

def runQuery(sql):
  # TODO: connect to database, run the query, return list of rows
  return [];

Then we can generate a notebook with the following cells

1. Markdown field with the experiment name, hypothesis, link to GrowthBook results, etc.
2. Python imports (numpy, scipy, the GrowthBook stats engine, and a plotting library if needed)
3. The runQuery definition for the datasource
4. Run SQL queries (number of users plus one for each metric)
5. Clean up the raw SQL rows and get the data ready for the stats engine
6. Call the stats engine for each metric
7. Post-process the stats results
8. Display any warnings (e.g. SRM)
9. Render the results output as a table

We are evaluating GrowthBook right now, and it looks fantastic.

It would be great to see support for image storage in systems such as Google Storage or S3 to reduce the amount of infrastructure needed to support GrowthBook.

Maybe this could be via a plugin system to allow multiple different options.

Require email addresses to be verified before using GrowthBook (self-hosted)

For self-hosted deployments:

• Only require by default when an email SMTP server is configured

• Allow an admin of the organization to manually verify users (for cases when the email fails to send)

• Update user model to add verification status, date, hash fields for verification messages.

• Upon register, set user state to pending, but only if SMPT enabled. Update auth rules to route users to "pending / resend email" page.

• New email template with verification url + hash.

• Option to resend email.

• Verification hashes timeout after 30 days.

• Team settings page shows pending users, allows manual verification.

• Successful verification sends "welcome" email.

• Ensure first user registration verifies instantly (probably already works this way)

• Optional: email validation (FE: simple, BE: additional MX validation)

When we fetch experiment results, the variation_id from the data source is returned. We then do a mapping in code to connect those variation ids to the ones defined in Growth Book. If there's a mismatch, it currently fails silently. We should detect and warn about both of these cases:

1. The data source returns a variation_id that Growth Book doesn't know about
2. Growth Book has a variation_id that the data source isn't returning

Should help debug issues like #61

For SQL data sources, add an SSH Tunnel option. Useful for connecting to data sources behind a firewall when opening a port is not an option.

Currently, we assume experiments use either userId or anonymousId to split traffic and there are isAnon branches throughout the code. This approach doesn't scale to more types of randomization units, so need to rethink the structure of things.

Generally speaking, randomization units compared to unique users can be:

1. one-to-one (e.g. userId)
2. one-to-many (e.g. companyId, teamId, geoRegion)
3. many-to-one (e.g. deviceId, sessionId, profileId, childId)
4. many-to-many (e.g. pageTemplateId, url)

Some of these are a little fuzzy and depending on your situation, you may move some of these around (e.g. if your users move locations frequently, you may want to consider geoRegion many-to-many instead of one-to-many).

one-to-many ids are easy to deal with. Metrics can be calculated with userId and then rolled up to the randomization unit.

many-to-one and many-to-many are harder to deal with. You either need an entirely separate set of metrics for each one or complex attribution logic to determine which id gets "credit" for a user conversion.

Currently the metric conversion window is hard-coded at 3 days. This default should be configurable with an environment variable and metrics should be able to override the default.

Metrics like clicking a button should happen within 1 day max of a user being put in an experiment. Others like onboarding may take a week or two.

You can't type 1.03 in a number input. It will end up as 1.3 instead because while typing, 1.0 has a trailing zero and it is truncated.

This is a fundamental limitation of our custom useForm hook. We should switch to https://github.com/react-hook-form/react-hook-form instead, which has way more features, better performance, and is actively maintained

Support SSO for self-hosted deployments (Google, Okta, Auth0, etc.). This will be our first Enterprise feature that is not covered under the MIT license

Changes

• New packages/enterprise directory with non-open source license (plus change root level LICENSE to reference this)
• Enterprise license key support
• New environment variable AUTH_METHOD which defaults to local, but can be set to saml or google instead.
• New environment variables for Google auth (GOOGLE_OAUTH_KEY and GOOGLE_OAUTH_SECRET)
• New environment variables for SAML auth (SAML_ENTITY_ID, SAML_ACS_URL, SAML_X509_CERT, SAML_ATTR_EMAIL, and SAML_ATTR_NAME)
• Redirect to IdP instead of showing local auth form
• Endpoint to process Google auth response
• Endpoint to process SAML auth response
• Remove "edit profile" and "change password" functionality when using SSO
• Hook up logout button

Nothing happens when trying to uploading screenshots (see screenshot below). Works fine in 0.4.0, but not in 0.5.0.

By default, we include everyone who has an experiment exposure event in the database in the analysis.

We currently allow specifying an Activation Metric to limit analysis to users who perform a certain action first. For example, if exposure is logged on page view, but only visible inside a modal, you could specify "Viewed Modal" as the activation metric.

There are other cases when you want to filter out specific users from an analysis, but it's not based on them performing an action. For example if you log exposure on page load for everyone, but the change only affects mobile or premium users.

In some cases, segments could work. You would pick a specific segment for an experiment and only those users would be included in the analysis. This works well for things like "premium users" where it's potentially reusable across many experiments and you can easily get a list of user_ids.

For other cases, it might be better to specify a WHERE condition for the experiment query. This works better for experiment-specific conditions or cases where you can't reliably get a direct list of user_ids. For example:

-- factor in experiment dimensions
device = "mobile"
-- and/or use arbitrary subqueries
AND user_id NOT IN (
  SELECT user_id FROM experiment_viewed
  WHERE experiment_id = "some_conflicting_experiment"
)

Currently hardcoded as '>5'. Change to '>=' and allow changing the number via config.yml, an organization setting, and/or an environment variable.

Ability to specify one or more "projects" for an organization. Each project can have its own set of metrics and experiments. Users can be part of one or more projects.