Git Product home page Git Product logo

matched_markets's Introduction

A Time-Based Regression Matched Markets Approach for Designing Geo Experiments

Copyright (C) 2020 Google LLC. License: Apache 2.0

Disclaimer

This is not an officially supported Google product. For research purposes only.

Description

Randomized experiments (Vaver & Koehler, 2010) have been recognized as the gold standard to measure the causal effect of advertising, but how to design and analyze them properly is a non-trivial statistical problem. Geo experiments have the advantage of being privacy safe. However, in geo experiments, it may not always be possible to rely on randomization to create balanced groups due to the geo heterogeneity, the small number of units, and/or constraints that the advertisers want to impose, such as including certain geos in a specific experimental group. Hence, Au (2018) introduced a greedy search algorithm to find the best experimental groups among those that satisfy:

  • the constraints imposed by the advertiser (budget, geo assignments, ...)
  • the assumptions of Time Based Regression (Kerman, 2017)

This directory contains

  • Python package for geo experiment design using Time Based Regression and Matched Markets.

  • Python package for geo experiment post analysis using Time Based Regression.

  • Colab demos for design and post analysis, separately.

Installation

Usage

For Python programming, here is an example usage.

import itertools
import numpy as np
import pandas as pd
from matched_markets.methodology.tbrmmdata import TBRMMData
from matched_markets.methodology.tbrmatchedmarkets import TBRMatchedMarkets
from matched_markets.methodology.tbrmmdiagnostics import TBRMMDiagnostics
from matched_markets.methodology.tbrmmdesignparameters import TBRMMDesignParameters

n_geos = 5
n_days = 21
geos = {str(geo) for geo in range(n_geos)}
dates = pd.date_range('2020-03-01', periods=n_days)
df_data = [{'date': date, 'geo': geo} for geo, date in
           itertools.product(geos, dates)]
df = pd.DataFrame(df_data)
response_column = 'sales'

# Create sales data.
def day_geo_sales(geo, n_days):
  # Larger geos have different means and variances.
  return [
      100 * geo + 10 * geo * day + day + np.random.randint(10)
      for day in range(n_days)
  ]

df[response_column] = 0.0
for geo in geos:
  sales_time_series = day_geo_sales(int(geo), n_days)
  df.loc[df.geo == geo, response_column] = sales_time_series

parameters = TBRMMDesignParameters(n_test=14, iroas=3.0,
                                   budget_range=(0.1, 300000))
data = TBRMMData(df, response_column)

mm = TBRMatchedMarkets(data, parameters)
designs = mm.greedy_search()

Without programming, the best way to learn how to use the package is probably by following one of the notebooks, and the recommended way of opening them is Google Colaboratory.

Contribution

References

Available at Matched Markets paper. TBR paper. GeoX paper

matched_markets's People

Contributors

marcolongfils avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

matched_markets's Issues

Both Colab Notebooks Have Error with `Select Design Cell`

I believe this is an error that rose from the recent update on April 5/6. Attached is a screenshot of the error when running a notebook example.

The issue comes from the "selected design" cell and is dealing with these lines that seemed to have been added:

# these are numerical identifier used to denote the two groups
group_treatment = GeoAssignment.TREATMENT
group_control = GeoAssignment.CONTROL
group_excluded = GeoAssignment.EXCLUDED

If I'm not mistaken this is due to the GeoAssignment class not being imported from common_classes, but I'm not sure.

image

TypeError in step 3

in step 3: Summary of the possible designs, if/when there are no suitable designs (e.g. if budget too low, time period too long, etc.. ) the code returns a type error. Perhaps a more descriptive error message would be helpful.

TypeError Traceback (most recent call last)
in
6 matched_designs = MMclass.exhaustive_search()
7 else:
----> 8 matched_designs = MMclass.greedy_search()
9
10 if len(matched_designs) == 0:

1 frames
/usr/local/lib/python3.8/dist-packages/matched_markets/methodology/tbrmmscore.py in score(self)
87 if self._score is None:
88 self._score = Scoring(
---> 89 int(self.diag.corr_test), int(self.diag.aatest.test_ok),
90 int(self.diag.bbtest.test_ok), int(self.diag.dwtest.test_ok),
91 round(self.diag.corr, 2), 1 / self.diag.required_impact)

TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Experimental Plan [ Real-world implementation problems ]

Dear developers,

I hope this message finds you well. I am writing to express my gratitude for the matched_markets library and raise some concerns about its practical implementation.

Firstly, I would like to inquire about budget allocation in the experimental plan. Once we have the experimental design ready, the marketing specialist may wonder how to allocate the budget. For example, suppose there are three treatment geos and a budget of $3k for an experiment duration of four weeks. Should the budget be allocated proportionally, such as $1,000 for each geo and $250/week, or should it be allocated depending on historical budget or geo market size? Any guidance or best practices on this matter would be highly appreciated.

Secondly, I would like to address the issue of budget and time constraints. The marketing specialist may not have any budget constraints, and therefore, may not know the recommended budget for measuring a particular marketing channel/strategy. Is there a way to provide an optimal budget for a particular experimental design? The same applies to the duration of the experiment, where the rule of thumb is typically one sales life cycle. Any insights or advice on these matters would be valuable.

Thank you for your time and efforts in developing the matched_markets library. I look forward to hearing your feedback and suggestions.

Best regards,
Albert

Getting error when trying to install the package

Hi,
I tried to install the package using pip install git+git://github.com/google/matched_markets.git but I am getting the following error:

error: could not create 'build': File exists

Do you know what might be the cause?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.