We need to have a transparent data retention policy described in this repo and implemented in our OPS infrastructure.

I propose: Raw data retention period: 6 months

The ProfileFetcher code doesn't include a client_id attribute in the returned JSON blob. This triggers a bug in the new logging we've added at

The simple fix is to just stuff the client_id into the ProfileFetcher output as this won't affect the recommenders at all anyway.

We are reworking the addon recommendations where some addons which need to be promoted regardless of whatever recommendations TAAR returns.

We'll need to sync with amo team to determine how we want this to be implemented.

Our requirements are that we must be able to definitively know the set of addons displayed to the user so that we can measure TAAR efficacy.

TAAR has a hard dependency on AWS's DynamoDB service for storing client data. We need to migrate to GCP's BigTable to do a full migration from Amazon to Google.

It looks like we have a bug here. The last with block should be out of the if not os.path.exists... block

The JSON data backing the recommenders needs to expire on a daily basis and refresh automatically.

Normalize logging format so that the guids surfaced in a successful recommendation are similarly available for both the ensemble and linear versions of taar.

[Affected versions]:

• Firefox Beta 61.0b2

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a new Firefox profile with the latest version of the "TAAR v3" add-on installed (downloaded from https://bugzilla.mozilla.org/show_bug.cgi?id=1469546).
• Create the following prefs in the "about:config" page:
  "shieldStudy.logLevel" as string and set it to "All";
  "extensions.taarexpv3_shield_mozilla_org.test.variationName" as string and set it to "control", "intervention-a" or "intervention-b"

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to three websites and observe the behavior.

[Expected result]:

• The TAAR pop-up is not triggered.

[Actual result]:

• The TAAR pop-up is triggered.

[Notes]:

• This is also reproducible with profiles older than 30 days.
• Attached a screen recording of the issue:

The telemetry ID is being deprecated in favor of a sha256 hashed ID.

Rename variables from client_id toclient_hash in taar for clarity.

Update all ETL jobs in python_mozetl related to TAAR to compute and use hashes instead of raw telemetry client ID.

So that we can check version in code.

Should be reworked to use clients daily

Need to add a testcase to show that the CollaborativeRecommender is properly recommending an add-on based on the shortest distance to the ideal model.

We would like to access the specific addons recommended per user regardless of what module was triggered. This will be used for downstream sanity checking against telemetry data sources as well as for online tampering detection as requested in the RRA.

The main idea is to use a cache object passed to the RecommendationManager instead of reinventing the wheel. This object will be passed by taar api and it will likely be powered by the django cache system.

We need to land a patch into to log events detailing :

• hashed telemetry id
• which addons were recommended

The intent is to be able to compare the main telemetry data which shows all addon installations vs the add-ons which we presents to a particular Firefox instance.

We need to be able to run analytics to compare organic vs suggested installations. Note that this depends on #117

The recommender manager should not create instances of its recommenders until it's necessary. Some of the recommenders may have expensive code in their constructors and we can avoid running that code when either the user_info is not available or when the first recommender can recommend.

We should use something like Colander to do minimal data validation of the inbound JSON data to the recommendation engines.

As we add more recommendation engines and tune the input parameters, the test cases will fall out of sync with real-world data.

For addons that are present in multiple candidate donrs, the LRs from each donor shoudl be multiplyied togetehr for a more refined addon ranking.

the similarity function used in this recommender should be encapsulated in a function that can be imported by the corresponsing etl job to ensure consistency between model generation and recommendations to the client.

Discovery pane exposes some minimal client information in the form:
"https://discovery.addons.mozilla.org/%LOCALE%/firefox/discovery/pane/%VERSION%/%OS%/%COMPATIBILITY_MODE%"

The locale can be grabbed from this url and pass it to TAAR as an additional parameter along with the client_id so that TAAR can make locale-based recommendation even for users who are not yet present in the Hbase instance.

We don't support legacy addons at all anymore - disable this in the ensemble.

We do still need it in the linear recommender though.

Implement pareto hybridization in taar as per: https://dl.acm.org/citation.cfm?id=2365962

The DiscoPane admin tool has the ability to define promoted addons. We need to able to instrument TAAR to surface those addons.

See also: Disco-TAAR PRD

The fullstack call time for TAAR seems to have gone up substantially - or at least it's much more variable comparing TAARv2 and TAARv3

Screenshots below from New Relic show very consistent performance ~70 ms for TAARv2 and variation between 60ms to 120ms for TAARv3.

I suspect this is the probably because of the way we're generating the random sample of add-on suggestions from the whitelist. We currently randomly shuffle the whitelist and pop items off the end which isn't the smartest thing in the world.

Performance profiling will give us definitive answers to this but as a first attempt, we should probably just store an index into the shuffled whitelist array and just treat the array as a circular buffer and read items off as random add-on suggestions.

More importantly, we should try to get an automated performance benchmark where we can spin up webheads in the dev enviroment and throw a load test and snapshot the metrics out of New Relic so we don't have surprise regressions in performance.

(taar) {13:33}~/Dev/mozilla/taar:crankycoder-features/issue-65 ✓ ➭ python setup.py test
zip_safe flag not set; analyzing archive contents...

Installed /home/bird/Dev/mozilla/taar/.eggs/pytest_runner-4.0-py2.7.egg
Searching for setuptools_scm
Reading https://pypi.python.org/simple/setuptools_scm/
Downloading https://pypi.python.org/packages/76/6e/3ba3d9c3409af4375b6c4512f78423381e9ed99375e7791586189207ea6a/setuptools_scm-1.15.7-py2.7.egg#md5=4385ebde52ea23f85fb490acb902095e
Best match: setuptools-scm 1.15.7
Processing setuptools_scm-1.15.7-py2.7.egg
Moving setuptools_scm-1.15.7-py2.7.egg to /home/bird/Dev/mozilla/taar/.eggs

Installed /home/bird/Dev/mozilla/taar/.eggs/setuptools_scm-1.15.7-py2.7.egg
running pytest
Traceback (most recent call last):
  File "setup.py", line 36, in <module>
    zip_safe=False,
  File "/home/bird/miniconda3/envs/taar/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/bird/miniconda3/envs/taar/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/bird/miniconda3/envs/taar/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "build/bdist.linux-x86_64/egg/ptr.py", line 157, in run
  File "build/bdist.linux-x86_64/egg/ptr.py", line 118, in install_dists
AttributeError: class test has no attribute 'install_dists'

This was run on py2.7

Push the wsgi app instance into the root context. Each of the recommenders can register a URL per LazyJSONLoader so that we can determine when the JSON was last loaded from S3.

recent test added uses a const for mock data and one test updates it which means that it will be changed for other tests.

using a fixture would give fresh data each test to manipulate at will

We need a load testing tool to perform baseline benchmarks against the TAAR service.

For now we can use https://molotov.readthedocs.io/en/stable/ to run HTTP load tests against our service using a predefined set of hashed telemetry IDs.

We can use the options dictionary to override addon recommendations. Need to document something for amo team to use.

Since we've changed the row key to just be the client id we can use 'get' instead [1]. Based on some quick research get's may perform better than scan.

[1] https://github.com/mozilla/taar/blob/master/taar/hbase_client.py#L53

The off-line recommender jobs produce JSON blobs that get consumed by TAAR. These JSON files are refreshed/updated weekly.

We should make sure that, if a file is downloaded and cached by TAAR and a new version is available, the refreshed model is used.

Blocked by #121

We need to generate weights for ensemble recommender that also use the taar-lite recommender.

#103 requires that a new JSON file with the whitelist of curated addons is stored within S3.

This is a sample log line that is produced by https://github.com/mozilla/taar/blob/master/taar/recommenders/recommendation_manager.py#L52

{"EnvVersion":"2.0","Severity":6,"Fields":{"message":"Recommender selected","recommender":"<taar.recommenders.collaborative_recommender.CollaborativeRecommender object at 0x7f98287cbc18>","client_id":"<snip>"},"Hostname":"ip-172-31-34-129","Pid":19,"Timestamp":"2017-09-27 19:52:23","Logger":"taar_api","Type":"taar.recommenders.recommendation_manager"}

It would be nice if recommender field only returned the class name rather then the full object info. It should make querying the data easier.

The longitudinal dataset is being retired by the end of 2019, we'll need to migrate to whatever the datapipeline team has setup as a replacement.

Lets keep going all in on the make file and add commands for setting up environment

This started happening sometime in the last week or so, here's an example log:

https://circleci.com/gh/mozilla/python_mozetl/70#tests/containers/4

This may be related to a pyspark upgrade:

mozilla/python_mozetl@0b97226

Note that upgrade actually just keeps the version we're using for testing in sync with the one we're using in production, so we really want to fix the root cause here rather than just revert versions.

For now I've filed a PR to temporarily disable the failing tests:

mozilla/python_mozetl#253

Make sure that appropriate location is updated for all generator jobs when PR against mozilla/python_mozetl lands. README.md currently points to my fork so it is not a dead link.

Also logged in Bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1471216

[Affected versions]:

• Firefox Beta (61.0) locales other than en-Us, en-GB, de, fr, ru, pt-BR, es-ES, pl, it, ar, ja.

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a new Firefox profile with the latest version of the "TAAR v3" add-on (downloaded from https://bugzilla.mozilla.org/show_bug.cgi?id=1469546) installed.
• Create the following prefs in the "about:config" page:
  • "shieldStudy.logLevel" as string and set it to "All";
  • "extensions.taarexpv3_shield_mozilla_org.test.variationName" as string and set it to "control", "intervention-a" or "intervention-b"

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to three websites and observe the behavior.

[Expected result]:

• The websites are correctly opened.

[Actual result]:

• A "File not found" doorhanger is opened on the third website.

[Notes]:

• Also, the shield study's pings are displayed in the Browser Console and on the "about:telemetry" page.
• Attached a screen recording of the issue:
  

We're going to add weights to the recommender results.

@mlopatka will take the SimliiarityRecommender, and @crankycoder will take the other recommender classes.

The Ensemble Recommender has a unique behavior that does not exist in the existing Collaborative, Legacy, Locale and Similarity recommenders.

Ensemble recommender can recommend addon GUIDs that are already installed by the user.

We need to filter out GUIDs just prior to applying the limit to constrain the number of recommendations.

We should really document the sequence of commands needed in order to run the tests locally, from installing the dependencies and running the tests.

In #32 we found that the similarity recommender takes about half a second to make a recommendation. Further investigating this we found that the culprit is this.

The branches should now be labelled 'intervention-a', 'intervention-b' and 'control'.

See: https://docs.google.com/document/d/1r8RC8BBjS9DQSBfanlD-MlCS4XWN3-bozyWOIFlD8aA/

Intervention A is the ensemble method with a constrained whitelist used.

Intervention B is a hybrid between editorial curated selections and the machine recommendations

Control group is the third branch.

See comments on PR #66.

An integration test has been added but is skipped.

We want to be able to use taar-lite as a recommendation engine within TAAR's ensemble recommender.

This means:

• adapt taar-lite to be an ETL job to generate recommendation graphs
• generate a list of recommendation lists for each of the addons that are installed by a particular client

Merging the list of lists by weight should be relatively straightforward. Merging duplicate recommendations should more heavily weight those particular GUIDs but we have no predefined merge method yet.

mozilla/python_mozetl#134

Somehow there are null values in the donor_client strings returned here that may end up in the candidate donor pool. This needs to be prevented or filtered after the fact.

The 1.0 tuple should be the frequency of that addon in that locale divided by the total number of clients in that locale. this should allow the ensemble method to be even more effective at deriving a good stacking weight.