mozilla-services / absearch Goto Github PK

We will need to ask Ops to assist in configuring nginx to...

to drop all non GET requests
to respect Cache-Control header
to cache on GET path without query parameters

CircleCI builds are failing due to inaccess:

git version 2.30.2
Cloning git repository
Cloning into '.'...
Warning: Permanently added the ECDSA host key for IP address '140.82.112.4' to the list of known hosts.
Load key "/home/circleci/.ssh/id_rsa": invalid format
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

exit status 128```

we should provide a way to reset redis on stage after load tests

• Create a release tag
• change config values for new changes
• File an ops request for nginx changes? ref #199

Please update this service to work with the Dockerflow standard.

It's currently broken and returning 500's

https://github.com/mozilla-services/absearch/blob/master/absearch/server.py#L259

Part of the Long Term Support Plan

Python2 has been deprecated since jan 2020, it may be worthwhile to change ABsearch to python3

Site https://search.stage.mozaws.net has failed the web security baseline scan.

The failing tests are:

X-Frame-Options Header Not Set [10020] x 1

• https://search.stage.mozaws.net/__api__ (200 OK)

X-Content-Type-Options Header Missing [10021] x 2

• https://search.stage.mozaws.net/__api__ (200 OK)
• https://search.stage.mozaws.net/ (200 OK)

Strict-Transport-Security Header Not Set [10035] x 2

• https://search.stage.mozaws.net/robots.txt (404 Not Found)
• https://search.stage.mozaws.net/sitemap.xml (404 Not Found)

Content Security Policy (CSP) Header Not Set [10038] x 3

• https://search.stage.mozaws.net/__api__ (200 OK)
• https://search.stage.mozaws.net/robots.txt (404 Not Found)
• https://search.stage.mozaws.net/sitemap.xml (404 Not Found)

This issue was automatically raised.

This issue is managed automatically by the baseline scan:

• If the failing tests change then it will be updated
• If it is closed before the tests pass then a new one will be opened
• When all of the tests pass then it will be closed

Full details, including how to test for these issues locally, can be found on this Security Baseline Service dashboard.
If you have any questions or concerns please get in contact with @psiinon

This bug #11 can be caught during a load test against stage. We should make sure that the distribution we want in the client is correct.

The production branch on this repository is not protected against force pushes. This setting is recommended as part of Mozilla's Guidelines for a Sensitive Repository.

Anyone with admin permissions for this repository can correct the setting using this URL.

If you have any questions, or believe this issue was opened in error, please contact us and mention SOGH0001 and this repository.

Thank you for your prompt attention to this issue.
--Firefox Security Operations team

404 pages currently send ~800bytes of HTML, update 404 to return empty body

Part of the Long Term Support Plan

1. Install a local Redis (subprocess.Popen(['redis-server', '--port', '7777'], ...)
2. Install tox
3. ?

I'm having errors like this when running tests:

  File "absearch/.tox/py27/local/lib/python2.7/site-packages/boto/auth.py", line 989, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials

By default, the Bottle server sets its "quiet mode' to False, which means it logs every single request it handles to the console, which causes a lot disk IO and disk spaces. Besides the Nginx server on the instance already has an access log that logs the same thing Bottle does.

Can we make it optional to run Bottle with quiet mode being True?

What I'm proposing is to add quiet=abconf['quiet'] to https://github.com/mozilla-services/absearch/blob/master/absearch/server.py#L236 so the "quiet mode" can be enabled/disabled via a conf file.

API reference: http://bottlepy.org/docs/stable/api.html

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

We should check security updates for the list of dependencies

Linux builds sometimes end up with locale codes like fr-FR instead of fr. Rather than bloat the config, I'd like us to have a fallback where if ab-CD isn't found, we look for ab.

Not urgent, but we'd want it before we ship.

Site https://search.services.mozilla.com has failed the web security baseline scan.

The failing tests are:

X-Frame-Options Header Not Set [10020] x 1

• https://search.services.mozilla.com/__api__ (200 OK)

X-Content-Type-Options Header Missing [10021] x 2

• https://search.services.mozilla.com/__api__ (200 OK)
• https://search.services.mozilla.com/ (200 OK)

Strict-Transport-Security Header Not Set [10035] x 2

• https://search.services.mozilla.com/robots.txt (404 Not Found)
• https://search.services.mozilla.com/sitemap.xml (404 Not Found)

Content Security Policy (CSP) Header Not Set [10038] x 3

• https://search.services.mozilla.com/__api__ (200 OK)
• https://search.services.mozilla.com/robots.txt (404 Not Found)
• https://search.services.mozilla.com/sitemap.xml (404 Not Found)

This issue was automatically raised.

This issue is managed automatically by the baseline scan:

• If the failing tests change then it will be updated
• If it is closed before the tests pass then a new one will be opened
• When all of the tests pass then it will be closed

Full details, including how to test for these issues locally, can be found on this Security Baseline Service dashboard.
If you have any questions or concerns please get in contact with @psiinon

We have the config hashes in __info__, but nothing to verify that the version of code deployed (to either stage or prod) actually corresponds to a release.

Suggested output: {"hash":"d76b982484e8475567dfea674ecc3a2c3c7b7bf0", "tag":"0.2.0"}

This will require removing boto from requirements.txt and all usage of it throughout the source code

Part of the Long Term Support Plan

One of the steps we do to build absearch package is to run python setup.py install command [1], and it is now failing with the following error:

Installed /tmp/absearch/lib/python2.7/site-packages/argparse-1.4.0-py2.7.egg
Searching for configparser
Reading https://pypi.python.org/simple/configparser/
Best match: configparser 3.7.3
Downloading https://files.pythonhosted.org/packages/4a/4d/5d4c07cd28476ecad84ea5ad43961e50b6fd74cd24b9b81113650b4de6ee/configparser-3.7.3.tar.gz#sha256=27594cf4fc279f321974061ac69164aaebd2749af962ac8686b20503ac0bcf2d
Processing configparser-3.7.3.tar.gz
Writing /tmp/easy_install-ILnxBE/configparser-3.7.3/setup.cfg
Running configparser-3.7.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ILnxBE/configparser-3.7.3/egg-dist-tmp-PQgfLe
zip_safe flag not set; analyzing archive contents...
backports.__init__: module references __path__
Adding UNKNOWN 0.0.0 to easy-install.pth file

Installed /tmp/absearch/lib/python2.7/site-packages/UNKNOWN-0.0.0-py2.7.egg
error: Could not find required distribution configparser

AFAICT, our last push, thus successful build, happened on 6/27/2018, at that time, the configparser version it used was 3.5.0.

Can you please see what we can do to fix it?

We need to stop forwarding these metrics, or send them to a dedicated Graphite host like location does. The volume is overwhelming our metrics service.

The app will still work with the existing config but we should poke ops since a new version won't be loaded.

https://github.com/mozilla-services/absearch/blob/master/absearch/settings.py#L83

Dean, Bob, what should we add there ? thx

useful to know which version of the data has been deployed. can be a md5 hash of the data file

absearch.redis.get and absearch.redis.incr never show up even if redis is populated

The README will need to be updated to reflect the edits made from long term support changes

Should look into merging scripts :

mozilla-services/absearchdata into mozilla-services/cloudops-infra/tree/master/projects/absearch

Merge verifications scrips from mozilla-services/absearchdata into the main absearch repo.

I have no idea what the right thing to do is here, but it's becoming an issue as we do things with absearch that it wasn't meant to do. If you have something like this:

          "apr18-1": {
            "settings": {
              "searchDefault": "Google",
              "visibleDefaultEngines": ["amazondotcom", "bing", "google-2018", "twitter", "wikipedia", "ddg"]
            },
            "filters": {
              "sampleRate": 100,
              "products": ["firefox"],
              "channels": ["esr"],
              "minVersion": 52.6
            },
            "interval": 86400
          },
          "apr18-2": {
            "settings": {
              "searchDefault": "Google",
              "visibleDefaultEngines": ["amazondotcom", "bing", "ebay", "google-2018", "twitter", "wikipedia", "ddg"]
            },
            "filters": {
              "sampleRate": 100,
              "products": ["firefox"],
              "channels": ["esr"],
              "minVersion": 60
            },
            "interval": 86400
          }
        }

The first cohort is always picked even if you specify a version of 60. I would expect the test code to be smart enough to pick the cohort based on the minVersion and not just go down the first path.

Changing the order in the file has no effect.

We need the ability to say "always use this engine set for 60+" but it's not possible right now.

A testing tool of some sort to make sure responses of API remains the same after all changes are made

This will require removing datadog from requirements.txt and all usage of it throughout the source code

Part of the Long Term Support Plan

Add a Cache-Control header in responses from the app that defaults to 300 seconds

Part of the Long Term Support Plan

We're increasingly running into problems where we need to have an absearch config stop being used at a certain version.

Issue #31 was one way to do this, but we think an easier way is to just support maxVersion for cohorts.

nginx logs + redis

This Mozilla repository has been identified as lacking a license. Consistent with Mozilla's Licensing Policy an open source license should be applied to the code in this repository.

Please add an appropriate LICENSE.md file to the root directory of the project. In general, Mozilla's licensing policies are as follows:

• Client-side products created by Mozilla employees or contributors should use the Mozilla Public License, Version 2.0 (MPL).

• Server-side products or utilities that support Mozilla products may use either the MPL or the Apache License 2.0 (Apache 2.0).

In special cases, another license might be appropriate. If the repository is a fork of another repository it must apply the license of the original. Similarly, another license might be appropriate to match that of a broader project (for example Rust crates that Firefox depends on are published under an Apache 2.0 / MIT dual license, as that is the dual license used by the Rust programming language and projects).

Please ensure that any license added to the LICENSE.md file matches other licensing information in the repository (for example, it should match any license indicated in a setup.py or package.json file).

Mozilla staff can access more information in our Software Licensing Runbook – search for “Licensing Runbook” in Confluence to find it.

If you have any questions you can contact Daniel Nazer who can be reached at dnazer on Mozilla email or Slack.

OPENLIC-2023-01

This will require removing redis from requirements.txt and all usage of it throughout the source code

Part of the Long Term Support Plan

We can drastically improve performances by caching the results - one option is to use nginx for this see mozilla-services/cliquet#401 - and add a few headers in the server.

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

• Still needs to be comfirmed with ops
  This will require removing statsd from requirements.txt and all usage of it throughout the source code

Part of the Long Term Support Plan

In this line : https://github.com/mozilla-services/absearch/blob/master/absearch/settings.py#L262

we're using 'cohort' instead of 'picked' to increment the counter. That means we're incrementing the last test cohort counter from the list instead of the one picked.

The effect of this bug are that the tests cohorts counters are filled and no (or very few clients) gets them.

The corresponding tests were verifying distributions, not counters - so everything looked right