Hi,
Don't know where to put this: The "move" action does not work anymore when upgrading Thunderbird from v45 to v52: the target folder is empty. This problem disappears when switching back to v45-8-0.
See screen dump.

The ElasticSearch upgrade to V2 broke a number of tests.

Currently all search related tests are marked as search_tests and get skipped in the test runs with -a '!search_tests'.

This issue tracks the update / re-write of those tests to work with current infra.

See also #2931

We want to 12factorize Kitsune. Some progress has been made in the docker branch already on which we should build upon and complete this.

Prior to a full re-design and ideally before FF 57 the header and footer should be updated with the latest and greatest mozilla branding.

In settings.py there's a TokenLogin backend in addition to the Model Backend for authetication. It appears that since bug 921480 are not using this in production. According to the bug this was somehow related to Persona Authentication.

Since this is not in use in production, we don't need to have the code laying around. I'll remove from default settings and this issue tracks it's removal from code.

The SCL3 datacenter and the AWS/K8S codebases will diverge to reduce conflicts and increase development speed.

It's decided to copy master to legacy and keep the later for SCL3 based deployments. The master branch will remain the active development branch, targeting AWS/K8S.

• Create legacy branch
• Update README with details on the current branch status.

Create S3 buckets for sumo user media in Oregon

• No versioning

Suggested names:

• sumo-user-media-stage
• sumo-user-media-prod

Part of #2899

We need an ElasticSearch server for dev

• SCL3 version 2.4.5
• Plugins
  • Polish analyzer https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-stempel.html

All the data can get rebuilt from the database.

The dev instance deployed at http://sumo-dev.frankfurt.moz.works does not redirect correctly.

STR:

• Go to http://sumo-dev.frankfurt.moz.works
• Page loads
• Click on 'Ask a question' (or this link http://sumo-dev.frankfurt.moz.works/en-US/questions/new)
• Redirect error

This should be related to mozmeao/infra#540

My preference would be to have mozmeao/infra#540 fixed which will benefit all apps (and also has to be done for the new portland k8s cluster) but I'm open to other suggestions to unblock this.

/cc @pmac @jgmize @metadave

This will help consolidate billing and monitoring, as well as potentially help address or debug some email issues we've seen in the past.

Page for get-involved would look like:
Help people on twitter and Facebook

Step 1: Link to Guidelines
Step 2: Sign up process - sumo account -> redirect to form created (keeping the twitter account and sumo account from army of awesome is ideal here)
Step 3: Link to Training article (includes tweetdeck, filters, contact info and moderators)

Links on other pages would remain the same
Dashboards would be removed.

Content in progress:
https://bugzilla.mozilla.org/show_bug.cgi?id=1385135

Add whitenoise to serve static files in the dev (and stage?) instances.

In production we'll probably go with another more efficient approach which is being investigated in #2949.

Whitenoise activation should be configurable via the ENABLE_WHITENOISE env variable.

django-axes needs to now the client ip. Now it's configurated to HTTP_X_CLUSTER_CLIENT_IP.

Does this header work in k8s?

Parking lot for issues I am having getting the install running on mac.

docker-compose Version Not Supported

3.4 does not appear to be supported by the version of docker I have installed (Version 17.09.0-ce-mac35, no update available). 3.3 does work

Missing Install Docs

We don't have install docs for docker.

What I have done thus far...

1. Change version in docker-compose file to 3.3
2. Run docker-compose up
3. Made and drank a cup of tea.
4. Copy .env-dist to .env
5. docker exec into kitsune container and run the following:
   • ./manage.py migrate
   • ./manage.py createsuperuser
   • ./manage.py generatedata

After the above got errors for all static content, example web_1 | ValueError: The file 'build/common-min.js' could not be found with <kitsune.sumo.storage.SumoFilesStorage object at 0x7f3471785fd0>.

Tried running collect static within container.. generates error AttributeError: 'NoneType' object has no attribute 'strip'

Stuck here for now, also tried this on my linode Linux machine..ran into the same problems.

I do not know what this is yet.. will fill in details / bug from Patrick once we have it.

Create two CloudFront distributions for User Media

• One for production use with origin the production S3 bucket created in #2933
• One for stage use with origin the stage S3 bucket created in #2933

Part of #2899

For dev we'll probably just serve static media from the primary docker container via whitenoise, but that won't scale to prod traffic. We have some options:

• Sync static files to S3 and serve via a CDN
• Deploy a separate Nginx container for serving said files, optionally with a CDN in front
• Put a CDN in front of the entire site (potentially a lot more work)

Current unit test suite is broken. Fix it.

Update: We excluded a few tests from the test suite and it now completes successfully. An issue must be created for each (group of) test(s) which have been excluded and it should be linked to this tracker issue.

Setup Jenkins pipeline to:

• Build
• Test
• Upload to docker hub
• Deploy to k8s

ElasticSearch 2 is reaching EOL on Feb 28th 2018. elasti.co extends its support until Aug 28th 2018

Tracking the shutdown of Army of Awesome.

https://bugzilla.mozilla.org/show_bug.cgi?id=1386808

Using Terraform, create a multi-AZ RDS instance in AWS.

We'll need to specify a 30 minute window for backups, and a 30 minute window for general maintenance. @glogiotatidis Do you have an idea of when would be a good time for these windows?

See this issue for version info.

Kitsune has ZenDesk related code which is very possible to be deprecated. Evaluate it's usefulness and remove if not used.

Travis needs 15-20 minutes to complete testing a pull request. We need to improve this time to iterate faster.

Half of the time is spend re-building the image. Current docker version in Travis (17.03) does not support the cache_from directive in docker-compose.yml so we need to find another way to use pre-built docker images

Current database on SCL3

• MySQL 5.6.33-log

We should investigate the existence of raw SQL queries in the codebase or in libraries (especially django-tidings) before deciding on the Engine type and version.

From the below email there is no way to go back to the original question without marking the question as solved or indicating that it doesn't solve the problem.

Created:

• sumo_dev
• sumo_stage
• sumo_prod

with the needed policies to access S3 user media buckets.

Do we want to create different users for user media and for static media? /cc @pmac @metadave

Uploaded media needs to go somewhere external to the servers in AWS. S3 is good for this with a CDN (probably CloudFront) in front.

Currently we use RabbitMQ as the Celery broker. Celery officially supports Redis too which makes more sense to use since we already have a running Redis instance.

Redis need to be configured to not evict any keys from the Celery database so we don't have missing jobs.

Development documentation under docs/ needs to explain how to setup things with Docker.

Currently SUMO locks /admin behind Auth0 and SSO on the webserver level.

Use mozilla-django-iodc to move the authentication on the App level.

I have some issues with my emails, so I have to manually (by using OS commands) move messages from one folder to another. My Inbox folder has 11000+ messages in Maildir format.

When I move approx. 200 messages from Inbox to Inbox/BAD, and start folder repair of Inbox folder, memory usage of Thunderbird increases several GB and remains that high even after operation finishes.
It is always reproducible.
I'm using Thunderbird 52.4.0 (64bit) on Linux Gentoo.

Blocked by mozmeao/infra#654

On SCL3 we use Memcache as a cache backend for Django.

We have Redis clusters already setup in AWS and we agreed to switch to Redis.

The production database is a large (~100Gb). We need to calculate the time needed to restore it in the real database engine that's going to be used in AWS.

If the time needed is above the threshold considered acceptable to do the move we should investigate other options, like incremental import of data.

A dev import into a db.t2.small instance took about 30 hours. FTR the complete lithium migration took about 48 hours.

Per this community decision to start the Social Support Program, https://support.mozilla.org/en-US/forums/social-media/712414?last=71489 and https://support.mozilla.org/en-US/forums/social-media/712299?last=70945, AoA is closing :(

However a new program with new materials will be replacing it. See issue [Bug 1385135]
https://support.mozilla.org/en-US/forums/social-media/712533?last=72211

Using Terraform, provision a cross AZ Elasticache cluster in AWS. We'll need to determine what instance types to use for the cluster.

Test gets skipped.

Traceback (most recent call last):
  File "/app/kitsune/wiki/tests/test_templates.py", line 1100, in test_restore_draft_revision_with_older_based_on
    trans_content = pq(trans_resp.content)
  File "/usr/local/lib/python2.7/site-packages/pyquery/pyquery.py", line 226, in __init__
    elements = fromstring(context, self.parser)
  File "/usr/local/lib/python2.7/site-packages/pyquery/pyquery.py", line 70, in fromstring
    result = getattr(lxml.html, meth)(context)
  File "/usr/local/lib/python2.7/site-packages/lxml/html/__init__.py", line 726, in fromstring
    doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
  File "/usr/local/lib/python2.7/site-packages/lxml/html/__init__.py", line 614, in document_fromstring
    value = etree.fromstring(html, parser, **kw)
  File "lxml.etree.pyx", line 3103, in lxml.etree.fromstring (src/lxml/lxml.etree.c:70569)
  File "parser.pxi", line 1828, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:106403)
  File "parser.pxi", line 1716, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:105194)
  File "parser.pxi", line 1086, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:99876)
  File "parser.pxi", line 580, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:94350)
  File "parser.pxi", line 690, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:95786)
  File "parser.pxi", line 629, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:95015)
XMLSyntaxError: line 634: Tag footer invalid

See also #2931