Python 3 migration about clusterfuzz HOT 18 CLOSED

Need to move to latest mozprocess version sometime - https://pypi.org/project/mozprocess/#files. As you said, it supports Python 3 now.

from clusterfuzz.

FYI the URL for NDB on Python 3 seems to have changed to https://github.com/googleapis/python-ndb/blob/master/MIGRATION_NOTES.md

from clusterfuzz.

git ls-files | grep requirements.txt | xargs -n1 -t caniusepython3 -r
caniusepython3 -r docker/base/requirements.txt 
Finding and checking dependencies ...

🎉  You have 0 projects blocking you from using Python 3!

caniusepython3 -r docker/ci/requirements.txt 
Finding and checking dependencies ...

You need 1 project to transition to Python 3.
Of that 1 project, 1 has no direct dependencies blocking its transition:

  googledatastore
caniusepython3 -r src/local/requirements.txt 
Finding and checking dependencies ...

🎉  You have 0 projects blocking you from using Python 3!

caniusepython3 -r src/platform_requirements.txt 
Finding and checking dependencies ...

🎉  You have 0 projects blocking you from using Python 3!

caniusepython3 -r src/requirements.txt 
Finding and checking dependencies ...

🎉  You have 0 projects blocking you from using Python 3!

from clusterfuzz.

I think googledatastore should go away after moving to the new ndb package, right @oliverchang ?

from clusterfuzz.

right, googledatastore will no longer be required afterwards.

from clusterfuzz.

Step 1a completed:

git ls-files "*.py" | xargs futurize -w -n --stage1 -x libfuturize.fixes.fix_absolute_import

from clusterfuzz.

Need to enable --py3k in pylint here, but first need help on fixing absolute import things.

butler/lint.py:    _execute_command_and_track_error('pylint ' + file_path)

from clusterfuzz.

Some notes for tracking

ml/rnn/generator.py:   rnn_runner = new_process.ProcessRunner('python')
bot/tasks/ml_train_task.py: new_process.ProcessRunner('python')
protos/generate.sh: python2 -m grpc_tools.protoc --proto_path=$SCRIPT_DIR --python_out=$SCRIPT_DIR --grpc_python_out=$SCRIPT_DIR $SCRIPT_DIR/*.proto

from clusterfuzz.

appengine.mail dependency removed in 26bd818. we now provide support for sendgrid (also one of recommendations by appengine).

from clusterfuzz.

Futurize Stage 1 migration is now complete with 3016965

from clusterfuzz.

ndb still looks like it's in early development based on commits on https://github.com/googleapis/python-ndb ...

to replace the users module, firebase auth looks like the easiest and nicest way forward: https://cloud.google.com/appengine/docs/standard/python/authenticating-users-firebase-appengine

(from the suggested options on https://cloud.google.com/appengine/docs/standard/python/oauth/ )

from clusterfuzz.

It's unclear if firebase auth has a local emulator of sorts so that people can still do run_server without setting up any real projects though...

Perhaps local server instances could have auth disabled completely (i.e. no auth required) to keep things simple? @inferno-chromium what do you think? I think the current local server user system just lets you set a fake email... So we wouldn't be regressing much.

from clusterfuzz.

yes no auth for local sounds good. and i had the same idea on firebase, firebase support multiple other auth systems, so that will be great!

from clusterfuzz.

great, I can give firebase auth a shot next week :)

from clusterfuzz.

Rough plan for the remainder of the migration:

• Complete remaining futurize migrations for bot code.
• Add support for using the Python 3 ndb (dynamically based on if we're running on Python 2 or 3).
• Get all tests passing under Python 3, and more generally get our code in a state which makes it easy to run under both Python 2 and 3.
• Find a replacement for App Engine memcache.
• Migrate App Engine deployment to Python 3. Completely remove dependency on App Engine SDK and migrate to App Engine Python 3 Standard.
• Migrate preemptible bot deployment to Python 3, and gradual rollout to non-preemptible.

@mbarbella-chromium WDYT ?

from clusterfuzz.

The only thing I'd add is to evaluate packages in requirements for Python 3 compaitbility, but that will be part of getting all of the tests passing under Python 3. SGTM overall.

from clusterfuzz.

Current state is that most unit tests pass. All App Engine unit tests pass.

The next steps:

1. Set up CI for Python 3 tests to make sure we don't regress anything.
2. Pick a deployment (chromium?) to dogfood Python 3 appengine.
3. Start staging of Python 3 bots.

from clusterfuzz.

Main part is done here.

from clusterfuzz.