"Module not found" error using groupby-apply in dask-yarn on EMR about dask-yarn HOT 2 CLOSED

Thanks for filing an excellent issue report.

It also creates a python source file (dask_test.py) containing a simple function on all nodes, and adds the file to the PYTHONPATH

What's happening here is that YARN manages the worker environments, so you setting PYTHONPATH manually beforehand isn't seen by the workers (since YARN doesn't run user's .bashrc files). Mucking with PYTHONPATH is not the recommended way to install code in python, as it relies on the environment variables being set appropriately in all environments. I recommend one of the following:

• Make your code a true package, and install it properly. Note that this doesn't require you to put it on pypi, pip can install fine from source or from a git url (e.g. pip install git+https://github.com/dask/dask-yarn.git).
• Copy your code manually into $HOME/miniconda/lib/python3.6/site-packages/ (depends on your python version/location).
• If your code is small enough you could define everything in the notebook/script/whatever and cloudpickle will handle serializing the code to all nodes. Anything in __main__ will be serialized automatically.

Using Client.upload_file will work if your cluster is fixed in size, but won't work if you add/remove nodes during operation (it only uploads things once, so new workers won't get the files added). I don't recommend using it, it's around mostly for legacy reasons.

If you really want to set PYTHONPATH manually, you can pass worker_env={'PYTHONPATH': ...} to YarnCluster, and it will be set in your worker environments.

from dask-yarn.

Great. Both of these two options worked for me:

• Make your code a true package, and install it properly. Note that this doesn't require you to put it on pypi, pip can install fine from source or from a git url (e.g. pip install git+https://github.com/dask/dask-yarn.git).
• If your code is small enough you could define everything in the notebook/script/whatever and cloudpickle will handle serializing the code to all nodes. Anything in main will be serialized automatically.

Thanks for the quick response.

from dask-yarn.