Update dependencies and ensure all notebooks are working about dask-examples HOT 5 CLOSED

• prophet requires pystan==2.19.1.1, but pystan 3.3.0 is released (this is a known issue with prophet)
• pystan, httpstan, and simdjson are not available for aarch64 via conda or PyPI

Fortunately, it seems as though prophet has the ability to use the cmdstan backend, which does not require pystan at all. At the moment, this requires manually installing dependencies, but hopefully this will improve. xref facebook/prophet#2041 and facebook/prophet#2088

from dask-examples.

Trying to build the cmdstan examples, with the cmdstan package from conda-forge results in a missing lgcc error:

(base) root ➜ /opt/conda/bin/cmdstan $ CC=aarch64-conda-linux-gnu-cc CXX=aarch64-conda_cos7-linux-gnu-c++ LD_LIBRARY_PATH=$CONDA_DIR/lib make examples/bernoulli/bernoulli

--- Compiling, linking C++ code ---
aarch64-conda_cos7-linux-gnu-c++ -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem /opt/conda/include -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I /opt/conda/include/    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_5.7.0/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include    -DBOOST_DISABLE_ASSERTS   -DTBB_INTERFACE_NEW  -DTBB_INTERFACE_NEW     -c  -x c++ -o examples/bernoulli/bernoulli.o examples/bernoulli/bernoulli.hpp
aarch64-conda_cos7-linux-gnu-c++ -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem /opt/conda/include -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I /opt/conda/include/    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_5.7.0/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include    -DBOOST_DISABLE_ASSERTS   -DTBB_INTERFACE_NEW  -DTBB_INTERFACE_NEW    -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib        -Wl,-L,"/opt/conda/lib/" -Wl,-rpath,"/opt/conda/lib/" -ltbb    examples/bernoulli/bernoulli.o src/cmdstan/main.o        -Wl,-L,"/opt/conda/lib/" -Wl,-rpath,"/opt/conda/lib/" -ltbb stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_kinsol.a   -o examples/bernoulli/bernoulli
/opt/conda/bin/../lib/gcc/aarch64-conda-linux-gnu/9.4.0/../../../../aarch64-conda-linux-gnu/bin/ld: cannot find -lgcc
collect2: error: ld returned 1 exit status
make: *** [make/program:59: examples/bernoulli/bernoulli] Error 1

For context, this is in an Ubuntu aarch64 Docker container. I suppose this should be filed as an issue against conda-forge/cmdstan-feedstock, but I wanted to drop it here for now as I'm closing up for the day.

from dask-examples.

So I put together a devcontainer for VS Code remote, it's living over on this branch: https://github.com/bryanwweber/dask-examples/tree/devcontainer. What that allows is to use the VS Code Remove Development extension and run all the notebooks in a Docker container with all the dependencies already installed. That will hopefully be quite useful for anyone wanting to add/extend examples, but I don't think it needs to be merged here.

With that container, I ran pytest and there were 4 notebooks that failed, presumably due to syntax changes in Dask/Pandas since Dask v2.20? My next task here is to debug those notebooks and fix them up:

• FAILED dataframes/02-groupby.ipynb
• FAILED dataframes/01-data-access.ipynb
• FAILED machine-learning/xgboost.ipynb
• FAILED dataframes/04-reading-messy-data-into-dataframes.ipynb

One question I had for anyone following along: what to do about the Binder image, in terms of including dependencies of the Notebooks? Several dependencies are installed with pip inside the relevant Notebook example. I think I understand that we are trying to keep the image relatively small; however, I don't think it's very good practice to install with pip inside a Notebook, although I don't have specific evidence for that thought...

from dask-examples.

Another question I have is, what is the purpose of the repo2docker CI job? Does that image get uploaded anywhere, or is it just to see that repo2docker continues to work? I also note that the build-and-deploy CI job has failed on main with the last two pushes, although it succeeded on the two branches that were merged. https://github.com/dask/dask-examples/actions/runs/1786946502 https://github.com/dask/dask-examples/actions/runs/1653328956

from dask-examples.

FWIW here, VS Code offers the ability to set up an external folder to store devcontainer configuration: https://code.visualstudio.com/docs/remote/create-dev-container#_alternative-repository-configuration-folders

from dask-examples.