Now that #26 is finished we should look at how we can improve our CI testing.

idk maybe YARN on CI?

In theory most of what we do is with add files in Spark which should be handled, but the decompressed directory I'm less certain about. We should investigate this.

Since some folks run local mode for testing we should document that we don't work here

We currently have one example notebook, would be good to update the example to distribute PyArrow since this will be useful in Spark 2.3+ for vectorized UDF users.

Current README is just a md file. For publishing to pypi we should generate an RST (spark uses a package to do this).

https://github.com/pantsbuild/pex

As was done in snakestagram, explore dependency management with conda.

du -hs snakestagram*
448M	snakestagram
100M	snakestagram-latest-darwin-x64.tar.gz

Uncompressed 448MB conda environment that is fully copyable, 100MB compressed.

Adding on tensorflow to this conda environment makes this 692MB uncompressed, and 145MB compressed.

In the words of @holdenk, "165MB to all the workers is totally fine and not a problem, I totally said this".

We have a wrapper shell script (at least for conda) that we could source files in and pip install
we could then use sc.addFile + foreach to install the new packages at runtime

Like spacy & nltk language downloads.

Since we're announcing a pre-alpha here, we should have some starter instructions to replace what's in the current README.

Write now we create a bunch of temp files but don't really clean them up. There is a flag to do part of this but it needs to be tested and see what else we might be missing (also if we need to change how we download mini conda?)

In the Alternatives section of the README, I noticed it said:

Build you own conda env, or use one of the commercial integrated solutions which has a nice UI for this.

Which makes me wonder how we could start this off, what people would expect to be able to do, as well as what our message protocol from frontend to kernel would be.

Right now we do some terrible things with overriding the PYTHON_PATH, which is great and works in the general case. If the Spark+K8 folks end up integrating better first party support we should integrate with that for K8 deployments.

Look at a way to support cluster mode based on something like https://maprdocs.mapr.com/home/Zeppelin/InstallPySparkConda.html

First version accidently shipped without it. Note: there is a seperate follow up to do this "better" later.

Looks like setup.py is pointing to an old markdown readme.

any successor to coffee_boat?

@rgbkrk found another project testing Spark integrations with circle ci which might be a good starting point. See https://github.com/tribbloid/spookystuff/blob/master/.circleci/config.yml

idk

For now we leave it out since many providers won't support it right now. Long story, buy me a ☕ .