Git Product home page Git Product logo

s3nb's Introduction

S3-backed notebook manager for IPython

Setup

  1. Install:

    from pypi:

    pip install s3nb

    from source with pip:

    pip install git+https://github.com/monetate/s3nb

    or from source the old fashioned way:

    git clone [email protected]:monetate/s3nb.git
    cd s3nb
    python ./setup.py install
  2. Configure

    # set this - notebooks will be stored relative to this uri
    S3_NOTEBOOK_URI=s3://path/to/notebooks/
    
    # and this
    IPYTHON_MAJOR_VERSION=4
    
    # optionally set this - checkpoints will be stored locally, relative to this path (for IPython 3)
    CHECKPOINT_ROOT_DIR=~/.checkpoints
    
    # optionally set this
    PROFILE=s3nbserver
    
    # shouldn't need to edit beyond this point
    
    ## IPython 2.x
    IPYNB_MANAGER=S3NotebookManager
    IPYNB_MANAGER_CFG=notebook_manager_class
    
    ## IPython 3.x or 4.x
    if [ $IPYTHON_MAJOR_VERSION == 3 ] || [ $IPYTHON_MAJOR_VERSION == 4 ]; then
        IPYNB_MANAGER=S3ContentsManager
        IPYNB_MANAGER_CFG=contents_manager_class
    fi
    
    IPYTHONDIR=${IPYTHONDIR:-$HOME/.ipython}
    PROFILE_DIR=${IPYTHONDIR}/profile_${PROFILE}
    
    if [ ! -d $PROFILE_DIR ]; then
        ipython profile create $PROFILE
        IPYNB_CONFIG=${PROFILE_DIR}/ipython_notebook_config.py
        mv $IPYNB_CONFIG $IPYNB_CONFIG.orig
        cat > $IPYNB_CONFIG <<EOF
    c = get_config()
    c.NotebookApp.${IPYNB_MANAGER_CFG} = 's3nb.${IPYNB_MANAGER}'
    c.${IPYNB_MANAGER}.s3_base_uri = '$S3_NOTEBOOK_URI'
    EOF
    fi
    
    
    if  [ $IPYTHON_MAJOR_VERSION == 3 ] || [$IPYTHON_MAJOR_VERSION == 4 ] ; then
        echo "c.S3ContentsManager.checkpoints_kwargs = {'root_dir': '${CHECKPOINT_ROOT_DIR}'}"  >> ${IPYNB_CONFIG}
    fi
  3. If you haven't already, configure AWS variables for boto. Follow these instructions.

  4. Run

    jupyter notebook --config=~/.ipython/s3nbserver/ipython_notebook_config.py

Development

  1. Provision a virtual machine with vagrant up
  2. Create an IPython profile with make configure -e S3_BASE_URI=YOUR_BUCKET
  3. Share you AWS credentials with the virtual machine with make creds -e AWS_USER=YOUR_USER
  4. Run the notebook server with make run

s3nb's People

Contributors

austinrochford avatar hanleyhansen avatar jeffpatti avatar patrick-russell avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

s3nb's Issues

Doe not work with shallow s3 base URIs

If c.S3ContentsManager.s3_base_uri = 's3://s3nb/', no notebooks will be found.
If c.S3ContentsManager.s3_base_uri = 's3://s3nb/notebooks/', if functions as expected.

Temporary files cause issue under Windows 7

Seems that NamedTemporaryFiles under Windows are opened automatically. This causes issues when codec.open is then used to open the file in the IPython V3 code. I changed the code to use the following method as opposed to codecs_open in the save and get methods.

Am happy to integrate it in if need be.

Thanks for this work.

Regards
Paul

     def _codecs_open(self, filename, file_handle, mode='rb', encoding=None, errors='strict', buffering=1):
        # Copy of codecs.open
        if encoding is not None:
            if 'U' in mode:
                # No automatic conversion of '\n' is done on reading and writing
                mode = mode.strip().replace('U', '')
                if mode[:1] not in set('rwa'):
                    mode = 'r' + mode
            if 'b' not in mode:
                # Force opening of the file in binary mode
                mode = mode + 'b'
        try:
            file = open(filename, mode, buffering)
        except:
            file = file_handle
        if encoding is None:
            return file
        info = codecs.lookup(encoding)
        srw = codecs.StreamReaderWriter(file, info.streamreader, info.streamwriter, errors)
        # Add attributes to simplify introspection
        srw.encoding = encoding
        return srw

Save was changed as follows (get the same)

 def _save_notebook(self, path, nb):
        self.log.debug('_save_notebook: %s', locals())

        k = boto.s3.key.Key(self.bucket)
        k.key = self._path_to_s3_key(path)

        try:
            with tempfile.NamedTemporaryFile() as t:
                with self._codecs_open(t.name, t, mode='w', encoding='utf-8') as f:
                    # write tempfile with utf-8 encoding
                    nbformat.write(nb, f, version=nbformat.NO_CONVERT)
                    # upload as bytes (t's fp didn't advance)
                    t.seek(0)
                    k.set_contents_from_file(t)
        except Exception as e:
            raise web.HTTPError(400, u"Unexpected Error Writing Notebook: %s %s" % (path, e))

Api calls to /api/contents fail.

This error comes out:

{"reason": null, "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/IPython/html/base/handlers.py\", line 441, in wrapper\n result = yield gen.maybe_future(method(self, *args, **kwargs))\n File \"/usr/lib64/python2.7/site-packages/tornado/gen.py\", line 1015, in run\n value = future.result()\n File \"/usr/lib64/python2.7/site-packages/tornado/concurrent.py\", line 237, in result\n raise_exc_info(self._exc_info)\n File \"/usr/lib64/python2.7/site-packages/tornado/gen.py\", line 285, in wrapper\n yielded = next(result)\n File \"/usr/lib/python2.7/site-packages/IPython/html/services/contents/handlers.py\", line 126, in get\n path=path, type=type, format=format, content=content,\n File \"/usr/lib/python2.7/site-packages/s3nb-0.0.4.37-py2.7.egg/s3nb/ipy3.py\", line 190, in get\n model = self._s3_key_file_to_model(k, timeformat=S3_TIMEFORMAT_GET_KEY)\n File \"/usr/lib/python2.7/site-packages/s3nb-0.0.4.37-py2.7.egg/s3nb/ipy3.py\", line 63, in _s3_key_file_to_model\n self.log.debug(\"_s3_key_file_to_model: %s: %s\", key, key.name)\nAttributeError: 'NoneType' object has no attribute 'name'\n", "message": "Unknown server error"}

awesome implementation, but can't create new folderss

I seem to be getting a "method not allowed" error when I attempt to rename folders. I can do other things, such as create folders and notebooks, so I don't believe permissions are an issue. Thanks for building this!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.