Git Product home page Git Product logo

swift-jupyter's Introduction

Unmaintained Project

Warning: This project has been unmaintained since around the end of 2020.

Consider using Swift-Colab instead.

Swift-Jupyter

This is a Jupyter Kernel for Swift, intended to make it possible to use Jupyter with the Swift for TensorFlow project.

Installation Instructions

Option 1: Using a Swift for TensorFlow toolchain and Virtualenv

Requirements

Operating system:

  • Ubuntu 18.04 (64-bit); OR
  • other operating systems may work, but you will have to build Swift from sources.

Dependencies:

  • Python 3 (Ubuntu 18.04 package name: python3)
  • Python 3 Virtualenv (Ubuntu 18.04 package name: python3-venv)

Installation

swift-jupyter requires a Swift toolchain with LLDB Python3 support. Currently, the only prebuilt toolchains with LLDB Python3 support are the Swift for TensorFlow Ubuntu 18.04 Nightly Builds. Alternatively, you can build a toolchain from sources (see the section below for instructions).

Extract the Swift toolchain somewhere.

Create a virtualenv, install the requirements in it, and register the kernel in it:

git clone https://github.com/google/swift-jupyter.git
cd swift-jupyter
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
python register.py --sys-prefix --swift-toolchain <path to extracted swift toolchain directory>

Finally, run Jupyter:

. venv/bin/activate
jupyter notebook

You should be able to create Swift notebooks. Installation is done!

Option 2: Using a Swift for TensorFlow toolchain and Conda

Requirements

Operating system:

  • Ubuntu 18.04 (64-bit); OR
  • other operating systems may work, but you will have to build Swift from sources.

Installation

1. Get toolchain

swift-jupyter requires a Swift toolchain with LLDB Python3 support. Currently, the only prebuilt toolchains with LLDB Python3 support are the Swift for TensorFlow Ubuntu 18.04 Nightly Builds. Alternatively, you can build a toolchain from sources (see the section below for instructions).

Extract the Swift toolchain somewhere.

Important note about CUDA/CUDNN: If you are using a CUDA toolchain, then you should install CUDA and CUDNN on your system without using Conda, because Conda's CUDNN is too old to work with the Swift toolchain's TensorFlow. (As of 2019-04-08, Swift for TensorFlow requires CUDNN 7.5, but Conda only has CUDNN 7.3).

2. Initialize environment

Create a Conda environment and install some packages in it:

conda create -n swift-tensorflow python==3.6
conda activate swift-tensorflow
conda install jupyter numpy matplotlib

3. Register kernel

Register the Swift kernel with Jupyter:

python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs \
  --swift-toolchain <path to extracted swift toolchain directory>

Finally, run Jupyter:

jupyter notebook

You should be able to create Swift notebooks. Installation is done!

Option 3: Using Docker to run Jupyter Notebook in a container

This repository also includes a dockerfile which can be used to run a Jupyter Notebook instance which includes this Swift kernel. To build the container, the following command may be used:

# from inside the directory of this repository
docker build -f docker/Dockerfile -t swift-jupyter .

The resulting container comes with the latest Swift for TensorFlow toolchain installed, along with Jupyter and the Swift kernel contained in this repository.

This container can now be run with the following command:

docker run -p 8888:8888 --cap-add SYS_PTRACE -v /my/host/notebooks:/notebooks swift-jupyter

The functions of these parameters are:

  • -p 8888:8888 exposes the port on which Jupyter is running to the host.

  • --cap-add SYS_PTRACE adjusts the privileges with which this container is run, which is required for the Swift REPL.

  • -v <host path>:/notebooks bind mounts a host directory as a volume where notebooks created in the container will be stored. If this command is omitted, any notebooks created using the container will not be persisted when the container is stopped.

To improve Docker image building, use the new Docker Buildkit system by either setting the DOCKER_BUILDKIT environment variable or configuring the Docker daemon.json. The simplest way is by prepending DOCKER_BUILDKIT=1 to your docker build command:

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile -t swift-jupyter .

Option 4: Using Docker to run a Swift kernel connected to your local Jupyter Notebook

As of Jupyter Notebook 6.0, you can use --gateway-url= to specify a separate Jupyter Kernel Gateway. (Or use the nb2kg server extension for pre-6.0 versions of Notebook.) This allows running the Swift for Tensorflow Jupyter kernel in a Docker container while running Jupyter Notebook somewhere else, such as your local machine.

First build the basic Swift kernel Docker image (as above), then build the kernel gateway image based on that:

# from inside the directory of this repository
docker build -f docker/Dockerfile -t swift-jupyter .
docker build -f kernel_gateway/Dockerfile -t swift-kg .

Using the new Docker Buildkit system is recommended, as described in the section above.

Then run the kernel gateway:

docker run -p 9999:9999 --cap-add SYS_PTRACE swift-kg

The functions of these parameters are the same as in the section above.

With the gateway running, start Jupyter Notebook in your notebook directory and pass the URL of your kernel gateway:

jupyter notebook --gateway-url 127.0.0.1:9999

(optional) Building toolchain with LLDB Python3 support

Follow the Building Swift for TensorFlow instructions, with some modifications:

  • Also install the Python 3 development headers. (For Ubuntu 18.04, sudo apt-get install libpython3-dev). The LLDB build will automatically find these and build with Python 3 support.
  • Instead of running utils/build-script, run utils/build-toolchain-tensorflow, so that you build a whole toolchain that includes LLDB.

This will create a tar file containing the full toolchain. You can now proceed with the installation instructions from the previous section.

(optional) Building LLDB Python3 support without Swift for TensorFlow

Install the Python 3 development headers. (For Ubuntu 20.04, sudo apt-get install libpython3-dev).

Have a place to checkout relevant toolchains, and checkout the relevant code:

mkdir /opt/swift && cd /opt/swift
git clone https://github.com/apple/llvm-project.git
git clone https://github.com/apple/swift.git
git clone https://github.com/apple/swift-corelibs-libdispatch.git
git clone https://github.com/apple/swift-cmark.git cmark

Make sure you checked out the right branch for all dependencies. For example, the llvm-project should check the branch starting with swift, such as swift/release/5.3. You should be able to find the correct branch with name release/* in all these projects except the llvm-project.

Go to swift/utils, and run:

./build-script --release --lldb

This will build LLDB with Python3 support. Copying everything under build/Ninja-.../lldb-...-x86_64/lib and everything under build/Ninja-.../lldb-...-x86_64/bin to your Swift environment. For example: /opt/swift-5.3/usr/.

There may be some issues with lib/python3 directory not being exactly the same as we should expect. It is safe to rename site-packages to dist-packages.

With the updated LLDB toolchain, you should be able to register the Swift kernel now.

Usage Instructions

Rich output with Python

You can call Python libraries using Swift's Python interop to display rich output in your Swift notebooks. (Eventually, we'd like to support Swift libraries that produce rich output too!)

Prerequisites:

  • You must use a Swift toolchain that has Python interop. As of February 2019, only the Swift for TensorFlow toolchains have Python interop.

After taking care of the prerequisites, run %include "EnableIPythonDisplay.swift" in your Swift notebook. Now you should be able to display rich output! For example:

let np = Python.import("numpy")
let plt = Python.import("matplotlib.pyplot")
IPythonDisplay.shell.enable_matplotlib("inline")
let time = np.arange(0, 10, 0.01)
let amplitude = np.exp(-0.1 * time)
let position = amplitude * np.sin(3 * time)

plt.figure(figsize: [15, 10])

plt.plot(time, position)
plt.plot(time, amplitude)
plt.plot(time, -amplitude)

plt.xlabel("time (s)")
plt.ylabel("position (m)")
plt.title("Oscillations")

plt.show()

Screenshot of running the above two snippets of code in Jupyter

let display = Python.import("IPython.display")
let pd = Python.import("pandas")
display.display(pd.DataFrame.from_records([["col 1": 3, "col 2": 5], ["col 1": 8, "col 2": 2]]))

Screenshot of running the above two snippets of code in Jupyter

Inline plots

You can display images using Swift too.

%install-swiftpm-flags -Xcc -isystem/usr/include/freetype2 -Xswiftc -lfreetype
%install '.package(url: "https://github.com/IBM-Swift/BlueCryptor.git", from: "1.0.28")' Cryptor
%install '.package(url: "https://github.com/KarthikRIyer/swiftplot", .branch("master"))' SwiftPlot AGGRenderer
%include "EnableJupyterDisplay.swift"

Now you should be able to display images! (Currently only PNG format is supported. You also need to provide the image as a base64 String. Eventually we'd like to support other formats as well.)

For example:

import Foundation
import SwiftPlot
import AGGRenderer

func function(_ x: Float) -> Float {
    return 1.0 / x
}var aggRenderer = AGGRenderer()
var lineGraph = LineGraph()
lineGraph.addFunction(
    function,
    minX: -5.0,
    maxX: 5.0,
    numberOfSamples: 400,
    label: "1/x",
    color: .orange)
lineGraph.plotTitle = "FUNCTION"
lineGraph.drawGraph(renderer: aggRenderer)
display(base64EncodedPNG: aggRenderer.base64Png())

Screenshot of running the above snippet of code in Jupyter

To learn more about displaying plots using SwiftPlot take a look at the documentation here.

%install directives

%install directives let you install SwiftPM packages so that your notebook can import them:

// Specify SwiftPM flags to use during package installation.
%install-swiftpm-flags -c release

// Install the DeckOfPlayingCards package from GitHub.
%install '.package(url: "https://github.com/NSHipster/DeckOfPlayingCards", from: "4.0.0")' DeckOfPlayingCards

// Install the SimplePackage package that's in the kernel's working directory.
%install '.package(path: "$cwd/SimplePackage")' SimplePackage

The first argument to %install is a SwiftPM package dependency specification. The next argument(s) to %install are the products that you want to install from the package.

%install directives currently have some limitations:

  • You must install all your packages in the first cell that you execute. (It will refuse to install packages, and print out an error message explaining why, if you try to install packages in later cells.)
  • %install-swiftpm-flags apply to all packages that you are installing; there is no way to specify different flags for different packages.
  • Packages that use system libraries may require you to manually specify some header search paths. See the %install-extra-include-command section below.

Troubleshooting %installs

If you get "expression failed to parse, unknown error" when you try to import a package that you installed, there is a way to get a more detailed error message.

The cell with the "%install" directives has something like "Working in: /tmp/xyzxyzxyzxyz/swift-install" in its output. There is a binary usr/bin/swift where you extracted the toolchain. Start the binary as follows:

SWIFT_IMPORT_SEARCH_PATH=/tmp/xyzxyzxyzxyz/swift-install/modules <path-to-toolchain>/usr/bin/swift

This gives you an interactive Swift REPL. In the REPL, do:

import Glibc
dlopen("/tmp/xyzxyzxyzxyz/swift-install/package/.build/debug/libjupyterInstalledPackages.so", RTLD_NOW)

import TheModuleThatYouHaveTriedToInstall

This should give you a useful error message. If the error message says that some header files can't be found, see the section below about %install-extra-include-command.

%install-extra-include-command

You can specify extra header files to be put on the header search path. Add a directive %install-extra-include-command, followed by a shell command that prints "-I/path/to/extra/include/files". For example,

// Puts the headers in /usr/include/glib-2.0 on the header search path.
%install-extra-include-command echo -I/usr/include/glib-2.0

// Puts the headers returned by `pkg-config` on the header search path.
%install-extra-include-command pkg-config --cflags-only-I glib-2.0

In principle, swift-jupyter should be able to infer the necessary header search paths without you needing to manually specify them, but this hasn't been implemented yet. See this forum thread for more information.

%include directives

%include directives let you include code from files. To use them, put a line %include "<filename>" in your cell. The kernel will preprocess your cell and replace the %include directive with the contents of the file before sending your cell to the Swift interpreter.

<filename> must be relative to the directory containing swift_kernel.py. We'll probably add more search paths later.

Running tests

Locally

Install swift-jupyter locally using the above installation instructions. Now you can activate the virtualenv and run the tests:

. venv/bin/activate
python test/fast_test.py  # Fast tests, should complete in 1-2 min
python test/all_test.py  # Much slower, 10+ min
python test/all_test.py SimpleNotebookTests.test_simple_successful  # Invoke specific test method

You might also be interested in manually invoking the notebook tester on specific notebooks. See its --help documentation:

python test/notebook_tester.py --help

In Docker

After building the docker image according to the instructions above,

docker run --cap-add SYS_PTRACE swift-jupyter python3 /swift-jupyter/test/all_test.py

swift-jupyter's People

Contributors

alew3 avatar asuhan avatar bgogul avatar bradlarson avatar cboone avatar craigcitro avatar dan-zheng avatar dependabot[bot] avatar jekbradbury avatar jph00 avatar kant avatar karthikriyer avatar karwa avatar liuliu avatar marcrasi avatar mikkeyboi avatar pcuenca avatar pvieito avatar rxwei avatar sgugger avatar spencerkohan avatar texasmichelle avatar vojtamolda avatar vvmnnnkv avatar williamhyzhang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

swift-jupyter's Issues

ModuleNotFoundError during Kernel Initialization (Windows)

Traceback (most recent call last):
  File "C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\site-packages\lldb\__init__.py", line 35, in <module>
    import _lldb
ModuleNotFoundError: No module named '_lldb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\mleun\projects\swift-jupyter\swift_kernel.py", line 19, in <module>
    import lldb
  File "C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\site-packages\lldb\__init__.py", line 38, in <module>
    from . import _lldb
ImportError: DLL load failed: The specified module could not be found.
[I 05:16:55.241 NotebookApp] KernelRestarter: restarting kernel (1/5)

Steps to reproduce:

  1. Use conda setup instructions (in Anaconda Prompt, activate the conda environment), acquire toolchain from notes here (#97)
  2. To use changes for the fixed Windows paths in the above PR:
    • Clone this repository, and cd into the directory
    • git remote add -f mikkeyboi https://github.com/mikkeyboi/swift-jupyter.git
    • git checkout mikkeyboi/master
  3. Run register.py while inside conda environment
    • ie: python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs --swift-toolchain path/to/toolchain
  4. Make a jupyter notebook session jupyter notebook while within conda environment
  5. Create a new notebook with Swift kernel
  6. While the kernel is starting, I get the error.

Alternatively..

  1. cd into path/to/toolchain/usr/lib/site-packages and run interactive python ipython
  2. import lldb

I think it's not loading the _lldb.pyd file that's inside the lldb module. Either that, or I'm missing DLLs (can confirm with Dependency Walker).

@compnerd Were you able to load the lldb module during your testing (specifically the _lldb.pyd file)?

Linux: Could not determine host triple

Hi,

thanks a lot for providing an awesome tool for Swift for Tensorflow.

On Ubuntu 19.10, I am running the latest Swift for Tensorflow toolchain from here. I can install packages, compile and run programs with it just fine.

I installed Swift-jupyter via the conda route.

When I try to execute a first cell that contains install instructions, like this one:

%install '.package(url: "https://github.com/vojtamolda/Plotly.swift.git", .exact("0.2.0"))' Plotly
%include "EnableIPythonDisplay.swift"

I get the following error:

Installing packages:
	.package(url: "https://github.com/vojtamolda/Plotly.swift.git", .exact("0.2.0"))
		Plotly
With SwiftPM flags: []
Working in: /tmp/tmprpl_p8ja/swift-install
Fatal error: could not determine host triple: malformed: file /swift-base/swiftpm/Sources/SPMBuildCore/Triple.swift, line 149

Install Error: swift-build returned nonzero exit code -4.

Again, using the toolchain manually, installing packages and building programs using the Swift Package Manager (SPM) works.

I also had a look at the Triple.swift file from SPM and I don't understand why it should fail, since the reported triple from swift -print-target-info is:

{
  "target": {
    "triple": "x86_64-unknown-linux-gnu",
    "unversionedTriple": "x86_64-unknown-linux-gnu",
    "moduleTriple": "x86_64-unknown-linux-gnu",
    "librariesRequireRPath": false
  },
  "paths": {
    "runtimeLibraryPaths": [
      "/home/pahl/progs/swift/usr/lib/swift/linux"
    ],
    "runtimeLibraryImportPaths": [
      "/home/pahl/progs/swift/usr/lib/swift/linux/x86_64"
    ],
    "runtimeResourcePath": "/home/pahl/progs/swift/usr/lib/swift"
  }
}

which should be fine.

Many thanks in advance for your help.

Kind regards,
Axel

Xcode11 toolchain linked against Python2.7

/Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.11.xctoolchain/System/Library/PrivateFrameworks/LLDB.framework/Resources/Python/lldb$ otool -L _lldb.so | grep Python
	/System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.16)

Successor notice?

I have been working on Swift-Colab for a while, and it seems like a logical successor to google/swift-jupyter. Google is not actively developing this repository anymore, or really anything revolving around the Swift for TensorFlow project. And I learned the hard way that they won't assist my effort to bring it back.

I put a crash course on how Swift-Colab continued the legacy of google/swift-jupyter in ColabSupportHistory.md. It doesn't support Docker anymore, but it's entirely possible to add that as a feature in Swift-Colab 3.0. @marcrasi I am wondering whether you could put a notice on this repo's README, linking to the spin-off that has more up-to-date documentation.

ability to read from Google Drive when running in Colab

It would be nice to read from Google Drive when running Swift in Colab.

You can do this in Python Colab: https://colab.research.google.com/notebooks/io.ipynb. So the obvious thing to try is to use Swift's Python interop to Python.import the necessary libraries. Unfortunately, this does not work because the Python library prompts the user for an authentication key with an interactive text widget, and the Swift Python interop does not support interactive widgets.

So the right solution may be to modify the Python library so that you can pass it the authentication key directly instead of needing to paste it into a text widget. See here for a bit more detail about how it might be done: https://forums.fast.ai/t/python-textfield-output-not-working/51000/5?u=marcrasi

Error initing KernelCommunicator

I tried installing through conda, as well through system python3 in Macbook Air M1. In both cases I ended up with the following error after starting

jupyter-notebook --debug

  File "/Users/sindhus/Library/Python/3.8/lib/python/site-packages/ipykernel/kernelbase.py", line 353, in dispatch_shell
    await result
  File "/Users/sindhus/Library/Python/3.8/lib/python/site-packages/ipykernel/kernelbase.py", line 643, in execute_request
    reply_content = self.do_execute(
  File "/Users/sindhus/code/swift-jupyter/swift_kernel.py", line 982, in do_execute
    self._init_swift()
  File "/Users/sindhus/code/swift-jupyter/swift_kernel.py", line 216, in _init_swift
    self._init_kernel_communicator()
  File "/Users/sindhus/code/swift-jupyter/swift_kernel.py", line 295, in _init_kernel_communicator
    raise Exception('Error initing KernelCommunicator: %s' % result)
Exception: Error initing KernelCommunicator: SwiftError(result=<lldb.SBValue; proxy of <Swig Object of type 'lldb::SBValue *' at 0x112896fc0> >, description="error: couldn't start parsing - no stack frame\n")

Kernel did start running otherwise, it is syntax highlighting correctly as well. I am running without tensorflow:

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/share/jupyter/kernels/swift/kernel.json

{
  "argv": [
    "/Library/Developer/CommandLineTools/usr/bin/python3",
    "/Users/sindhus/code/swift-jupyter/swift_kernel.py",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Swift",
  "language": "swift",
  "env": {
    "PYTHONPATH": "/Library/Developer/Toolchains/swift-5.5.1-RELEASE.xctoolchain/System/Library/PrivateFrameworks/LLDB.framework/Resources/Python",
    "LD_LIBRARY_PATH": "/Library/Developer/Toolchains/swift-5.5.1-RELEASE.xctoolchain/usr/lib/swift/macosx",
    "REPL_SWIFT_PATH": "/Library/Developer/Toolchains/swift-5.5.1-RELEASE.xctoolchain/System/Library/PrivateFrameworks/LLDB.framework/Resources/repl_swift"
  }
}

Any help would be appreciated!

Assertion `isInt<32>(RealOffset)' failed

This used to be https://bugs.swift.org/browse/TF-747, but I'm moving the issue here and adding more details.

It's flaky and hard to reproduce, but here are the kinds of things you need to try:

If you trigger it, the kernel will crash and this will print out on the terminal:

/swift-base/llvm-project/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:307: void llvm::RuntimeDyldELF::resolveX86_64Relocation(const llvm::SectionEntry &, uint64_t, uint64_t, uint32_t, int64_t, uint64_t): Assertion `isInt<32>(RealOffset)' failed.

Our guess at the reason and how to fix:

If you load up too many shared libraries in jupyter then most of the address space gets used and the memory manager starts giving you pieces of memory that are too far apart for the relocation, causing this error. The only real fix we can think of is migrating LLDB to use orcjit, which Saleem thinks is O(months) of work.

PythonKit + TensorFlow is "too much" for the purposes of this problem, so notebooks using both are going to be flaky.

Docker image fails to run Swift code

I installed a Docker image as per Option 3 in https://github.com/google/swift-jupyter/blob/master/README.md on a Mac with OSX version 10.15.7 and Docker desktop version 2.4.0.0.

I created a new notebook with a Swift kernel and tried to run the single code cell:

print("hello")

which produces the following error:

Cannot create Swift scratch context (couldn't load the Swift stdlib)Cannot create Swift scratch context (couldn't load the Swift stdlib)[IPKernelApp] ERROR | Exception in message handler:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 268, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 545, in execute_request
user_expressions, allow_stdin,
File "/swift-jupyter/swift_kernel.py", line 967, in do_execute
self._init_swift()
File "/swift-jupyter/swift_kernel.py", line 217, in _init_swift
self._init_int_bitwidth()
File "/swift-jupyter/swift_kernel.py", line 303, in _init_int_bitwidth
result)
Exception: Expected value from Int.bitWidth, but got: SuccessWithoutValue()

The notebook exists on disk and produces the same error every time I load it and attempt to run the cell above.

Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named ipykernel.zmqshell

Steps:

  • Clean installation of Ubuntu 18.04
  • Install swift-tensorflow v0.11.0 in ~/swift-tensorflow/
  • clone this repo, cd into it
  • sudo apt-get install python3-venv
  • python3 -m venv sju
  • . sju/bin/activate
  • pip install -r requirements.txt
  • python register.py --sys-prefix --swift-toolchain ~/swift-tensorflow/
  • mkdir ~/src/tensorflow && cs ~/src/tensorflow
  • git clone [email protected]:tensorflow/swift.git
  • cd ~/src/tensorflow/swift
  • jupyter notebook

In Jupyter web UI:

  • navigate to /notebooks/docs/site/tutorials/model_training_walkthrough.ipynb
  • Execute all the cells
  • This cell fails:
// This cell is here to display the plots in a Jupyter Notebook.
// Do not copy it into another environment.
%include "EnableIPythonDisplay.swift"
IPythonDisplay.shell.enable_matplotlib("inline")

with the error:

Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named ipykernel.zmqshell
Traceback:
  File "/home/garymm/src/swift-jupyter/swift_shell/__init__.py", line 15, in <module>
    from ipykernel.zmqshell import ZMQInteractiveShell
: file PythonKit/Python.swift, line 673
Current stack trace:
0    libswiftCore.so                    0x00007f9e385090a0 swift_reportError + 50
1    libswiftCore.so                    0x00007f9e38574f40 _swift_stdlib_reportFatalErrorInFile + 115
2    libswiftCore.so                    0x00007f9e381defee <unavailable> + 1621998
3    libswiftCore.so                    0x00007f9e381dce00 _assertionFailure(_:_:file:line:flags:) + 827
4    libswiftCore.so                    0x00007f9e38235880 Dictionary.init<A>(_:uniquingKeysWith:) + 0
5    libPythonKit.so                    0x00007f9e390810df <unavailable> + 94431
Current stack trace:
	frame #3: 0x00007f9e38fc44cc $__lldb_expr28`static IPythonDisplay.enable(self=IPythonDisplay) at EnableIPythonDisplay.swift:67:30
	frame #4: 0x00007f9e38fc2057 $__lldb_expr28`main at EnableIPythonDisplay.swift:104:16

Trying to debug a bit, it seems the python being used by the Swift code is not the python from the venv. This code:

let sys = Python.import("sys")
sys.path

Prints:

['/home/garymm/src/swift-jupyter', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

The kernel file looks right to me:

$ cat ./sju/share/jupyter/kernels/swift/kernel.json
{
  "argv": [
    "/home/garymm/src/swift-jupyter/sju/bin/python",
    "/home/garymm/src/swift-jupyter/parent_kernel.py",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Swift",
  "language": "swift",
  "env": {
    "PYTHONPATH": "/home/garymm/swift-tensorflow/usr/lib/python3/dist-packages",
    "LD_LIBRARY_PATH": "/home/garymm/swift-tensorflow/usr/lib/swift/linux",
    "REPL_SWIFT_PATH": "/home/garymm/swift-tensorflow/usr/bin/repl_swift",
    "SWIFT_BUILD_PATH": "/home/garymm/swift-tensorflow/usr/bin/swift-build",
    "SWIFT_PACKAGE_PATH": "/home/garymm/swift-tensorflow/usr/bin/swift-package"
  }
}

Help running Jupyter

I am trying to resurrect Swift for TensorFlow, and side-loading Swift on Google Colab is vital to that effort. Running on Google Colab presents several challenges, so I need to run a Jupyter notebook successfully on my Mac first. I set up Docker, but running Swift on Jupyter produces errors I cannot work around.

I followed the instructions for option 3 on the README. Options 1 and 2 are not viable, as building Swift from source is extremely time-consuming. Furthermore, the new kernel must be independent from the TensorFlow toolchain.

I tried setting it up with the kernel gateway (option 4) but kept receiving this error. I assume it’s because the necessary image was removed from Docker Hub:

Screen Shot 2021-12-16 at 1 01 20 PM

Using option 3 instead, I created a blank Swift Jupyter notebook that executes print("hello world"). I ran the code cell and nothing appeared as output. Instead, the terminal logged the errors shown below. After switching the kernel from Swift to Python 3, it worked just fine. The bug happened on both a fresh Docker image and one I modified heavily:

Screen Shot 2021-12-16 at 1 06 47 PM

I need to isolate this error from the Swift for TensorFlow toolchain. To start off, I will find a vanilla Ubuntu Docker image and install the Swift 5.5 release toolchain. Then, I will get the Swift REPL running.

Your Docker image automatically downloads an S4TF toolchain, so I assume its code won’t work with an Apple toolchain. My only option is to reconstruct your repository file by file on top of an Apple toolchain. Furthermore, REPL isn’t supported with the TensorFlow toolchain:

Screen Shot 2021-12-16 at 2 27 58 PM

When you made the Swift Jupyter kernel, you started out from scratch and gradually built what you have now. Could you point me toward the first file you added, so I can start off by testing that? Then, in what order should I add the other files to reconstruct the entire repository?

The new Swift for TensorFlow won’t need a special Swift toolchain. It will run as a Swift package on top of the release toolchain, allowing it to run on iOS. Is it possible to remove everything related to TensorFlow from the Jupyter kernel? I understand that I may need to add some files back in to get it to run on Colab.

Using an Apple silicon Mac could be the cause of the bug. I have a 2018 Intel Mac mini as well. Do you recommend that I try to reproduce the bug on the mini?

An error occurred on apple M1

$ docker build -f docker/Dockerfile -t swift-jupyter .

[+] Building 81.7s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                       0.0s
 => => transferring dockerfile: 127B                                                       0.0s
 => [internal] load .dockerignore                                                          0.0s
 => => transferring context: 34B                                                           0.0s
 => ERROR [internal] load metadata for gcr.io/swift-tensorflow/base-deps-cuda10.2-cudnn7  81.6s
------
 > [internal] load metadata for gcr.io/swift-tensorflow/base-deps-cuda10.2-cudnn7-ubuntu18.04:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head https://gcr.io/v2/swift-tensorflow/base-deps-cuda10.2-cudnn7-ubuntu18.04/manifests/latest: Bad Gateway

How SWIFT_IMPORT_SEARCH_PATH supported in Python enabled LLDB

Hey, all.

Thanks for the great work on Swift Jupyter. I am trying to figure out how to compile latest Swift 5.3 with Python LLDB support without going off Swift for TensorFlow branch. So far, it worked well except the SWIFT_IMPORT_SEARCH_PATH bit. It appears that has no impact for the apple/llvm-project at release/5.3. I am wondering how this is implemented and whether we can port it over to the mainline. Thanks!

Cannot show expression values

The recent merge causes LLDB to crash when it tries to show an expression value.

I'm going to work around this by disabling showing expression values. Users can still see values using print().

This issue reminds us to re-enable showing expression values when we can. I expect some upstream fix will make it start working again soon.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.