Git Product home page Git Product logo

xeus-lua-kernel's Issues

`build:labextension:dev` stage fails

@jtpio
here is my attempt to compile xeus-lua as seperate extension.

First I had to add a https://github.com/DerThorsten/jupyterlite_xeus_lua/blob/main/src/xeus_lua.d.ts which I had not in the latest version of xeus-lua directly integrated in jupyterlite (at some point I also had such a file in the direct integrated xeus lua)

but jlpm run build fails horrible at the build:labextension:dev stage
( jlpm run build:lib without any error)

Any hint what could be the problem?

 tsc
$ jupyter labextension build --development True .
Building extension in .

Compilation starting…


<--- Last few GCs --->

[48218:0x55a0f730ff30]   364047 ms: Scavenge 8000.4 (8222.0) -> 7996.8 (8223.0) MB, 12.5 / 0.0 ms  (average mu = 0.169, current mu = 0.038) task 
[48218:0x55a0f730ff30]   364133 ms: Scavenge 8002.9 (8223.0) -> 7996.0 (8231.2) MB, 16.2 / 0.0 ms  (average mu = 0.169, current mu = 0.038) allocation failure 
[48218:0x55a0f730ff30]   366750 ms: Mark-sweep 8011.5 (8231.2) -> 7988.5 (8231.7) MB, 2509.7 / 0.1 ms  (average mu = 0.198, current mu = 0.226) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x7f3b40cc75c9 node::Abort() [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 2: 0x7f3b40ba94a3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > node::SPrintFImpl<int&, char const*>(char const*, int&, char const*&&) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 3: 0x7f3b4107a832 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 4: 0x7f3b4107ab0b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 5: 0x7f3b412411d6  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 6: 0x7f3b412412c1  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 7: 0x7f3b41255c24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 8: 0x7f3b41256524 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 9: 0x7f3b412bc5cd v8::internal::ScavengeJob::Task::RunInternal() [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
10: 0x7f3b4115abdc  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
11: 0x7f3b40d44794 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
12: 0x7f3b40d47f73 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
13: 0x7f3b40230901  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/./libuv.so.1]
14: 0x7f3b402427fc  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/./libuv.so.1]
15: 0x7f3b4023102c uv_run [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/./libuv.so.1]
16: 0x7f3b40bfafd7 node::SpinEventLoop(node::Environment*) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
17: 0x7f3b40d148d5 node::NodeMainInstance::Run(int*, node::Environment*) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
18: 0x7f3b40d14d88 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
19: 0x7f3b40c8a9fc node::Start(int, char**) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
20: 0x55a0f648f0fb main [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/node]
21: 0x7f3b40293565 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
22: 0x55a0f648f146  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/node]
An error occurred.
subprocess.CalledProcessError: Command '['node', '/home/derthorsten/src/jupyterlite_xeus_lua/node_modules/@jupyterlab/builder/lib/build-labextension.js', '--core-path', '/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/lib/python3.9/site-packages/jupyterlab/staging', '/home/derthorsten/src/jupyterlite_xeus_lua', '--development']' died with <Signals.SIGABRT: 6>.
See the log file for details:  /tmp/jupyterlab-debug-gu5vc6u_.log

Reducing the size of the wasm file

The current wasm file is about 40MB which is quite huge.

Wondering whether there would be a way to reduce the size. Maybe that would involve passing a couple of flags to the emscripten build command.

Next steps to package and distribute the kernel

Thinking about the next steps once we get the first version working as a federated extension:

  1. we take the xeus lua specific parts out
  2. we create a https://github.com/jupyterlite/xeus-kernel repo that other kernels like xeus-lua can depend on. This repo would contain most of the logic
  3. jupyterlite-xeus-lua would live in its own repo, for example https://github.com/jupyterlite/xeus-lua-kernel, and would depend on the @jupyterlite/xeus-kernel packages. It would also have the tools (Dockerfile) to create the wasm file, and would specify it as a parameter
  4. streamline (or at least document) the process to place the files generated by emscripten so they are picked up by the extension on build
  5. Clean up @ts-ignore in the code

Bundle the wasm file

now that we move the xeus lua related files to the asset dir via https://github.com/DerThorsten/jupyterlite_xeus_lua/blob/3a22e56672ef5aca0655fc64aaa39fac74e5b6a2/webpack.config.js#L4

we are facing the problem that xeus_lua.js it not found

ModuleNotFoundError: Module not found: Error: Can't resolve './xeus_lua' in '/home/derthorsten/src/jupyterlite_xeus_lua/lib'
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/Compilation.js:1930:28
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:793:13
    at eval (eval at create (/home/derthorsten/src/jupyterlite_xeus_lua/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:10:1)
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:275:22
    at eval (eval at create (/home/derthorsten/src/jupyterlite_xeus_lua/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:431:22
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:124:11
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:665:25
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:850:8
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:970:5
resolve './xeus_lua' in '/home/derthorsten/src/jupyterlite_xeus_lua/lib'
...
...
...

@jtpio you mentioned:

Locally excluding the emscripten generated files seems to help build the extension.

However the files are not properly emitted to labextension/static yet.

@jtpio any hint on how to continue on that?

Do we need to change the import statement?
https://github.com/DerThorsten/jupyterlite_xeus_lua/blob/3a22e56672ef5aca0655fc64aaa39fac74e5b6a2/src/worker.ts#L3

this is all a bit beyond my typescript / webpack competence ...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.