ScriptCore is a fork of Shopify's enterprise script service.
The enterprise script service (aka ESS) is a thin Ruby API layer that spawns a process, the enterprise_script_engine
, to execute an untrusted Ruby script.
The enterprise_script_engine
executable ingests the input from stdin
as a msgpack encoded payload; then spawns an mruby-engine; uses seccomp to sandbox itself; feeds library
, input
and finally the Ruby scripts into the engine; returns the output as a msgpack encoded payload to stdout
and finally exits.
I want to make these changes:
- Toolchain
- Expose mruby build config to allow developer modify mruby-engine executable, e.g: add some gems
- Expose
mrbc
to allow developer precompile mruby library that would inject to sandboxie - Watching and auto compiling mruby library when it change
- Rake tasks for compiling mruby-engine & mruby library
- Capistrano recipe
- Practice
- Rails generator for mruby library
- Find a good place for engines
- Find a good way to working with timezone on mruby side
- Find a good way to working with
BigDecimal
&Date
(mruby doesn't have these) on mruby side - Consider about IO operations
Clone the repository.
$ git clone https://github.com/rails-engine/script_core
Change directory
$ cd script_core
Fetch submodules
$ git submodule update --init --recursive
Run bundler
$ bundle install
Preparing database
$ bin/rails db:migrate
Build mruby engine & engine lib
$ bin/rails app:script_core:engine:build
$ bin/rails app:script_core:engine:compile_lib
Start the Rails server
$ bin/rails s
Open your browser, and visit http://localhost:3000
Add this line to your Gemfile:
gem 'script_core'
Or you may want to include the gem directly from GitHub:
gem 'script_core', github: 'rails-engine/script_core'
Then execute:
$ bundle
ScriptCore already has a default executable, because of mruby's gem is compiled in binary, or you may want to build a mruby library, build your own engine is necessary.
You can check spec/dummy/mruby
as reference.
Run the task in your app directory:
$ rails script_core:engine:new [engine_name]
engine_name
is optional, by default it would be mruby
that will generate mruby
directory in your app root folder.
Then execute:
$ rails script_core:engine:build [engine_name]
It will build mruby executables.
Remove .example
extension for engine.gembox.example
, customize it, then rebuild the engine.
Warning: because of seccomp
, you may meet compatibility problems, especially for IO relates gems.
Write your own lib for mruby environment in mruby/lib
directory.
Run the task in your app directory:
$ rails script_core:engine:compile_lib [engine_name]
Because of engine binaries are platform dependent, it's good to compile in every deployment.
Simply add mruby/bin
to .gitignore
.
You can wrap it for example:
module ScriptEngine
class << self
def engine
@engine ||= ScriptCore::Engine.new Rails.root.join("mruby/bin")
end
def eval(string, input: nil, instruction_quota_start: nil, environment_variables: {})
sources = [
["user", string],
]
engine.eval sources, input: input,
instruction_quota_start: instruction_quota_start,
environment_variables: environment_variables
end
end
end
Then use it:
ScriptEngine.eval "@output = 'hello world'"
- Add
/mruby/bin
into.gitignore
- Don't do any IO in mruby side
- Because of
seccomp
, it may have compatible issues with some mruby gems - mruby doesn't have
Date
, useTime
instead - mruby doesn't have
BigDecimal
, you can use Shopify'sDecimal
instead - mruby is poor support timezone, you'd better handle it by yourself
- mruby engine is fast, usually it only costs 3 - 5ms depends on complexity, but it consume a lot of memory (~300k at least per process)
The input is expected to be a msgpack MAP
with three keys (Symbol): library
, sources
, input
:
library
: a msgpackBIN
set of MRuby instructions that will be fed directly to themruby-engine
input
: a msgpack formated payload for thesources
to digestsources
: a msgpackARRAY
ofARRAY
with two elements each (tuples):path
,source
; the actual code to be executed by the mruby-engine
The output is msgpack encoded as well; it is streamed to the consuming end though. Streamed items can be of different types.
Each element streamed is in the format of an ARRAY
of two elements, where the first is a Symbol
describing the element type:
measurement
: a msgpackARRAY
of two elements: aSymbol
describing the measurement, and anINT64
with the value in µs.output
: a msgpackMAP
with two entries (keys are symbols): **extracted
with whatever the script put in@output
, msgpack encoded; and **stdout
with aSTRING
containing whatever the script printed to "stdout".stat
: aMAP
keyed with symbols mapping to theirINT64
values
When the ESS fails to serve a request, it communicates the error back to the caller by returning a non-zero status code.
It can also report data about the error, in certain cases, over the pipe. In does so in returning a tuple, as an ARRAY
with the type being the symbol error
and the payload being a MAP
. The content of the map will vary, but it always will have a __type
symbol key that defines the other keys.
Run ./bin/rake
to build the project. This effectively runs the spec
target, which builds all libraries, the ESS and native tests; then runs all tests (native and Ruby).
To rebuild the entire project (which is useful when switching from one OS to another), use ./bin/rake mrproper
.
The sample script bin/sandbox
reads Ruby input from a file or stdin, executes it, and displays the results.
You can invoke ESS from your own Ruby code as follows:
result = ScriptCore.run(
input: {result: [26803196617, 0.475]}, # <1>
sources: [
["stdout", "@stdout_buffer = 'hello'"],
["foo", "@output = @input[:result]"], # <2>
],
instructions: nil, # <3>
timeout: 10.0, # <4>
instruction_quota: 100000, # <5>
instruction_quota_start: 1, # <6>
memory_quota: 8 << 20 # <7>
)
expect(result.success?).to be(true)
expect(result.output).to eq([26803196617, 0.475])
expect(result.stdout).to eq("hello")
- <1> invokes the ESS, with a map as the
input
(available as@input
in the sources) - <2> two "scripts" to be executed, one sets the
@stdout_buffer
to a value, the second returns the value associated with the key:result
of the map passed in in <1> - <3> some raw instructions that will be fed directly into MRuby; defaults to nil
- <4> a 10 second time quota to spawn, init, inject, eval and finally output the result back; defaults to 1 second
- <5> a 100k instruction limit that that the engine will execute; defaults to 100k
- <6> starts counting the instructions at index 1 of the
sources
array - <7> creates an 8 megabyte memory pool in which the script will run
Consists of our code base, plus seccomp
and msgpack
libraries, as well as the mruby
stuff. All in ext/enterprise_script_service
Note: lib seccomp
is omitted on Darwin.
Ruby code is in lib/
- GoogleTest tests are in
tests/
, which also includes the Google Test library. - RSpec tests are in
spec/
- There is a
CMakeLists.txt
that's mainly there for CLion support; we don't use cmake to build any of this. - You can use vagrant to bootstrap a VM to test under Linux while on Darwin; this is useful when testing
seccomp
.
git submodule update --init --recursive
$ vagrant up
$ vagrant ssh
vagrant@vagrant-ubuntu-bionic-64:~$ cd /vagrant
vagrant@vagrant-ubuntu-bionic-64:/vagrant$ bundle install
vagrant@vagrant-ubuntu-bionic-64:/vagrant$ git submodule init
vagrant@vagrant-ubuntu-bionic-64:/vagrant$ git submodule update
vagrant@vagrant-ubuntu-bionic-64:/vagrant$ bin/rake
Bug report or pull request are welcome.
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request
Please write unit test with your code if necessary.
The gem is available as open source under the terms of the MIT License.