vijos / jd4 Goto Github PK

View Code? Open in Web Editor NEW

93.0 16.0 31.0 405 KB

Judging daemon for programming contests

Home Page: https://vijos.org

License: GNU Affero General Public License v3.0

Python 91.19% Dockerfile 1.42% Cython 7.39%

linux online-judge sandbox daemon asyncio vijos python cython linux-namespaces cgroup

jd4's Introduction

Replaced By VJ4!

Repository Obsoleted

Why? We found that we were using PHP(the best language) as languages like Java in this project, which made us unhappy(but PM said that he was happy). Too long, too slow, not elegant.

Good News!

We are developing a totally new version of OpenVJ, written in Python, significantly reduced the code size while keeping high performance thanks to the new async feature provided by Python. It will provide similar functionalities as planned in OpenVJ (at least SaaS). It will be released under open source license to replace this repository soon. Keep an eye on it :-)

OpenVJ

OpenVJ is a next-generation open-source online judge (OJ) system. It is developed and maintained by Vijos, supported by:

High School Attached to Hunan Normal University
Beijing NO.80 High School
Shanghai Tongji University

Important Notice

The project is still under developing. DO NOT TRY TO DEPLOY IT.

Requirement

Linux / OS X
PHP >= 5.6
php-mongo
php-redis
php-mcrypt
MongoDB >= 3
Redis
Composer

License

GNU Affero General Public License (AGPL) V3

jd4's People

Contributors

Stargazers

Watchers

jd4's Issues

Add more language support for special judge.

It seems that the special judge supports .c only.

How to make jd4 talk with a unix socket file?

I started vj4 with

python3 -m vj4.server --listen=unix:/dev/shm/vj4.sock

and jd4 with

docker run --privileged --name jd4 \
     -v /opt/jd4/config/config.yaml:/root/.config/jd4/config.yaml \
     -v /opt/jd4/cache:/root/.cache/jd4 \
     -v /dev/shm/vj4.sock:/dev/shm/vj4.sock \
     vijos/jd4

nginx listens on port 80 and redirects to /dev/shm/vj4.sock

http service works well but if I set

server_url: http://172.17.0.1:80/

jd4 will report error:

[E 171005 03:58:05 daemon:171] 400, message='Invalid response status'
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 169, in daemon
    await gather(*done)
  File "/usr/lib/python3.5/asyncio/futures.py", line 381, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 310, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 294, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 240, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 150, in do_judge
    await session.judge_consume(JudgeHandler)
  File "/venv/lib/python3.5/site-packages/jd4/api.py", line 51, in judge_consume
    async with self.ws_connect(self.full_url('judge/consume-conn/websocket')) as ws:
  File "/venv/lib/python3.5/site-packages/aiohttp/client.py", line 603, in __aenter__
    self._resp = yield from self._coro
  File "/venv/lib/python3.5/site-packages/aiohttp/client.py", line 400, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 400, message='Invalid response status'

I guess jd4 needs to listen on the sock file directly, but how could I set config.yaml?

Thanks in advance!

Not showing whole compile error

example record: 58d0b308d3d8a17c654245e0

CGroup file not found

[E 170312 20:25:41 base_events:1090] Task exception was never retrieved
future: <Task finished coro=<accept_and_limit.<locals>.limit_task() done, defined at /home/iceboy/git/j
d4/jd4/case.py:42> exception=FileNotFoundError(2, 'No such file or directory')>
Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/case.py", line 45, in limit_task
    time_remain_ns = time_limit_ns - max(cgroup.cpu_usage_ns, idle_usage_ns())
  File "/home/iceboy/git/jd4/jd4/cgroup.py", line 85, in cpu_usage_ns
    return int(read_text_file(path.join(self.cpuacct_cgroup_dir, 'cpuacct.usage')))
  File "/home/iceboy/git/jd4/jd4/util.py", line 5, in read_text_file
    with open(file) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/cpuacct/jd4/_8g06b6j/cpuacct.us
age'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/iceboy/git/jd4/jd4/case.py", line 54, in limit_task
    while cgroup.kill():
  File "/home/iceboy/git/jd4/jd4/cgroup.py", line 72, in kill
    procs = self.procs
  File "/home/iceboy/git/jd4/jd4/cgroup.py", line 67, in procs
    map(int, read_text_file(path.join(self.cpuacct_cgroup_dir, 'cgroup.procs')).splitlines()),
  File "/home/iceboy/git/jd4/jd4/util.py", line 5, in read_text_file
    with open(file) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/cpuacct/jd4/_8g06b6j/cgroup.pro
cs'

Failed when config.yaml contains Chinese characters

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 5, in <module>
    from jd4.api import VJ4Session
  File "/venv/lib/python3.5/site-packages/jd4/api.py", line 7, in <module>
    from jd4.config import cookie_jar, save_cookies
  File "/venv/lib/python3.5/site-packages/jd4/config.py", line 22, in <module>
    config = _load_config()
  File "/venv/lib/python3.5/site-packages/jd4/config.py", line 17, in _load_config
    return yaml.load(file, Loader=yaml.RoundTripLoader)
  File "/venv/lib/python3.5/site-packages/ruamel/yaml/main.py", line 634, in load
    loader = Loader(stream, version, preserve_quotes=preserve_quotes)
  File "/venv/lib/python3.5/site-packages/ruamel/yaml/loader.py", line 59, in __init__
    Reader.__init__(self, stream, loader=self)
  File "/venv/lib/python3.5/site-packages/ruamel/yaml/reader.py", line 80, in __init__
    self.stream = stream  # type: Any  # as .read is called
  File "/venv/lib/python3.5/site-packages/ruamel/yaml/reader.py", line 125, in stream
    self.determine_encoding()
  File "/venv/lib/python3.5/site-packages/ruamel/yaml/reader.py", line 169, in determine_encoding
    self.update_raw()
  File "/venv/lib/python3.5/site-packages/ruamel/yaml/reader.py", line 253, in update_raw
    data = self.stream.read(size)
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 38: ordinal not in range(128)

How to modify the languages?

If I want to add or reduce some languages, how many files should I modify?
Both in vj4 and jd4. Thx.

SystemError

I have tried to use the quick start(Pre-built docker package).Now I have a new problem,I can't use the judge to test my code,it show that "error('unpack requires a bytes object of length 4',)".What's wrong with my setup?

Not responding after connection broken

[W 170403 01:49:55 api:56] Connection lost with code 1006
[I 170403 01:49:55 daemon:171] Retrying after 30 seconds
[E 170403 01:55:25 daemon:170] 
Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 163, in daemon
    await session.login_if_needed(config['uname'], config['password'])
  File "/home/iceboy/git/jd4/jd4/api.py", line 67, in login_if_needed
    await self.judge_noop()
  File "/home/iceboy/git/jd4/jd4/api.py", line 59, in judge_noop
    await self.get_json('judge/noop')
  File "/home/iceboy/git/jd4/jd4/api.py", line 40, in get_json
    params=kwargs) as response:
  File "/home/iceboy/.pyvenv/lib/python3.5/site-packages/aiohttp/client.py", lin
e 587, in __aenter__
    self._resp = yield from self._coro
  File "/home/iceboy/.pyvenv/lib/python3.5/site-packages/aiohttp/client.py", lin
e 274, in _request
    break
  File "/home/iceboy/.pyvenv/lib/python3.5/site-packages/aiohttp/helpers.py", li
ne 765, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
[I 170403 01:55:25 daemon:171] Retrying after 30 seconds

[E 170403 02:24:38 daemon:170] invalid content type text/html
Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 163, in daemon
    await session.login_if_needed(config['uname'], config['password'])
  File "/home/iceboy/git/jd4/jd4/api.py", line 67, in login_if_needed
    await self.judge_noop()
  File "/home/iceboy/git/jd4/jd4/api.py", line 59, in judge_noop
    await self.get_json('judge/noop')
  File "/home/iceboy/git/jd4/jd4/api.py", line 41, in get_json
    return await json_response_to_dict(response)
  File "/home/iceboy/git/jd4/jd4/api.py", line 19, in json_response_to_dict
    raise Exception('invalid content type ' + response.content_type)
Exception: invalid content type text/html
[I 170403 02:24:38 daemon:171] Retrying after 30 seconds

Change the way to detect the test data files

Related:
#71 (comment)
#71 (comment)

config file not found

I installed the vijos using https://github.com/vijos/vj4-docker.git (I changed to vijos/vj4:debian as described here vijos/vj4#534 ). But I found a problem like this:
#########################################
[E 191106 20:40:50 daemon:57] config file not found
Traceback (most recent call last):
File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 47, in do_record
await self.do_submission()
File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 74, in do_submission
await self.judge(cases_file, package)
File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 96, in judge
cases = list(read_cases(cases_file))
File "/venv/lib/python3.5/site-packages/jd4/case.py", line 295, in read_cases
raise FormatError('config file not found')
jd4.error.FormatError: config file not found
#########################################

If a install it on AliYun centos7 machine, it works. But if I install it on the OXS OS Mojave system it gives me this error. I have a VM centos7 machine also running on this MAC. it also give me this error. What's the problem? if it's a bug?

A Bug?

It seems like a bug...

Java code encoding

e.g. https://vijos.org/records/59020e55d3d8a165f9afb92c

help~docker run error

BrokenPipeError

[E 170312 20:20:17 daemon:54] [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 44, in handle
    await self.submission()
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 72, in submission
    await self.judge(cases_file, package)
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 105, in judge
    status, score, time_usage_ns, memory_usage_bytes, stderr = await case.judge(sandbox, package)
  File "/home/iceboy/git/jd4/jd4/case.py", line 91, in judge
    cgroup_file='/in/cgroup'))
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/iceboy/git/jd4/jd4/case.py", line 121, in do_stdin
    copyfileobj(src, dst, CHUNK_SIZE)
  File "/usr/local/lib/python3.5/shutil.py", line 76, in copyfileobj
    fdst.write(buf)
BrokenPipeError: [Errno 32] Broken pipe

This might because the user program exited before reading all input. We should ignore this error.

Parallel bugs

Between 61da7a3 and da7ca34.

Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 42, in handle
    await self.submission()
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 70, in submission
    await self.judge(cases_file, package)
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 119, in judge
    status, score, time_usage_ns, memory_usage_bytes, stderr = await judge_task
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 111, in pool_judge
    return await case.judge(sandbox, package)
  File "/home/iceboy/git/jd4/jd4/case.py", line 72, in judge
    self.process_limit))
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/iceboy/git/jd4/jd4/cgroup.py", line 135, in wait_cgroup
    await sleep(.001)
TypeError: object NoneType can't be used in 'await' expression

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 111, in pool_judge
    return await case.judge(sandbox, package)
  File "/home/iceboy/git/jd4/jd4/case.py", line 72, in judge
    self.process_limit))
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/iceboy/git/jd4/jd4/case.py", line 98, in do_stdin
    with self.open_input() as src, open(stdin_file, 'wb') as dst:
  File "/usr/local/lib/python3.5/zipfile.py", line 1261, in open
    fheader = zef_file.read(sizeFileHeader)
  File "/usr/local/lib/python3.5/zipfile.py", line 662, in read
    self._file.seek(self._pos)
ValueError: seek of closed file

Task exception was never retrieved
future: <Task finished coro=<JudgeHandler.judge.<locals>.pool_judge() done, defined at /home/iceboy/git/jd4/jd4/daemon.py:108> exception=ValueError('seek of closed file',)>

Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 42, in handle
    await self.submission()
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 68, in submission
    cases_file, package = await gather(cache_open(self.session, domain_id, pid), self.build())
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 90, in build
    package, message = await build_fn(sandbox, self.request.pop('code').encode())
  File "/home/iceboy/git/jd4/jd4/langs.py", line 22, in _compiler_build
    read_pipe(output_file, _MAX_OUTPUT))
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/iceboy/git/jd4/jd4/compile.py", line 84, in build
    status = await sandbox.marshal(lambda: self.do_build(
  File "/home/iceboy/git/jd4/jd4/sandbox.py", line 62, in marshal
    length, = unpack('I', await self.reader.read(4))
  File "/usr/local/lib/python3.5/asyncio/streams.py", line 482, in read
    yield from self._wait_for_data('read')
  File "/usr/local/lib/python3.5/asyncio/streams.py", line 419, in _wait_for_data
    'already waiting for incoming data' % func_name)
RuntimeError: read() called while another coroutine is already waiting for incoming data

main_task/other_task pattern still incorrect

if other_task throws an exception, but the parent coroutine is waiting for main_task, it deadlocks.

[Enhancement] suggest adding arguments when running custom judge

maybe like this:

./check <Input_File> <Output_File> <Answer_File> [<Result_File> [-appes]],

By doing so, it will be easier to use famous special judge libs, e.g. testlib.h.

waiting?

what happened to https://vijos.org/d/newbzoj/records/590d1f60d3d8a13210994b45

None object is not subscriptable

e.g. https://vijos.org/discuss/5109fdea4e41125014000bb5

Apr  5 14:35:38 vj4 python3.5[385]: #033[31m[E 170405 14:35:38 web_protocol:358]#033[0m Error handling request
Apr  5 14:35:38 vj4 python3.5[385]: Traceback (most recent call last):
Apr  5 14:35:38 vj4 python3.5[385]:   File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_protocol.py", line 417, in start
Apr  5 14:35:38 vj4 python3.5[385]:     resp = yield from self._request_handler(request)
Apr  5 14:35:38 vj4 python3.5[385]:   File "/usr/local/lib/python3.5/dist-packages/aiohttp/web.py", line 289, in _handle
Apr  5 14:35:38 vj4 python3.5[385]:     resp = yield from handler(request)
Apr  5 14:35:38 vj4 python3.5[385]:   File "/home/vijos/vj4/vj4/handler/base.py", line 217, in __iter__
Apr  5 14:35:38 vj4 python3.5[385]:     yield from super(Handler, self).__iter__()
Apr  5 14:35:38 vj4 python3.5[385]:   File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_urldispatcher.py", line 657, in __iter__
Apr  5 14:35:38 vj4 python3.5[385]:     resp = yield from method()
Apr  5 14:35:38 vj4 python3.5[385]:   File "/home/vijos/vj4/vj4/handler/discussion.py", line 156, in get
Apr  5 14:35:38 vj4 python3.5[385]:     (vnode['title'], node_url(self, 'discussion_node', discussion.node_id(ddoc))),
Apr  5 14:35:38 vj4 python3.5[385]: TypeError: 'NoneType' object is not subscriptable#033[0m

Convert line endings

Not responding to PING during judgement causes WebSocket connections to close during long test cases.

The WebSocket server will close the connection if it does not receive any PONG within an expiry time.

Add special judge

Wow... Only 8 opening issues here!

System wide time/memory limit

Case packages are user uploaded. User could set time/memory limit to a large value, or add large number of cases to perform DOS attack.

docker run failed for Operation not permitted

As run the docker, it failed unexpectedly.

Long story for short:
(1) command

docker run --privileged \
    -v ~/.config/jd4/config.yaml:/root/.config/jd4/config.yaml \
    -v ~/.cache/jd4:/root/.cache/jd4 \
    vijos/jd4

(2)error

File "/root/downloads/yan/jd4/jd4/sandbox.py", line 121, in _handle_child
    create_namespace()
  File "jd4/_sandbox.pyx", line 35, in jd4._sandbox.create_namespace
    write_text_file('/proc/self/uid_map', '1000 {} 1'.format(host_euid))
  File "/root/downloads/yanzm/jd4/jd4/util.py", line 44, in write_text_file
    f.write(text)
PermissionError: [Errno 1] Operation not permitted

(3)what have been done
I checked the /proc/self ， I found the dir is not permitted to write。

I tried use root or common user, the same error occured.

thanks for any help.

Chinese output in Java encoded incorrectly

migrated from vijos/vj4#487

Add sumbit answer problem support

Don't seem to be covered by #38

VJ4Error

[I 170317 12:14:47 daemon:68] Submission: system, 1790, 58cb62b7d3d8a12d66e8f0c7
[I 170317 12:14:47 api:85] Getting problem data: system, 1790
[E 170317 12:14:48 daemon:52] 16
Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 42, in handle
    await self.submission()
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 71, in submission
    with await cases_file_task as cases_file:
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 360, in __iter__
    return self.result()  # May raise too.
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/iceboy/git/jd4/jd4/cache.py", line 23, in cache_open
    await session.problem_data(domain_id, pid, file_path)
  File "/home/iceboy/git/jd4/jd4/api.py", line 95, in problem_data
    *error.get('args', []))
jd4.api.VJ4Error: 16

hang

stack trace:

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    "__main__", mod_spec)
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    exec(code, run_globals)
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
      self.run_forever()
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    "__main__", mod_spec)
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
    self._callback(*self._args)
    self._run_once()
    self._run_once()
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
    "__main__", mod_spec)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    result = coro.send(None)
    self.run_forever()
    self.run_forever()
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    result = coro.send(None)
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
    return self._sock.recv_into(b)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
KeyboardInterrupt
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
KeyboardInterrupt
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
    self._run_once()
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    return self._sock.recv_into(b)
    return self._sock.recv_into(b)
KeyboardInterrupt
    self._run_once()
KeyboardInterrupt
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
    handle._run()
    handle._run()
    handle._run()
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    return self._sock.recv_into(b)
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    self._callback(*self._args)
KeyboardInterrupt
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
    result = coro.send(None)
    self._run_once()
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    return self._sock.recv_into(b)
KeyboardInterrupt
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    return self._sock.recv_into(b)
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
KeyboardInterrupt
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
KeyboardInterrupt
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
KeyboardInterrupt
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    return self._sock.recv_into(b)
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
    handle._run()
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 11, in <module>
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 102, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
    handle._run()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    handle._run()
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
    result = coro.send(None)
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
KeyboardInterrupt
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 68, in _handle_child
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 60, in _init
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 98, in create_sandboxes
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 61, in _handle_child
    return self._sock.recv_into(b)
KeyboardInterrupt
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 175, in <module>
    get_event_loop().run_until_complete(daemon())
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1388, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/lib/python3.5/selectors.py", line 445, in select
    fd_event_list = self._epoll.poll(timeout, max_ev)
KeyboardInterrupt
KeyboardInterrupt

Add integration test per language using docker

Create setup.py to allow install

also build cython module

Migrate to aiohttp>=3

Judge time/memory limit for custom judge.

[Error] BrokenPipeError(32, 'Broken pipe') On website

Env:

Centos 7
Docker version 19.03.2, build 6a30dfc
jd4 build by default Dockerfile
vj4 build by default Dockerfile
judge user set by default

docker-compuse.yml:

version: '3'
services:
    web:
        #build: .
        image: philogag/vj4:office
        restart: always
        command: vj4.server
        env_file: .env
        volumes:
            - "./data/run:/var/run/vj4"
        links:
            - mongodb
            - rabbitmq
        ports: [ "8888:8888" ]
        depends_on:
            - rabbitmq
            - mongodb

    # Before uncomment these lines, you should have created necessary files
    # like config.yaml for it.
    # See README.md for instructions.
    #-------------------------------------------------------------------------------
    judge:
        restart: always
        privileged: true
        #image: vijos/jd4
        image: philogag/jd4:office
        #command: bash -c "source /venv/bin/activate && \ python3 -m jd4.daemon"
        volumes:
            - "./data/judge/config.yaml:/root/.config/jd4/config.yaml"
            - "./data/judge/cache:/root/.cache/jd4"
            - "./data/judge/langs.yaml:/root/.config/jd4/langs.yaml"
        links:
            - web
            # - web_backend # uncomment this only if you have this service
        depends_on:
            - web

Jd4 log:

when start docker

[D 190923 09:18:38 selector_events:65] Using selector: EpollSelector
[I 190923 09:18:38 pool:22] Using parallelism: 2
PermissionError: [Errno 1] Operation not permitted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 6, in <module>
    from jd4.case import read_cases
  File "/venv/lib/python3.5/site-packages/jd4/case.py", line 15, in <module>
    from jd4.compile import build
  File "/venv/lib/python3.5/site-packages/jd4/compile.py", line 13, in <module>
    from jd4.pool import get_sandbox, put_sandbox
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 29, in <module>
    _init()
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 23, in _init
    sandboxes_task = create_sandboxes(parallelism)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 161, in create_sandboxes
    _handle_child(child_socket, root_dir, in_dir, out_dir)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 120, in _handle_child
    create_namespace()
  File "jd4/_sandbox.pyx", line 35, in jd4._sandbox.create_namespace
  File "/venv/lib/python3.5/site-packages/jd4/util.py", line 44, in write_text_file
    f.write(text)
PermissionError: [Errno 1] Operation not permitted
PermissionError: [Errno 1] Operation not permitted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 6, in <module>
    from jd4.case import read_cases
  File "/venv/lib/python3.5/site-packages/jd4/case.py", line 15, in <module>
    from jd4.compile import build
  File "/venv/lib/python3.5/site-packages/jd4/compile.py", line 13, in <module>
    from jd4.pool import get_sandbox, put_sandbox
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 29, in <module>
    _init()
  File "/venv/lib/python3.5/site-packages/jd4/pool.py", line 23, in _init
    sandboxes_task = create_sandboxes(parallelism)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 161, in create_sandboxes
    _handle_child(child_socket, root_dir, in_dir, out_dir)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 120, in _handle_child
    create_namespace()
  File "jd4/_sandbox.pyx", line 35, in jd4._sandbox.create_namespace
  File "/venv/lib/python3.5/site-packages/jd4/util.py", line 44, in write_text_file
    f.write(text)
PermissionError: [Errno 1] Operation not permitted
[I 190923 09:18:38 cgroup:29] Initializing cgroup: /sys/fs/cgroup/cpuacct/jd4, /sys/fs/cgroup/memory/jd4, /sys/fs/cgroup/pids/jd4
[I 190923 09:18:38 api:88] Login
[I 190923 09:18:38 daemon:133] Update problem data
[D 190923 09:18:38 daemon:137] Invalidated system/1000
[I 190923 09:18:38 api:59] Connected

when submit code

[I 190923 09:18:38 daemon:70] Submission: system, 1000, 5d831827fd777a00018ab9de
[I 190923 09:18:38 api:107] Getting problem data: system, 1000
[E 190923 09:18:38 daemon:57] unpack requires a bytes object of length 4
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 47, in do_record
    await self.do_submission()
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 72, in do_submission
    package = await self.build()
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 86, in build
    package, message, _, _ = await shield(build(self.lang, self.code.encode()))
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/venv/lib/python3.5/site-packages/jd4/compile.py", line 153, in build
    return await build_fn(code)
  File "/venv/lib/python3.5/site-packages/jd4/compile.py", line 122, in _compiler_build
    await compiler.prepare(sandbox, code)
  File "/venv/lib/python3.5/site-packages/jd4/compile.py", line 79, in prepare
    await sandbox.reset()
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 43, in reset
    self.call(SANDBOX_RESET_CHILD))
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/sandbox.py", line 47, in call
    length, = unpack('I', await self.reader.read(4))
struct.error: unpack requires a bytes object of length 4

should delete file if error occur during download

^[[32m[I 170508 15:40:26 daemon:76]^[[0m Submission: newbzoj, 590c9882d3d8a13210993757, 5910916ad3d8a1022827f0c5^[[0m
^[[32m[I 170508 15:40:26 api:80]^[[0m Getting problem data: newbzoj, 590c9882d3d8a13210993757^[[0m
^[[31m[E 170508 15:40:57 daemon:171]^[[0m Response payload is not completed
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 169, in daemon
    await gather(*done)
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 150, in do_judge
    await session.judge_consume(JudgeHandler)
  File "/venv/lib/python3.5/site-packages/jd4/api.py", line 55, in judge_consume
    await handler_type(self, request, ws).handle()
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 32, in handle
    await self.do_record()
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 50, in do_record
    await self.do_submission()
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 79, in do_submission
    with await cases_file_task as cases_file:
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/venv/lib/python3.5/site-packages/jd4/cache.py", line 23, in cache_open
    await session.problem_data(domain_id, pid, file_path)
  File "/venv/lib/python3.5/site-packages/jd4/api.py", line 96, in problem_data
    buffer = await response.content.read(CHUNK_SIZE)
  File "/venv/lib/python3.5/site-packages/aiohttp/streams.py", line 542, in read
    return (yield from super().read(n))
  File "/venv/lib/python3.5/site-packages/aiohttp/streams.py", line 298, in read
  File "/venv/lib/python3.5/site-packages/aiohttp/streams.py", line 542, in read
    return (yield from super().read(n))
  File "/venv/lib/python3.5/site-packages/aiohttp/streams.py", line 298, in read
    yield from self._wait('read')
  File "/venv/lib/python3.5/site-packages/aiohttp/streams.py", line 226, in _wait
    yield from waiter
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed^[[0m
^[[32m[I 170508 15:40:57 daemon:172]^[[0m Retrying after 30 seconds^[[0m
^[[32m[I 170508 15:41:27 api:68]^[[0m Session is valid^[[0m
^[[32m[I 170508 15:41:27 daemon:140]^[[0m Update problem data^[[0m
^[[32m[I 170508 15:41:28 api:52]^[[0m Connected^[[0m
^[[32m[I 170508 15:41:28 daemon:76]^[[0m Submission: newbzoj, 590c9882d3d8a13210993757, 5910916ad3d8a1022827f0c5^[[0m
^[[31m[E 170508 15:41:28 daemon:63]^[[0m File is not a zip file
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 50, in do_record
    await self.do_submission()
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 80, in do_submission
    await self.judge(cases_file, package)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 103, in judge
    cases = list(read_legacy_cases(cases_file))
  File "/venv/lib/python3.5/site-packages/jd4/case.py", line 116, in read_legacy_cases
    zip_file = ZipFile(file)
  File "/usr/lib/python3.5/zipfile.py", line 1026, in __init__
    self._RealGetContents()
  File "/usr/lib/python3.5/zipfile.py", line 1094, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file^[[0m

how to use it with vj4

as the title

Unclosed connection when connection breaks

[W 170316 21:12:29 api:61] Connection lost with code 1006
[I 170316 21:12:29 daemon:173] Retrying after 30 seconds
[I 170316 21:12:59 api:73] Session is valid
[I 170316 21:12:59 daemon:150] Update problem data
[I 170316 21:13:00 api:57] Connected
[W 170316 21:13:20 api:61] Connection lost with code 1006
[I 170316 21:13:20 daemon:173] Retrying after 30 seconds
[I 170316 21:13:50 api:73] Session is valid
[I 170316 21:13:50 daemon:150] Update problem data
[I 170316 21:13:51 api:57] Connected
[W 170316 21:16:52 api:61] Connection lost with code 1006
[I 170316 21:16:52 daemon:173] Retrying after 30 seconds
[I 170316 21:17:22 api:73] Session is valid
[I 170316 21:17:22 daemon:150] Update problem data
[I 170316 21:17:23 api:57] Connected
[I 170316 21:19:00 daemon:68] Submission: system, 1294, 58ca90c4d3d8a12d66e8f030
[E 170316 21:19:02 base_events:1090] Unclosed connection
client_connection: Connection<('vijos.org', 443, True)>
[E 170316 21:19:02 base_events:1090] Unclosed response
client_response: <ClientResponse(https://vijos.org/judge/consume-conn/websocket) [101 Switching Protocols]>
<CIMultiDictProxy('Server': 'nginx/1.10.3', 'Date': 'Thu, 16 Mar 2017 13:13:00 GMT', 'Content-Type': 'application/octet-stream', 'Connection': 'upgrade', 'Upgrade': 'websocket', 'Sec-Websocket-Accept': 'zwlDyAfAjceKj2doPBtG+lSQSNA=')>

[E 170316 21:19:02 base_events:1090] Unclosed connection
client_connection: Connection<('vijos.org', 443, True)>
[E 170316 21:19:02 base_events:1090] Unclosed response
client_response: <ClientResponse(https://vijos.org/judge/consume-conn/websocket) [101 Switching Protocols]>
<CIMultiDictProxy('Server': 'nginx/1.10.3', 'Date': 'Thu, 16 Mar 2017 13:13:51 GMT', 'Content-Type': 'application/octet-stream', 'Connection': 'upgrade', 'Upgrade': 'websocket', 'Sec-Websocket-Accept': 'MdNJA9G0y/G0h7lULfd00UexAM0=')>

Bug

Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 44, in handle
    await self.submission()
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 72, in submission
    await self.judge(cases_file, package)
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 105, in judge
    status, score, time_usage_ns, memory_usage_bytes, stderr = await case.judge(
sandbox, package)
  File "/home/iceboy/git/jd4/jd4/case.py", line 62, in judge
    self.process_limit))
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
concurrent.futures._base.CancelledError

Exception ignored in: 'jd4._compare.StreamReader.read'
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/zipfile.py", line 844, in read
    data = self._read1(n)
  File "/usr/local/lib/python3.5/zipfile.py", line 912, in _read1
[E 170315 10:50:44 daemon:54] [Errno 2] No such file or directory: 'foo'
Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 44, in handle
    await self.submission()
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 72, in submission
    await self.judge(cases_file, package)
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 105, in judge
    status, score, time_usage_ns, memory_usage_bytes, stderr = await case.judge(
sandbox, package)
  File "/home/iceboy/git/jd4/jd4/case.py", line 62, in judge
    self.process_limit))
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/iceboy/git/jd4/jd4/compile.py", line 32, in execute
    return await sandbox.marshal(lambda: self.do_execute(
  File "/home/iceboy/git/jd4/jd4/sandbox.py", line 65, in marshal
    raise err
FileNotFoundError: [Errno 2] No such file or directory: 'foo'
    data += self._read2(n - len(data))
  File "/usr/local/lib/python3.5/zipfile.py", line 944, in _read2
    data = self._fileobj.read(n)
  File "/usr/local/lib/python3.5/zipfile.py", line 662, in read
    self._file.seek(self._pos)
ValueError: seek of closed file

Run error

[E 180626 15:42:39 web_protocol:356] Error handling request
Traceback (most recent call last):
File "/home/wqiu/.local/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 416, in start
resp = yield from self._request_handler(request)
File "/home/wqiu/.local/lib/python3.6/site-packages/aiohttp/web.py", line 325, in _handle
resp = yield from handler(request)
File "/home/wqiu/.local/lib/python3.6/site-packages/aiohttp/web_middlewares.py", line 93, in impl
return (yield from handler(request))
File "/home/wqiu/.local/lib/python3.6/site-packages/sockjs/route.py", line 161, in websocket
session = self.manager.get(sid, True, request=request)
File "/home/wqiu/work/cqrd/codecampus/vj4/vj4/app.py", line 110, in get
timeout=self.timeout, loop=self.loop, debug=self.debug))
File "/home/wqiu/work/cqrd/codecampus/vj4/vj4/handler/base.py", line 338, in init
super(Connection, self).init(*args, **kwargs)
TypeError: init() missing 1 required positional argument: 'request'

A 'System Error' occurred when using Vijos & jd4

I configured vijos (including jd4) on my own server, after submitting a program, it showed system error, prompting me BrokenPipeError(32, 'Broken pipe').

My code: (as you see, it's A+B problem)

//cpp
#include<cstdio>

using namespace std;

int a,b;

int main()
{
	scanf("%d%d",&a,&b);
	printf("%d\n",a+b);
	return 0;
}

I couldn't find out what's wrong with jd4, so I come for help.

WSServerHandshakeError: 400

It seems not easy to build up the whole env of vj4. Let me introduce my problem brifly.
1)on one dev machine, run the vj4 server with python3 -m vj4.server --debug --listen http://0.0.0.0:8888, open port 8888.
2)on another machine, use docker to start the jd4 server, it went wrong with error output:

[D 180604 06:04:29 selector_events:65] Using selector: EpollSelector
[I 180604 06:04:29 pool:22] Using parallelism: 2
[I 180604 06:04:30 api:76] Session is valid
[I 180604 06:04:30 daemon:133] Update problem data
[E 180604 06:04:32 daemon:164] 400, message='Invalid response status'
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 162, in daemon
    await gather(*done)
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 143, in do_judge
    await session.judge_consume(JudgeHandler)
  File "/venv/lib/python3.5/site-packages/jd4/api.py", line 59, in judge_consume
    async with self.ws_connect(self.full_url('judge/consume-conn/websocket')) as ws:
  File "/venv/lib/python3.5/site-packages/aiohttp/client.py", line 690, in __aenter__
    self._resp = yield from self._coro
  File "/venv/lib/python3.5/site-packages/aiohttp/client.py", line 465, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 400, message='Invalid response status'

I found almost the similar problem at issue 49, BUT i did't use
nginx. My config.yaml ：
server_url: http://140.82.**.***:8888
uname: judge
password: ****

thanks for any help in advance.

New cases file format

Use yaml as config
Case insensitive
Recursive
Cover all existing use cases
Automatically convertible from "legacy case"
Support special judge

Initial thinking:

time: 1s
memory: 16m
score: 10
children:
- input: input/input0.txt
  output: output/output0.txt
- input: input/input1.txt
  output: output/output1.txt
- judge:
    file: judge.cc
    lang: cc
    time: 5s
    memory: 64m
    data: ['data/state', 'data/transition']
  score: 30
  children:
  - input: input/input2.txt
    data: data/state
    memory: 32m
  - input: input/input3.txt
    data: data/state
    score: 50
    memory: 2048m

This is an extreme case. Normal config should be much simpler.

Make `extra` for custom judge configurable

python error

sh: /dev/null: No such file or directory

jd4 docker connect error: web:80 ssl:False [Connect call failed('127.0.0.1', 80)]

I'm using docker to bring up jd4 image. But jd4 never connects, saying:
(I changed the port to 80 in docker-compose.yaml)

Traceback (most recent call last):
File "/venv/lib/python3.5/site-packages/jd4/daemon.py", line 157, in daemon
await session.login_if_needed(config['uname'], config['password'])
File "/venv/lib/python3.5/site-packages/jd4/api.py", line 93, in login_if_needed
await self.judge_noop()
File "/venv/lib/python3.5/site-packages/jd4/api.py", line 85, in judge_noop
await self.get_json('judge/noop')
File "/venv/lib/python3.5/site-packages/jd4/api.py", line 47, in get_json
params=kwargs) as response:
File "/venv/lib/python3.5/site-packages/aiohttp/client.py", line 690, in aenter
self._resp = yield from self._coro
File "/venv/lib/python3.5/site-packages/aiohttp/client.py", line 267, in _request
conn = yield from self._connector.connect(req)
File "/venv/lib/python3.5/site-packages/aiohttp/connector.py", line 402, in connect
proto = yield from self._create_connection(req)
File "/venv/lib/python3.5/site-packages/aiohttp/connector.py", line 749, in _create_connection
_, proto = yield from self._create_direct_connection(req)
File "/venv/lib/python3.5/site-packages/aiohttp/connector.py", line 860, in _create_direct_connection
raise last_exc
File "/venv/lib/python3.5/site-packages/aiohttp/connector.py", line 832, in _create_direct_connection
req=req, client_error=client_error)
File "/venv/lib/python3.5/site-packages/aiohttp/connector.py", line 804, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host web:80 ssl:False [Connect call failed ('127.0.0.1', 80)]
[I 230227 07:13:41 daemon:165] Retrying after 30 seconds
[E 230227 07:14:11 daemon:164] Cannot connect to host web:80 ssl:False [Connect call failed ('127.0.0.1', 80)]
Traceback (most recent call last):
File "/venv/lib/python3.5/site-packages/aiohttp/connector.py", line 797, in _wrap_create_connection
return (yield from self._loop.create_connection(*args, **kwargs))
File "/usr/lib/python3.5/asyncio/base_events.py", line 775, in create_connection
raise exceptions[0]
File "/usr/lib/python3.5/asyncio/base_events.py", line 762, in create_connection
yield from self.sock_connect(sock, address)
File "/usr/lib/python3.5/asyncio/selector_events.py", line 451, in sock_connect
return (yield from fut)
File "/usr/lib/python3.5/asyncio/futures.py", line 380, in iter
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/selector_events.py", line 481, in _sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 80)

Anyone having the same problem?

Try to reduce execution time of trivial programs

https://vijos.org/discuss/5922c730d3d8a17e55ed6885

deadlock

[I 170312 19:45:32 api:85] Getting problem data: orz_yanQval, 58c53495d3d8a16c62a244e2
[E 170312 19:45:32 daemon:54] ('orz_yanQval', '58c53495d3d8a16c62a244e2')
Traceback (most recent call last):
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 44, in handle
    await self.submission()
  File "/home/iceboy/git/jd4/jd4/daemon.py", line 70, in submission
    cases_file, package = await gather(cache_open(self.session, domain_id, pid), self.build())
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/iceboy/git/jd4/jd4/cache.py", line 22, in cache_open
    await session.problem_data(domain_id, pid, file_path)
  File "/home/iceboy/git/jd4/jd4/api.py", line 95, in problem_data
    *error.get('args', []))
jd4.api.VJ4Error: ('orz_yanQval', '58c53495d3d8a16c62a244e2')

after getting this, the next time this records hits jd4 causes a deadlock.