dynilib / submission Goto Github PK

Hi I am running through your setup process for the first time and I have been getting stuck on creating the anaconda environment. The issue is when I run

docker build -t submission web/

The output is

Step 12/18 : RUN conda env create -f /tmp/env.yml
 ---> Running in 072e3598975f
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

ResolvePackageNotFound:
  - libxml2==2.9.8=0
  - libiconv==1.15=0

Not entirely sure why this is happening considering I thought Docker was more resistant to these issues.

As a workaround to get the server to build, what I did was removed these packages from env.yml and then installed them via apt instead in the dockerfile. So it would look like

RUN apt-get install libxml2 libc6 #libiconv is contained in libc6

After I submit the csv file to the backend, the page would jump to 502 Bad Gateway nginx/1.15.8. But I can find the upload file in /submition/instance/upload. Meanwhile, when I click View Score and the page shows Unknown error. The logs are below
File "./submission/views.py", line 178, in
dates = sorted(list({s.submitted_on.date() for s in submissions}))
AttributeError: 'NoneType' object has no attribute 'date'

The plot of scores stopped loading, and it's because the browser (firefox or chromium) reports "Invalid JSON string".

10:22:28.696 Error: Invalid JSON string:     
        {   
          "cols": [
                {"id":"","label":"Date","pattern":"","type":"datetime"},
                {"id":"","label":"testmonkey","pattern":"","type":"number"},{"id":"","label":"Comment","pattern":"","type":"string","role":"tooltip","p":{"html":true}},{"id":"","label":"topel","pattern":"","type":"number"},{"id":"","label":"Comment","pattern":"","type":"string","role":"tooltip","p":{"html":true}},{"id":"","label":"adavanne","pattern":"","type":"number"},{"id":"","label":"Comment","pattern":"","type":"string","role":"tooltip","p":{"html":true}},{"id":"","label":"toan","pattern":"","type":"number"},{"id":"","label":"Comment","pattern":"","type":"string","role":"tooltip","p":{"html":true}},{"id":"","label":"tsalem","pattern":"","type":"number"},{"id":"","label":"Comment","pattern":"","type":"string","role":"tooltip","p":{"html":true}},{"id":"","label":"ymchiqq","pattern":"","type":"number"},{"id":"","label":"Comment","pattern":"","type 1 jsapi_compiled_format_module.js:116:732
	gvjs_xe https://www.gstatic.com/charts/45.1/js/jsapi_compiled_format_module.js:116:732
	gvjs_Je https://www.gstatic.com/charts/45.1/js/jsapi_compiled_format_module.js:120:351
	gvjs_S https://www.gstatic.com/charts/45.1/js/jsapi_compiled_default_module.js:84:537
	plot_data http://lsis-argo.lsis.org/scores:85:16
	.success http://lsis-argo.lsis.org/scores:73:21
	b.Callbacks/c http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js:3:7852
	b.Callbacks/p.fireWith http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js:3:8658
	k http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js:5:14013
	.send/r http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js:5:18226
	onStopRequest resource://gre/modules/WebRequest.jsm:221:18

...I discovered that it was because one user entered a comment containing newline \r characters. I edited it to remove those characters which fixed it for now. But the JSON output probably needs to be filtered to remove newlines from comments, perhaps?

It is already in the database scheme, just add the field in the form and show it on the score graph.

We'll need to grab all the submission data when the contest closes. There are over 100 submissions, so a page-limit of 20 items per page will make this difficult.

There's a simple fix: set can_set_page_size = True in the config of the appropriate AdminModelView

Also please could you set can_export = True which will make downloading the result data much easier.

I haven't implemented this myself because I don't know quite which class I need to modify... looks like it needs a new class SubmissionAdmin(AdminModelView)?

Graph currently taking about 8 seconds to appear, simply because it takes that long for _get_submissions to construct a response when there are a couple of hundred submissions. 8 seconds is rather uncomfortable in web usability.

root@test-ubuntu:~/submission# docker logs submission
/usr/lib/python3/dist-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
self.warnings.warn(
2024-01-03 10:30:53,596 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-01-03 10:30:53,596 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2024-01-03 10:30:53,598 INFO RPC interface 'supervisor' initialized
2024-01-03 10:30:53,598 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-01-03 10:30:53,598 INFO supervisord started with pid 1
2024-01-03 10:30:54,600 INFO spawned: 'quit_on_failure' with pid 7
2024-01-03 10:30:54,602 INFO spawned: 'nginx' with pid 8
2024-01-03 10:30:54,603 INFO spawned: 'uwsgi' with pid 9
2024-01-03 10:30:54,604 INFO success: nginx entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-01-03 10:30:54,604 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024/01/03 10:30:54 [notice] 8#8: using the "epoll" event method
2024/01/03 10:30:54 [notice] 8#8: nginx/1.23.2
2024/01/03 10:30:54 [notice] 8#8: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2024/01/03 10:30:54 [notice] 8#8: OS: Linux 4.15.0-176-generic
2024/01/03 10:30:54 [notice] 8#8: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/01/03 10:30:54 [notice] 10#10: start worker processes
2024/01/03 10:30:54 [notice] 10#10: start worker process 11
2024/01/03 10:30:54 [notice] 10#10: start worker process 12
2024-01-03 10:30:54,608 INFO exited: nginx (exit status 0; expected)
[uWSGI] getting INI configuration from /myapp/uwsgi.ini
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
cheaper = 2
ini = /myapp/uwsgi.ini
module = submission
callable = app
processes = 16
ini = /etc/uwsgi/uwsgi.ini
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
;end of configuration

*** Starting uWSGI 2.0.21 (64bit) on [Wed Jan 3 10:30:54 2024] ***
compiled with version: 10.2.1 20210110 on 25 November 2022 15:10:22
os: Linux-4.15.0-176-generic #185-Ubuntu SMP Tue Mar 29 17:40:04 UTC 2022
nodename: fd786c38bcd3
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 16
current working directory: /myapp
detected binary path: /usr/local/bin/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.6.15 (default, Dec 21 2021, 12:03:22) [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x556ee4966840
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 1239640 bytes (1210 KB) for 16 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x556ee4966840 pid: 9 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 9)
spawned uWSGI worker 1 (pid: 28, cores: 1)
spawned uWSGI worker 2 (pid: 29, cores: 1)
running "unix_signal:15 gracefully_kill_them_all" (master-start)...
2024-01-03 10:30:56,370 INFO success: quit_on_failure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-01-03 10:30:56,370 WARN received SIGTERM indicating exit request
2024-01-03 10:30:56,370 INFO waiting for quit_on_failure, uwsgi to die
Wed Jan 3 10:30:56 2024 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 28)...
Gracefully killing worker 2 (pid: 29)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
goodbye to uWSGI.
2024-01-03 10:30:58,377 INFO stopped: uwsgi (exit status 0)
2024-01-03 10:30:59,378 INFO waiting for quit_on_failure to die
2024-01-03 10:30:59,379 INFO stopped: quit_on_failure (terminated by SIGTERM)
/usr/lib/python3/dist-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
self.warnings.warn(
2024-01-03 10:41:21,400 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-01-03 10:41:21,401 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2024-01-03 10:41:21,417 INFO RPC interface 'supervisor' initialized
2024-01-03 10:41:21,417 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-01-03 10:41:21,417 INFO supervisord started with pid 1
2024-01-03 10:41:22,419 INFO spawned: 'quit_on_failure' with pid 7
2024-01-03 10:41:22,421 INFO spawned: 'nginx' with pid 8
2024-01-03 10:41:22,422 INFO spawned: 'uwsgi' with pid 9
2024-01-03 10:41:22,467 INFO success: nginx entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-01-03 10:41:22,467 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024/01/03 10:41:22 [notice] 8#8: using the "epoll" event method
2024/01/03 10:41:22 [notice] 8#8: nginx/1.23.2
2024/01/03 10:41:22 [notice] 8#8: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2024/01/03 10:41:22 [notice] 8#8: OS: Linux 4.15.0-213-generic
2024/01/03 10:41:22 [notice] 8#8: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/01/03 10:41:22 [notice] 10#10: start worker processes
2024/01/03 10:41:22 [notice] 10#10: start worker process 11
2024/01/03 10:41:22 [notice] 10#10: start worker process 12
2024/01/03 10:41:22 [notice] 10#10: start worker process 13
2024/01/03 10:41:22 [notice] 10#10: start worker process 14
2024/01/03 10:41:22 [notice] 10#10: start worker process 15
2024/01/03 10:41:22 [notice] 10#10: start worker process 16
2024/01/03 10:41:22 [notice] 10#10: start worker process 17
2024/01/03 10:41:22 [notice] 10#10: start worker process 18
2024/01/03 10:41:22 [notice] 10#10: start worker process 19
2024/01/03 10:41:22 [notice] 10#10: start worker process 20
2024/01/03 10:41:22 [notice] 10#10: start worker process 21
2024/01/03 10:41:22 [notice] 10#10: start worker process 22
2024/01/03 10:41:22 [notice] 10#10: start worker process 23
2024/01/03 10:41:22 [notice] 10#10: start worker process 24
2024/01/03 10:41:22 [notice] 10#10: start worker process 25
2024/01/03 10:41:22 [notice] 10#10: start worker process 26
[uWSGI] getting INI configuration from /myapp/uwsgi.ini
2024-01-03 10:41:22,643 INFO exited: nginx (exit status 0; expected)
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
cheaper = 2
ini = /myapp/uwsgi.ini
module = submission
callable = app
processes = 16
ini = /etc/uwsgi/uwsgi.ini
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
;end of configuration

*** Starting uWSGI 2.0.21 (64bit) on [Wed Jan 3 10:41:22 2024] ***
compiled with version: 10.2.1 20210110 on 25 November 2022 15:10:22
os: Linux-4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023
nodename: fd786c38bcd3
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 16
current working directory: /myapp
detected binary path: /usr/local/bin/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.6.15 (default, Dec 21 2021, 12:03:22) [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55674a1fc840
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 1239640 bytes (1210 KB) for 16 cores
*** Operational MODE: preforking ***
2024-01-03 10:41:23,749 INFO success: quit_on_failure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-01-03 10:41:23,749 WARN received SIGTERM indicating exit request
2024-01-03 10:41:23,749 INFO waiting for quit_on_failure, uwsgi to die
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55674a1fc840 pid: 9 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 9)
spawned uWSGI worker 1 (pid: 28, cores: 1)
spawned uWSGI worker 2 (pid: 29, cores: 1)
running "unix_signal:15 gracefully_kill_them_all" (master-start)...
2024-01-03 10:41:27,021 INFO waiting for quit_on_failure, uwsgi to die
2024-01-03 10:41:30,024 INFO waiting for quit_on_failure, uwsgi to die
2024-01-03 10:41:33,028 INFO waiting for quit_on_failure, uwsgi to die
2024-01-03 10:41:34,029 WARN killing 'uwsgi' (9) with SIGKILL
Wed Jan 3 10:41:34 2024 - uWSGI worker 2 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Wed Jan 3 10:41:34 2024 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!
2024-01-03 10:41:34,031 INFO stopped: uwsgi (terminated by SIGKILL)
2024-01-03 10:41:34,032 INFO stopped: quit_on_failure (terminated by SIGTERM)

Exponential notation (e.g. 8.07E-06) is not supported for predictions in the submission csv files. So if you happen to have these notations in your files you should replace them (e.g. 8.07E-06 > 0.00000807).

This happens because of the regex used to parse files - it looks for the predictions as (\d+\.\d+|\d+).

This should be fixed - standard numbering formats should be supported.

dsfkjhljhf dskljhdas fkljdhasf kljhdf

As per email discussion: we want there to be two scores not one - a "preview" score and the real score that will be used in the end. So, we need to add a "previewscore" column.

I don't know how the db code works here but maybe it's as simple as adding a line after this one: https://github.com/julj/submission/blob/master/app/models.py#L62

But a big issue is: how to handle the files that have already been submitted? What goes in that new column, for them? We could recalculate new scores, as indicated by Herve. I suggest that a simple fallback option is simply to copy the value from "score"->"previewscore" at the moment that the new column is created.

@julj, you may need to comment on what is feasible at this point.

root@iZt4n5zf11kak9szz42jmzZ:/submission# docker run --name submission --link submission_db:submission_db -p 5455:80 -v $pwd/web/submission:/myapp -d submission --privileged=true
8d05bc9ee77cb9e895b8666fc6eb8f5b45c3cb1b68036122a46980538009e7e0
root@iZt4n5zf11kak9szz42jmzZ:/submission# docker logs submission
/entrypoint.sh: line 35: exec: --: invalid option
exec: usage: exec [-cl] [-a name] [command [argument ...]] [redirection ...]
root@iZt4n5zf11kak9szz42jmzZ:~/submission# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4e840d6a85c1 postgres "docker-entrypoint.s…" 30 minutes ago Up 30 minutes 5432/tcp submission_db

Please help me take a look