kattis / problemtools Goto Github PK

View Code? Open in Web Editor NEW

101.0 101.0 70.0 3.08 MB

Tools to manage problem packages using the Kattis problem package format.

License: MIT License

HTML 0.49% CSS 0.57% TeX 8.63% Makefile 0.45% Shell 1.81% Python 81.50% C++ 6.25% Dockerfile 0.31%

problemtools's People

Contributors

Stargazers

Watchers

problemtools's Issues

Input validator check on random garbage

The quantifier is wrong when checking input validators on random garbage.

Now there is a warning if there is an input validator which accepts random garbage.

The warning should only be issued if all input validators accept random garbage.

Support per test case explanations

A number of problems has commentary related to sample test cases. I don't see support for that in the Kattis problem format. Work-around is to add a section to problem.tex, but that will be typeset above, rather than below, the sample test cases.

Possible solutions:

allow sample-1.for-contestant.tex in data/sample directory separate for each sample test case to be explained
allow single sample-explanation.tex in problem_statement/ directory
allow problem writer to determine the position in problem.tex where sample test cases will be included via to-be-defined latex command

Memory limit is not checked

The memory limit is not applied to the submissions nor is validation_memory on validators.

Until this is fixed, I suggest to spit out a warning if a problem sets the memory limit explicitely in problem.yaml.

Minimum score and maximum score should be allowed to be equal

This is for example what you want for the sample group.

Current behaviour for range: 0 0:
ERROR in test case group data/sample: Invalid score range '0 0': minimum score must be smaller than maximum score

Expected behaviour:
no errors

Improve output of default_validator

We use the default_validator program in DOMjudge (with minor changes), see https://github.com/DOMjudge/domjudge/blob/master/sql/files/defaultdata/compare/compare.cc.
A team had a test case output that contained two null bytes after the normal lines of output. This gave an unenlightening error message

Wrong answer on line 4 of output (corresponding to line 4 in answer file)
Trailing output:

I'm not sure there is an easy way to fix this (or it should be fixed), but just reporting it here. Maybe the validator could report it when the output contains non-printable characters?

don't use -i with html tidy

HTML tidy with the -i option can introduce unwanted newlines and spaces inside of <pre> blocks. This can cause problem2html to produce sample data that looks incorrect (has leading spaces, e.g.).

Here's a minimal example that gets messed up by tidy -i:

<!DOCTYPE html>
<html><head><title>test</title></head><body>
<pre>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</pre>
</body>
</html>

Running tidy -i on this input produces:

<!DOCTYPE html>
<html>
<head>
  <meta name="generator" content=
  "HTML Tidy for HTML5 for Linux version 5.2.0">
  <title>test</title>
</head>
<body>
  <pre>
  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</pre>
</body>
</html>

If you remove one of the "a" characters, then the problem disappears (the <pre> remains on one line).

The problem seems to occur when the pre input is too long.

I read a bit online -- it looks like this behavior is a known "feature" of HTML tidy. See this example which does exactly what we do not want -- adding a newline to the beginning of a <pre> section:
http://www.html-tidy.org/documentation/#featured-options-and-solutions
So... I think the "best" fix is just to remove -i from when we invoke tidy.

time limit inclusion depends on current directory

See here

This means if the problem set is in somepath/someid, the file somepath/someid/.timelimit is read iff the current working directory is 'somepath'. I wouldn't even know where to start to fix this.

empty sample file not rendered in html

When either the sample input or sample output in a problem is empty, nothing is displayed. Instead, it should probably display a <pre> similar to the other samples, but with no content.

problemtools cannot be installed via pip into virtualenv

Since not everyone runs Debian, and even if we run Debian, we may be force to use a virtualenv, it would be great if problemtools installed in a virtual environment.

Right now, this fails. Specifically, if I do:

mkdir v
virtualenv ./v
. ./v/bin/activate
pip install git+https://github.com/Kattis/problemtools.git

then I would expect a working problem tools in ./v.
However, it's missing, for instance, the HTML templates.

This leaves me to run problemtools from the git working directory, which in turn means it's harder to include problemtools's libraries in my own application. I'm insufficiently familiar with pip/setuptools to know how to fix this.

\underline does not work

\underline is "ignored" when converting to html, it has no effect.

Workaround exists: use \uline instead.

The README should have a copy-paste:able command for installing dependencies on Ubuntu

Right now you have to manually remove commas and parentheses with versions and prepend sudo apt install, unless there's some simple apt command I'm missing.

Getting problemtools to work for me

Hi, I am working on some problems for the upcoming NCNA regional and would like to have the problemtools work for me. I installed them using pip2 on my Gentoo system, but running them is an epic fail. It is pretty obvious that there must be some "driver" file that includes the problem description. I found a file named "problemset.cls" that must be the documentclass for the "driver" file. Am I on the right track?

Setting stack limit does not work in OS X

This line in program.py raises an error, even when running as sudo:
resource.setrlimit(resource.RLIMIT_STACK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))

After IDI open this weekend, I decided to use your problemtools for the entire process with future contests. I got almost everything working on OS X, but this line causes a crash. I have not yet found a way around it.

This only affects running python solutions. If I remove this line, I am able to verify solutions in other languages, but all python processes crash.

I know you currently don't support OS X, but I thought I'd let you know anyway. I will create a pull request if I am able to fix it.

verifyproblem.py crashes on symlink to checker run script

I've created a checker in a subdirectory. It uses checktestdata as a library, so it's not a simple file that can be compiled. Instead the build file does 'make checker' and 'run' is a symlink to that binary 'checker'. Note that 'checker' need not exist yet before the build script is executed; I wanted to see whether that was accepted. Instead verifyproblem.py crashed, see the output below.

$ ../../../problemtools/src/verifyproblem.py .
Loading problem
Traceback (most recent call last):
File "../../../problemtools/src/verifyproblem.py", line 1085, in
with Problem(args.problemdir) as prob:
File "../../../problemtools/src/verifyproblem.py", line 1005, in enter
self.output_validators = OutputValidators(self)
File "../../../problemtools/src/verifyproblem.py", line 769, in init
self._validators = get_programs(os.path.join(problem.probdir, 'output_validators'), problem.tmpdir, error_handler=self)
File "../../../problemtools/src/verifyproblem.py", line 38, in get_programs
ret.append(Program(path, tmpdir, includedir=includedir))
File "/home/jaap/contest-jury/problemtools/src/program.py", line 284, in init
self.add_files(path)
File "/home/jaap/contest-jury/problemtools/src/program.py", line 271, in add_files
shutil.copy(src, dest)
File "/usr/lib/python2.7/shutil.py", line 119, in copy
copyfile(src, dst)
File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/home/jaap//run'

Rendering of carets in html

https://open.kattis.com/problems/greeneggs gives \^ instead of ^.

overfull hbox when sample input is too wide

Sorry for filling your issue inbox with minor issues, but it may be worth recording:

http://acmgnyr.org/year2014/a.pdf uses 20 3-digit integers in its sample input, and they run out of the box
in the generated PDF.

Should there be a feature to hint that the sample I/O should use a smaller fontsize in the problem statement?

build/run script not in subdirectory gives non-intuitive error message

Putting a program with build/run script directly in e.g. input_format_validators instead of a subdirectory gives an error message about a run script being found without a build script.

verifyproblem crashes when there are no test cases in the data/secret

See subject. Error trace is:

Traceback (most recent call last):st case sample/sample01...
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 1226, in <module>
    main()
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 1220, in main
    [errors, warnings] = prob.check(args)
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 1174, in check
    item.check(args)
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 1091, in check
    res = self.check_submission(sub, args, acr, timelim, timelim_margin)
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 1050, in check_submission
    (result1, result2) = self._problem.testdata.run_submission(sub, args, timelim_low, timelim_high)
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 383, in run_submission
    (r1, r2) = subdata.run_submission(sub, args, timelim_low, timelim_high)
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 388, in run_submission
    return (self.aggregate_results(subres1),
  File "/home/icpc.cs.vt.edu/icpc.cs.vt.edu/problemtools-1.20170420/problemtools/verifyproblem.py", line 411, in aggregate_results
    res.testcase = sub_results[-1].testcase
IndexError: list index out of range

Verbatim text renders centered inside centered verbatim in html output

See https://open.kattis.com/problems/printingcosts

The source LaTeX says:

\begin{center}
\begin{minipage}{5.1in}
\begin{verbatim}
    0        !   9        "   6        #  24        $  29        %  22
&  24        '   3        (  12        )  12        *  17        +  13
[snip]
t  17        u  17        v  13        w  19        x  13        y  24
z  19        {  18        |  12        }  18        ~   9        
\end{verbatim}
\end{minipage}
\end{center}

In HTML, look at last line to see the issue (the verbatim line starting with z). It is not left-aligned with the text above in the HTML formatted version.

In the PDF, the text inside \verbatim is rendered left-aligned, as expected. In HTML, the same text is center-justified, which is incorrect.

Input validation -- strict syntax checking

The input validation checker already warns if the input validator accepts things like the empty file, binary file, etc (https://github.com/Kattis/problemtools/blob/develop/problemtools/verifyproblem.py#L693).

When I am reviewing a problem for someone else I check for good input validators. I find that people often use inadequate syntax checking (e.g. their input validators check ranges on numbers, but not that each line is properly formatted with regular expressions). It's tedious for me to check for this and then explain why strict syntax checking is a good thing™.

Here's an idea on how to automate this checking. Suppose we have verifyproblem play with the spacing of some input files that are in the problem already and pass validation. That is, add extra spaces and/or newlines where they already exist. If there is already an input format validator that uses strict syntax checking with regular expressions, then for many (most?) types of problems those input files should fail validation. If they don't then verifyproblem could give a warning.

If this is considered worthwhile, I can work on the changes.

default validator's floating-point tolerance behavior

I've had some questions about Kattis's floating-point validator, and I figure this is the best place to start the discussion:

if(!(fabs(jval - tval) <= float_abs_tol) && 
			   !(fabs(jval - tval) <= float_rel_tol*fabs(jval))) {
				wrong_answer("Too large difference.\n Judge: %s\n Team: %s\n Difference: %le\n (abs tol %le rel tol %le)", 
							 judge.c_str(), team.c_str(), jval-tval, float_abs_tol, float_rel_tol);
}

For the sake of a concrete example, let's say that a problem asks the user to output pi, to absolute error 1e-3. The above code assumes that the judge output, jval, is exact: that it is specified to infinite precision. If this is not the case, for instance if the judge answer is
3.141
(which, note, is correct to absolute error 1e-3) then a contestant's answer of
3.1425
would be judged as incorrect, despite also being within the absolute error to the exact solution.

Now, one might say, "just add more digits to the judge solution." This alleviates but does not fully resolve the problem: if the judge solution is
3.141592653589
then a contestant's answer of
3.1425926535891
will be judged incorrect, despite technically meeting the judging criterion. Moreover, although we can provably generate arbitrarily many correct digits of pi, for many floating-point problems, the exact solution is unknown and unknowable, and even a carefully crafted judge solution will only print a finite number of accurate digits, before printing just numerical noise.

So I think Kattis needs to take into account both error in the contestant solution, and in the judge data. One possibility is to require judges to guarantee that their solutions are at least as precise as the tolerances in the problem statement; another is to allow them to assert their own, possible tighter, tolerances for their solutions. Let's look at the latter situation, since the former is just a special case:

==Absolute Error==

This one's easy: if the problem statement tolerance is float_abs_tol and the judge tolerance is float_abs_jtol, then Kattis should check
fabs(jval - tval) <= (float_abs_tol + float_abs_jtol)

==Relative Error==

==TL;DR==

the current output validator assumes that the judge output has infinite precision, and only correctly enforces the problem statement tolerances when this is true.
the judging formulas should be modified, as above, if the judge data itself might have some error.
it might be reasonable to assume, by default, that the judge output has at most as much error as the limits in the problem statement: in that case *_jtol = *_tol in the above formulas.

print of missing tidy breaks verifyproblem

The following commit seems to be breaking change:
https://github.com/Kattis/problemtools/pull/95/files

Now verifyproblem will not run with the following syntax error

Traceback (most recent call last):
  File "/home/travis/.local/bin/verifyproblem", line 11, in <module>
    load_entry_point('problemtools===v1.20170420-52-gedad3dc', 'console_scripts', 'verifyproblem')()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 572, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2752, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2405, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2411, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/travis/.local/lib/python2.7/site-packages/problemtools/verifyproblem.py", line 20, in <module>
    import problem2html
  File "/home/travis/.local/lib/python2.7/site-packages/problemtools/problem2html.py", line 80
    print "Warning: Command 'tidy' not found. Install tidy or run with --messy"
                                                                              ^
SyntaxError: invalid syntax

Putting the printed text to Python3 syntax in the parenthesis will make it work.

Check output_validators with random garbage as input

Most cases of bad output validators that we've had so far would probably have been caught if verifyproblem simply ran the output validator with random junk as submission output.
If this causes validator to violate protocol (e.g. by crashing) then that's an error of course.
If the validator accepts the random junk, a warning should be issued (similarly to what is done if input validator accepts random junk).

Two single-quotes in math mode not rendered properly in HTML export

This problem has LaTeX code which includes $s''$ .

Rendered in PDF, such text looks correct. In particular, for $s''$ the two single quotes appear as two single quotes. In HTML, however, it gets converted to $s"$ (s followed by a double-quote), which looks wrong (and is wrong).

It appears that the exported HTML contains <span class="tex2jax_process">$s”$</span>, which is wrong -- so this is an HTML export issue rather than a rendering issue.

Support 'wrong' as test data category

It would be nice if we could add test data to verify that the input checker fails on them but they should not be used for real judging.

EXPECTED_SCORE is ignored

The problme format specifies that a solution can include an EXPECTED_SCORE for scoring problems. This is good, but it is currently ignored. The spec doesn't say what should happen when the actual score != expected score. It should probably give an error.

Clean install on Ubuntu 18.04 failing with make builddeb

I'm having trouble installing the problemtools. I am working from Ubuntu 18.04 and after installing only git and the listed build dependencies in debian/control I get the following error: /usr/lib/python2.7/dist-packages/setuptools/dist.py:406: UserWarning: The version specified ('v1.20170420-15-ga52bdb2') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.

Complete output on the command make builddeb is as follows.

finn@finn-VirtualBox:~/problemtools$ make builddeb
dpkg-buildpackage -us -uc -tc -b
dpkg-buildpackage: info: source package kattis-problemtools
dpkg-buildpackage: info: source version 1.20180426
dpkg-buildpackage: info: source distribution artful
dpkg-buildpackage: info: source changed by Per Austrin [email protected]
dpkg-buildpackage: info: host architecture amd64
dpkg-source --before-build problemtools
fakeroot debian/rules clean
make[1]: Entering directory '/home/finn/problemtools'
dh clean --with python2 --buildsystem=pybuild
dh_auto_clean -O--buildsystem=pybuild
dh_clean -O--buildsystem=pybuild
make[1]: Leaving directory '/home/finn/problemtools'
debian/rules build
make[1]: Entering directory '/home/finn/problemtools'
dh build --with python2 --buildsystem=pybuild
dh_update_autotools_config -O--buildsystem=pybuild
dh_auto_configure -O--buildsystem=pybuild
I: pybuild base:217: python2.7 setup.py config
/usr/lib/python2.7/dist-packages/setuptools/dist.py:406: UserWarning: The version specified ('v1.20170420-15-ga52bdb2') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
"details." % self.metadata.version
running config
dh_auto_build -O--buildsystem=pybuild
I: pybuild base:217: /usr/bin/python setup.py build
/usr/lib/python2.7/dist-packages/setuptools/dist.py:406: UserWarning: The version specified ('v1.20170420-15-ga52bdb2') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
"details." % self.metadata.version
running build
running build_support
Running command: make -C support install DESTDIR=/home/finn/problemtools/.pybuild/cpython2_2.7/build/problemtools/support
make[2]: Entering directory '/home/finn/problemtools/support'
git submodule update --init
Submodule 'checktestdata' (https://github.com/DOMjudge/checktestdata) registered for path 'checktestdata'
Cloning into '/home/finn/problemtools/support/checktestdata'...
fatal: unable to access 'https://github.com/DOMjudge/checktestdata/': Failed to connect to 127.0.0.1 port 9: Connection refused
fatal: clone of 'https://github.com/DOMjudge/checktestdata' into submodule path '/home/finn/problemtools/support/checktestdata' failed
Failed to clone 'support/checktestdata'. Retry scheduled
Cloning into '/home/finn/problemtools/support/checktestdata'...
fatal: unable to access 'https://github.com/DOMjudge/checktestdata/': Failed to connect to 127.0.0.1 port 9: Connection refused
fatal: clone of 'https://github.com/DOMjudge/checktestdata' into submodule path '/home/finn/problemtools/support/checktestdata' failed
Failed to clone 'support/checktestdata' a second time, aborting
Makefile:19: recipe for target 'checktestdata/bootstrap' failed
make[2]: *** [checktestdata/bootstrap] Error 1
make[2]: Leaving directory '/home/finn/problemtools/support'
Traceback (most recent call last):
File "setup.py", line 94, in
'build': build
File "/usr/lib/python2.7/dist-packages/setuptools/init.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "setup.py", line 44, in run
self.run_command('build_support')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "setup.py", line 29, in run
subprocess.check_call(command)
File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-C', 'support', 'install', 'DESTDIR=/home/finn/problemtools/.pybuild/cpython2_2.7/build/problemtools/support']' returned non-zero exit status 2
E: pybuild pybuild:336: build: plugin distutils failed with: exit code=1: /usr/bin/python setup.py build
dh_auto_build: pybuild --build -i python{version} -p 2.7 returned exit code 13
debian/rules:14: recipe for target 'build' failed
make[1]: *** [build] Error 25
make[1]: Leaving directory '/home/finn/problemtools'
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
Makefile:5: recipe for target 'builddeb' failed
make: *** [builddeb] Error 2

Error trying to run verifyproblem on current dir

Hi all,

I get the error below when using CWD as problem dir. Running verifyproblem from any other directory (also not the problemtools directory) works fine.

P.S. Minor detail: could you make the scripts executable and add a she-bang line

!/usr/bin/env python

to allow executing them directly without interpreter?

$ python ../../problemtools/verifyproblem.py .
Loading problem .
Traceback (most recent call last):
File "../../problemtools/verifyproblem.py", line 1006, in
with Problem(dir) as prob:
File "../../problemtools/verifyproblem.py", line 946, in enter
shutil.copytree(self.srcdir, self.probdir)
File "/usr/lib/python2.7/shutil.py", line 177, in copytree
os.makedirs(dst)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: './testprobNgCLAM/.'

Scoring validator should not have to print score on non-AC

When using a scoring output validator, problemtools expects it to produce score.txt even when it returns non-AC. This should not be the case.

run single submission

I want to run a single submission instead of waiting ages for all the submissions that come alphabetically first.

Additional dependencies needed

g++4.9 is needed for comptability with regex.
libboost1.55-dev is needed
libgmp-dev is also needed.

Detection of GMPlib fails on mac os X

Encountered when installing. Here is the log:

Snowcrash:~ pehrs$ pip install git+https://github.com/kattis/problemtools
Collecting git+https://github.com/kattis/problemtools
  Cloning https://github.com/kattis/problemtools to /var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build
Requirement already satisfied (use --upgrade to upgrade): PyYAML in /usr/local/lib/python2.7/site-packages (from problemtools===v1.20160728-7-g0b9f432)
Requirement already satisfied (use --upgrade to upgrade): plasTeX in /usr/local/lib/python2.7/site-packages (from problemtools===v1.20160728-7-g0b9f432)
Installing collected packages: problemtools
  Running setup.py install for problemtools ... error
    Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-05JcaC-record/install-record.txt --single-version-externally-managed --compile:
    readlink: illegal option -- f
    usage: readlink [-n] [file ...]
    /usr/local/lib/python2.7/site-packages/setuptools/dist.py:294: UserWarning: The version specified ('v1.20160728-7-g0b9f432') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
      "details." % self.metadata.version
    running install
    running build
    running build_support
    Running command: make -C support install DESTDIR=/private/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/build/lib/problemtools/support
    cd checktestdata && ./bootstrap
    + set -e
    + aclocal -I m4
    + autoconf
    + ./configure --disable-parsergeneration
    checking whether configure should try to set CFLAGS... yes
    checking whether configure should try to set CXXFLAGS... yes
    checking whether configure should try to set LDFLAGS... yes
    checking for gcc... gcc
    checking whether the C compiler works... yes
    checking for C compiler default output file name... a.out
    checking for suffix of executables...
    checking whether we are cross compiling... no
    checking for suffix of object files... o
    checking whether we are using the GNU C compiler... yes
    checking whether gcc accepts -g... yes
    checking for gcc option to accept ISO C89... none needed
    checking whether C compiler accepts -Wall... yes
    checking whether C compiler accepts -fstack-protector... yes
    checking whether C compiler accepts -fPIE... yes
    checking whether C compiler accepts -D_FORTIFY_SOURCE=2... yes
    checking whether the linker accepts -fPIE... yes
    checking whether the linker accepts -pie... yes
    checking whether the linker accepts -Wl,-z,relro... no
    checking whether the linker accepts -Wl,-z,now... no
    checking for g++... g++
    checking whether we are using the GNU C++ compiler... yes
    checking whether g++ accepts -g... yes
    checking for gcc... (cached) gcc
    checking whether we are using the GNU C compiler... (cached) yes
    checking whether gcc accepts -g... (cached) yes
    checking for gcc option to accept ISO C89... (cached) none needed
    checking how to run the C++ preprocessor... g++ -E
    checking how to run the C preprocessor... gcc -E
    checking build system type... x86_64-apple-darwin15.6.0
    checking host system type... x86_64-apple-darwin15.6.0
    checking for boostlib >= 1.33.1... yes
    checking whether the Boost::Regex library is available... yes
    checking for exit in -lboost_regex-mt... yes
    checking for grep that handles long lines and -e... /usr/bin/grep
    checking for egrep... /usr/bin/grep -E
    checking for ANSI C header files... yes
    checking for sys/types.h... yes
    checking for sys/stat.h... yes
    checking for stdlib.h... yes
    checking for string.h... yes
    checking for memory.h... yes
    checking for strings.h... yes
    checking for inttypes.h... yes
    checking for stdint.h... yes
    checking for unistd.h... yes
    checking gmpxx.h usability... no
    checking gmpxx.h presence... no
    checking for gmpxx.h... no
    checking for abs in -lgmpxx... no
    configure: error: GMPlib not found
    make: *** [checktestdata/config.mk] Error 1
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/setup.py", line 94, in <module>
        'build': build
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py", line 563, in run
        self.run_command('build')
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/setup.py", line 44, in run
        self.run_command('build_support')
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/setup.py", line 29, in run
        subprocess.check_call(command)
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['make', '-C', 'support', 'install', 'DESTDIR=/private/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/build/lib/problemtools/support']' returned non-zero exit status 2

    ----------------------------------------
Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-05JcaC-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-E13lxc-build/
Snowcrash:~ pehrs$ brew search gmp
gmp ✔
homebrew/php/php53-gmp                 homebrew/php/php54-gmp                 homebrew/php/php55-gmp                 homebrew/php/php56-gmp                 homebrew/php/php70-gmp                 homebrew/php/php71-gmp                 homebrew/versions/gmp4               
Snowcrash:~ pehrs$ locate gmpxx.h
/usr/local/Cellar/gmp/5.1.3/include/gmpxx.h
/usr/local/Cellar/gmp/6.0.0a/include/gmpxx.h
/usr/local/include/gmpxx.h

Crash in verifyproblem in v1.20170420

I was running v1.20170420 of problemtools and it crashes with the following (on a fairly generic ACM-style problem). The problem is in development and it has no secret test data (yet). Perhaps that is triggering the bug? I can get more information if that's helpful.

Loading problem progressivescramble
Checking config
Checking statement
/drone/src/github.com/na-icpc/naq17/problemset.cls exists, will not copy it -- in case of weirdness this is likely culprit
/drone/src/github.com/na-icpc/naq17/problemset.cls exists, will not copy it -- in case of weirdness this is likely culprit
Checking validators
Checking graders
Checking data
Checking submissions
Traceback (most recent call last):
 File "/usr/local/bin/verifyproblem", line 9, in <module>
   load_entry_point('problemtools===v1.20170420-1-g30ed959', 'console_scripts', 'verifyproblem')()
 File "/usr/local/lib/python2.7/dist-packages/problemtools/verifyproblem.py", line 1220, in main
   [errors, warnings] = prob.check(args)
 File "/usr/local/lib/python2.7/dist-packages/problemtools/verifyproblem.py", line 1174, in check
   item.check(args)
 File "/usr/local/lib/python2.7/dist-packages/problemtools/verifyproblem.py", line 1091, in check
   res = self.check_submission(sub, args, acr, timelim, timelim_margin)
 File "/usr/local/lib/python2.7/dist-packages/problemtools/verifyproblem.py", line 1050, in check_submission
   (result1, result2) = self._problem.testdata.run_submission(sub, args, timelim_low, timelim_high)
 File "/usr/local/lib/python2.7/dist-packages/problemtools/verifyproblem.py", line 383, in run_submission
   (r1, r2) = subdata.run_submission(sub, args, timelim_low, timelim_high)
 File "/usr/local/lib/python2.7/dist-packages/problemtools/verifyproblem.py", line 388, in run_submission
   return (self.aggregate_results(subres1),
 File "/usr/local/lib/python2.7/dist-packages/problemtools/verifyproblem.py", line 411, in aggregate_results
   res.testcase = sub_results[-1].testcase
IndexError: list index out of range

I expect this was introduced in https://github.com/Kattis/problemtools/pull/63/files

make builddeb fails in a clean repository clone

Make a clean repository clone
Run make builddeb
It fails with

make[2]: Entering directory `/home/lukas/code/problemtools'
cd support/checktestdata && ./bootstrap
/bin/sh: 1: ./bootstrap: not found

git submodule update --init --recursive fixes the problem.

I think that either the make builddeb command should fetch the submodule or an extra step should be explicitely stated in the instructions in README.md.

problempart parsing is too greedy

../../problemtools/src/verifyproblem.py -p data .

fails with
verifyproblem.py: error: argument -p/--parts: Invalid problem part specified: .

because . is consumed by the argument parser instead of leaving it as positional argument for the directory name.

Pip install fails on Mac OS X

For some reason, the pip install fails on Mac OS X, with a complaint about the archive format.

Snowcrash:stockbroker pehrs$ pip install https://github.com/kattis/problemtools
Collecting https://github.com/kattis/problemtools
  Downloading https://github.com/kattis/problemtools
     - 92kB 388kB/s
  Cannot unpack file /private/var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-EnphdB-unpack/problemtools (downloaded from /var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-gNxtZB-build, content-type: text/html; charset=utf-8); cannot detect archive format
Cannot determine archive format of /var/folders/tg/vk556qn144jcxzgjdgrk79yw0000gn/T/pip-gNxtZB-build

languages.yaml should accept compilers in path and not hardwire absolute paths

Is there a reason for why languages.yaml hardwires the absolute path of all compilers?

It causes issues when I need to use a particular version of gcc/g++ on systems where /usr/bin/g++ is not the version problemtools wants, such as CentOS. On those systems, you start a shell such that /opt/rh/devtoolset-7/root/usr/bin/g++ is in the path before /usr/bin/g++ using RedHat's scl-utils.

problemtools completely breaks that with its assumption that the absolute path to an executable be provided in languages.yaml

Do you want it this way, or would you entertain a PR to accept commands from the path in addition?

Don't warn about duplicate test cases in different test groups

Add support for "libraries" setting in problemtools/addproblem

Add support in problemtools to use the libraries setting when installing a problem.

(Depends on Kattis Trac tickets 1034 and 1036)

release run git submodule when doing make

If I download the release from here and try to build the support by running make in the support directory, it will run git submodule ... which fails.

img base dir cannot be set in problem2html.py

You have an option to set the imgbasedir, but you don't expose it.
See this gist for what I mean.

Incorrect TLE submissions should be allowed to give WA

Currently, if we have a TLE submission, verifyproblem expects that it should always TLE or AC (but TLE on at least one input). It would be natural and nice if verifyproblem allowed a TLE submission to give WA on some inputs, as long as it gives TLE on at least one input.

The intent here is that if we wish to accept an $O(n^2)$ solution but $O(n^3)$ should TLE, then we could make a simple program which just reads the input, ignores the problem, and loops n^3 times. However, this will give the WA error described above.

@austrin points out (via email) that as we move to setting time limits by running submissions through Kattis judge rather than within the problemtools itself, this issue becomes trickier to deal with.

Better support of .timelimit file

If a problem has a .timelimit file, then verifyproblem should use the time limit stated there if it is within reasonable margin of the calculated (non-rounded) time limit (e.g. if it is within 10% or something like that?).

If it deviates by more, a warning should be issued, I guess.

.timelimit should maybe also be written by verifyproblem?

Check that answer files end with newline

Give a warning if they don't.

A seg-faulting solution hangs verifyproblem.py

Download http://www.csc.kth.se/~polacek/stol.zip, unzip it and run problemtools on the package. The TLE solution will hang on test case 11:

Running stol.cpp on test case secret/stol.11...

Changing MAXN from 400 to 1000 makes the solution time out. The one with 400 seg-faults:

[1]    822 segmentation fault  ./stol.cpp.bin < ../../data/secret/stol.11.in

It seems that a seg-faulting solution in the TLE directory makes verifyproblem.py hang.

Crash involving problem_author when reinstalling a problem from scratch

I added a problem in my VM using:
.../addproblem.sh -s 1 -a '2016-09-24 16:00' problems/anewalphabet

Then I reinstalled it from scratch (ran the command above again and selected option "2", reinstall from scratch. The result was this:

hamerly@ubuntu:~/icpc/qualifier_2016$ /usr/local/kattis/bin/admin/addproblem.sh -s 1 -a '2016-09-24 16:00' problems/anewalphabet
Problem "anewalphabet" already in database.
There is already a directory in /usr/local/kattis/data/src/ for the backup of the problem package
It seems like the problem was already (partially) installed.
What to do?
 1) Update problem (keep submissions).
   This will remove and reinstall all files (statement, data, validator) and update db entry.
 2) Reinstall from scratch.
   In addition to the above this will remove the db entry and therefore all old submissions.
 3) Abort? (Default)
 [1-3]>2
Are you REALLY sure you want to remove the problem completely before reinstalling?  This will remove all existing submissions to the problem! [y/N]y
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kattis/admin/addproblem.py", line 826, in <module>
    main(args)
  File "/usr/lib/python2.7/dist-packages/kattis/admin/addproblem.py", line 734, in main
    is_update = determine_update_mode(conn, problemname, args)
  File "/usr/lib/python2.7/dist-packages/kattis/admin/addproblem.py", line 269, in determine_update_mode
    curs.execute('DELETE FROM problem WHERE problem_id=%(problem_id)d;', locals())
  File "/usr/lib/python2.7/dist-packages/pgdb.py", line 259, in execute
    self.executemany(operation, (params,))
  File "/usr/lib/python2.7/dist-packages/pgdb.py", line 289, in executemany
    raise DatabaseError("error '%s' in '%s'" % (msg, sql))
DatabaseError: error 'ERROR:  update or delete on table "problem" violates foreign key constraint "problem_problem_author_problem_id_fkey" on table "problem_problem_author"
DETAIL:  Key (problem_id)=(6) is still referenced from table "problem_problem_author".
' in 'DELETE FROM problem WHERE problem_id=6;'

Uncaught exception in addproblem (see above for details).  THIS IS A BUG.

cc -by-nc-sa license unsupported

Greater New York 2014 is published under CC by-nc-sa but verifyproblem.py doesn't like it when I put in cc by-nc-sa as it's not a supported value.

So.... what does that mean? Do you need one of the listed licenses for open kattis (you can't do with a NC clause because Kattis is a company?), or should I list 'educational' - which is sort of correct, or something else altogether?

error: no functioning std::regex found

I am getting this error when i run make:

checking for proper std::regex implementation... configure: error: no functioning std::regex found

My OS is Ubuntu 14.04. I have run apt-get upgrade before running it. Google search told me I need to compile with g++-4.9, but that doesn't work too. Here is the full output I am getting:

$ CC="g++-4.9" make

make -C support
make[1]: Entering directory `/home/shafaet/Workspace/problemtools/support'
cd checktestdata && ./bootstrap
+ aclocal -I m4
+ autoconf
+ ./configure --disable-parsergeneration
checking whether configure should try to set CXXFLAGS... yes
checking whether configure should try to set LDFLAGS... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking whether C++ compiler accepts -Wall... yes
checking whether C++ compiler accepts -fstack-protector... yes
checking whether C++ compiler accepts -fPIE... yes
checking whether C++ compiler accepts -D_FORTIFY_SOURCE=2... yes
checking whether the linker accepts -fPIE... yes
checking whether the linker accepts -pie... yes
checking whether the linker accepts -Wl,-z,relro... yes
checking whether the linker accepts -Wl,-z,now... yes
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking for gcc... g++-4.9
checking whether we are using the GNU C compiler... yes
checking whether g++-4.9 accepts -g... yes
checking for g++-4.9 option to accept ISO C89... unsupported
checking how to run the C++ preprocessor... g++ -E
checking how to run the C preprocessor... g++-4.9 -E
checking whether g++ supports C++11 features by default... no
checking whether g++ supports C++11 features with -std=c++11... yes
checking for proper std::regex implementation... configure: error: no functioning std::regex found
make[1]: *** [checktestdata/config.mk] Error 134
make[1]: Leaving directory `/home/shafaet/Workspace/problemtools/support'
make: *** [all] Error 2

I tried running the snippet make file is trying to run directly from terminal:

#include <regex>
int main ()
{
std::regex testregex("[0-9]"); 

std::regex_search("test123",testregex);
  return 0;
}

This seems working fine with gcc-4.9. Any idea where the issue can be?

problem2pdf includes all samples twice

In a problem with two sample files 1.in and 2.in, the PDF generated by problem2pdf contains 1.in, 2.in, 1.in and finally 2.in. All sample inputs are included twice.

Running verifyproblem.py from data/sample causes an error

When running verifyproblem.py ../.. from data/sample inside a problem package, I get

ERROR in problem statement: Could not convert problem statement to html for language "en".  Run problem2html -l en on the problem to diagnose.

kattis / problemtools Goto Github PK

problemtools's People

Contributors

Stargazers

Watchers

Forkers

problemtools's Issues

!/usr/bin/env python

Recommend Projects

Recommend Topics

Recommend Org