Git Product home page Git Product logo

api-client's Introduction

online-judge-tools/api-client

test Documentation Status PyPI PyPI

What is this?

This is an API client for various online judges, used as the backend library of oj command. You can use the Python library (onlinejudge module) and the command-line interface (oj-api command) which talks JSON compatible with jmerle/competitive-companion.

How to install

$ pip3 install online-judge-api-client

Supported websites

website get sample cases get system cases get metadata get contest data login service submit code
Aizu Online Judge ✔️ ✔️
Anarchy Golf ✔️ ❔ (same to samples)
AtCoder ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
AtCoder Problems (virtual contests) ✔️
CodeChef ✔️ ✔️ ✔️
Codeforces ✔️ ✔️ ✔️ ✔️ issue
CS Academy ✔️
Facebook Hacker Cup ✔️
Google Code Jam ✔️
Google Kick Start ✔️
HackerRank ✔️ ✔️ ✔️
Kagamiz Contest System ✔️
Kattis ✔️
Library Checker ✔️ ✔️
PKU JudgeOnline ✔️
Sphere Online Judge ✔️
Topcoder ✔️ (archived problems only)
Toph ✔️ ✔️
yukicoder ✔️ ✔️ ✔️ ✔️

Instead of login-service, you can use oj command with Selenium as oj login https://.... This can login almost all services.

Supported subcommands of oj-api command

get-problem

oj-api get-problem PROBLEM_URL parses the given problem and prints the results as JSON compatible with jmerle/competitive-companion.

options

  • --system: get system cases, instead of sample cases
  • --full: dump all additional data

format

  • url: the URL of the problem
  • name: the name of the problem. This doesn't include alphabets (e.g. just "Xor Sum" is used instead of "D. Xor Sum") because such alphabets are attributes belonging to the relation between problems and contests rather than belonging to only problems. (not compatible to jmerle/competitive-companion)
  • context:
    • contest (optional): - url: the URL of the contest - name: the name of the contest
    • alphabet (optional): the alphabet of the problem in the contest
  • memoryLimit: the memory limit in megabytes (MB); not mebibytes (MiB). They sometimes become non-integers, but be rounded down for the compatibility reason with jmerle/competitive-companion.
  • timeLimit: the time limit in milliseconds (msec)
  • tests:
    • input: the input of the test case
    • output: the output of the test case

format (additional)

  • tests:
    • name (optional, when --system): the name of the system case (e.g. random-004.in, fft_killer_01, 99_hand.txt)
  • availableLanguages (optional, when --full):
    • id: the ID of language to submit the server (e.g. 3003)
    • description: the description of the language to show to users (e.g. C++14 (GCC 5.4.1))
  • raw (optional, when --full):
    • html (optional): the raw HTML used internally. This might contain sensitive info like CSRF tokens.
    • json (optional): the raw JSON used internally. This might contain sensitive info like access tokens.
    • etc.

example

$ oj-api get-problem https://atcoder.jp/contests/arc100/tasks/arc100_b | jq .result
{
  "url": "https://atcoder.jp/contests/arc100/tasks/arc100_b",
  "name": "Equal Cut",
  "context": {
    "contest": {
      "url": "https://atcoder.jp/contests/arc100",
      "name": "AtCoder Regular Contest 100"
    },
    "alphabet": "D"
  },
  "memoryLimit": 1024,
  "timeLimit": 2000,
  "tests": [
    {
      "input": "5\n3 2 4 1 2\n",
      "output": "2\n"
    },
    {
      "input": "10\n10 71 84 33 6 47 23 25 52 64\n",
      "output": "36\n"
    },
    {
      "input": "7\n1 2 3 1000000000 4 5 6\n",
      "output": "999999994\n"
    }
  ]
}

get-problem --compatibility

oj-api get-problem --compatibility PROBLEM_URL is the variant of get-problem strictly compatible with jmerle/competitive-companion.

format

See the document of jmerle/competitive-companion.

example

{
  "name": "D. Equal Cut",
  "group": "AtCoder Regular Contest 100",
  "url": "https://atcoder.jp/contests/arc100/tasks/arc100_b",
  "interactive": false,
  "memoryLimit": 1024,
  "timeLimit": 2000,
  "tests": [
    {
      "input": "5\n3 2 4 1 2\n",
      "output": "2\n"
    },
    {
      "input": "10\n10 71 84 33 6 47 23 25 52 64\n",
      "output": "36\n"
    },
    {
      "input": "7\n1 2 3 1000000000 4 5 6\n",
      "output": "999999994\n"
    }
  ],
  "testType": "single",
  "input": {
    "type": "stdin"
  },
  "output": {
    "type": "stdout"
  },
  "languages": {
    "java": {
      "mainClass": "Main",
      "taskClass": "Task"
    }
  }
}

get-contest

oj-api get-contest CONTEST_URL parses the given contest and prints the results as JSON.

format

  • url: the URL of the contest
  • name: the name of the contest
  • problems: problems. For details, see the description of get-problem.

example

$ oj-api get-contest https://atcoder.jp/contests/arc100 | jq .result
{
  "url": "https://atcoder.jp/contests/arc100",
  "name": "AtCoder Regular Contest 100",
  "problems": [
    {
      "url": "https://atcoder.jp/contests/arc100/tasks/arc100_a",
      "name": "Linear Approximation",
      "context": {
        "contest": {
          "url": "https://atcoder.jp/contests/arc100",
          "name": "AtCoder Regular Contest 100"
        },
        "alphabet": "C"
      }
    },
    {
      "url": "https://atcoder.jp/contests/arc100/tasks/arc100_b",
      "name": "Equal Cut",
      "context": {
        "contest": {
          "url": "https://atcoder.jp/contests/arc100",
          "name": "AtCoder Regular Contest 100"
        },
        "alphabet": "D"
      }
    },
    {
      "url": "https://atcoder.jp/contests/arc100/tasks/arc100_c",
      "name": "Or Plus Max",
      "context": {
        "contest": {
          "url": "https://atcoder.jp/contests/arc100",
          "name": "AtCoder Regular Contest 100"
        },
        "alphabet": "E"
      }
    },
    {
      "url": "https://atcoder.jp/contests/arc100/tasks/arc100_d",
      "name": "Colorful Sequences",
      "context": {
        "contest": {
          "url": "https://atcoder.jp/contests/arc100",
          "name": "AtCoder Regular Contest 100"
        },
        "alphabet": "F"
      }
    }
  ]
}

get-service

oj-api get-service SERVICE_URL prints the data of the service.

options

  • --list-contests: list all contests in the service

format

  • url: the URL of the service
  • name: the name of the service
  • contests (when --list-contests): contests. For details, see the description of get-problem.

example

$ oj-api get-service https://atcoder.jp/ --list-contests | jq .result
{
  "url": "https://atcoder.jp/",
  "name": "AtCoder",
  "contests": [
    {
      "url": "https://atcoder.jp/contests/abc162",
      "name": "AtCoder Beginner Contest 162"
    },
    {
      "url": "https://atcoder.jp/contests/judge-update-202004",
      "name": "Judge System Update Test Contest 202004"
    },
    {
      "url": "https://atcoder.jp/contests/abc161",
      "name": "AtCoder Beginner Contest 161"
    },
    {
      "url": "https://atcoder.jp/contests/abc160",
      "name": "AtCoder Beginner Contest 160"
    },
    ...
  ]
}

login-service

USERNAME=USERNAME PASSWORD=PASSWORD oj-api login-service SERVICE_URL logs in the given service.

options

  • --check: only check whether you are already logged in, without trying to log in

format

  • loggedIn: the result

example

$ USERNAME=kimiyuki PASSWORD='????????????????' oj-api login-service https://atcoder.jp/ | jq .result
{
  "loggedIn": true
}

submit-code

oj-api submit-code PROBLEM_URL --file FILE --language LANGUAGE_ID submits the file to the given problem. You can obtrain the LANGUAGE_ID from the list availableLanguages of oj-api get-problem --full PROBLEM_URL or the guess-language-id subcommand.

format

  • url: the URL of the submission result

example

$ oj-api submit-code https://atcoder.jp/contests/abc160/tasks/abc160_a --file main.py --language 3023 | jq .result
{
  "url": "https://atcoder.jp/contests/abc160/submissions/11991846"
}

guess-language-id

oj-api guess-language-id PROBLEM_URL --file FILE guesses the language id to submit the file to the given problem.

format

  • id: the language id
  • description: the description of the language id

example

$ oj-api guess-language-id http://codeforces.com/contest/1373/problem/A --file=main.py | jq .result
{
  "id": "31",
  "description": "Python 3.7.2",
}

JSON API responses

format

  • status: the status. This contains ok if the subcommand succeeded.
  • messages: error messages
  • result: the result

example

$ USERNAME=chokudai PASSWORD=hoge oj-api login-service https://atcoder.jp/ | jq .
{
  "status": "error",
  "messages": [
    "onlinejudge.type.LoginError: failed to login"
  ],
  "result": null
}

Tips

For end-users who don't develop any programs:

  • Please use the oj command instead of this oj-api command.
    • The oj-api command is intended to be used as a backend of other programs. It's not a frontend which humans use directly.

For developers of programs which use oj-api:

  • Please be polite to judge servers.
    • We are accessing the web pages using the unintended way, i.e. without GUI browsers. We should not forget that this is basically not welcomed.
  • Please be aware that this package uses Semantic Versioning.
    • The "public API" of this package is the documented features of oj-api command and the documented modules of onlinejudge module.
  • You can also use the oj login command to login servers with GUI browsers.
    • The implementation around authentication is very complicated and fragile. For example, Codeforces has four methods to login: password, Gmail, Facebook, and ICPC. So we need to use GUI browsers for the development cost and stability.
    • This feature of oj login might be imported to oj-api command in the future.

api-client's People

Contributors

aberent avatar avtomat2023 avatar beet-aizu avatar btk15049 avatar cake-monotone avatar fukamachi avatar fukatani avatar ganow avatar gitter-badger avatar hachi-88 avatar hamayanhamayan avatar hjr265 avatar kawacchu avatar kjnh10 avatar kmyk avatar knshnb avatar koba-e964 avatar kotatsugame avatar kzrnm avatar nahco314 avatar njkevlani avatar pachicobue avatar pieceofeden avatar serihiro avatar shino16 avatar subramanir2143 avatar uta8a avatar wapa5pow avatar yosupo06 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

api-client's Issues

users' .gitconfig affects downloading test cases from Library-Checker

Description

We invoke generate.py at https://github.com/yosupo06/library-checker-problems to generate test cases of problems in Library-Checker, and use git command to checkout this repository.
For example, core.autoCRLF=true causes a problem (yosupo06/library-checker-problems#598).

Detecting core.autoCRLF=true config or CRLF in sample files and show warnings on oj command or generate.py?

Error log

yosupo06/library-checker-problems#598 (comment)

Other notes

Many of SPOJ problems are not supported.

ex.

https://www.spoj.com/problems/MKLABELS/

<h3>Example</h3>
  |  
  | <pre>
  | <b>Input:</b>
  | 2
  | 3
  | 4
  | 5
  | 0
  |  
  | <b>Output:</b>
  | Case 1, N = 2, # of different labelings = 1
  | Case 2, N = 3, # of different labelings = 3
  | Case 3, N = 4, # of different labelings = 16
  | Case 4, N = 5, # of different labelings = 125
  | </pre>
  |  

Adapt to new DropBox API

Description / 説明

DropBox API changed on Sep 30, 2021. Since then, long access tokens are not granted.
We have to first obtain AUTHORIZATION_CODE (from Web UI), and after that obtain ACCESS_TOKEN from AUTHORIZATION CODE (and <APP_KEY>:<APP_SECRET>).

The generated ACCESS_TOKEN will expire in 14400 secs = 4 hrs. After that, we must regenerate ACCESS_TOKEN via refresh_token.

For compatibility reason, we should keep the old code as-is.

Motivation / 動機

The current DropBox App cannot have new users, presumably because of the limit of the number of users.
We are going to develop a new one, hopefully with no limits on the number of users.

Other notes / その他

  • Will you try to create a pull request?
    • yes

Codeforces changed the HTML around login

Summary / 概要

tests.login_service.LoginServiceCodeforcesTest testMethod=test_login_failure failed.

Steps to reproduce / 再現方法

  1. pytest tests/*.py

Expected behavior / 期待される挙動

The tests pass.

Actual behavior / 実際の挙動

https://github.com/online-judge-tools/api-client/runs/2272180619

=================================== FAILURES ===================================
________________ LoginServiceCodeforcesTest.test_login_failure _________________

self = <tests.login_service.LoginServiceCodeforcesTest testMethod=test_login_failure>

    def test_login_failure(self) -> None:
        url = 'https://codeforces.com/'
        expected = {'status': 'error', 'messages': ['onlinejudge.type.LoginError: Invalid handle or password.'], 'result': None}
    
        with update_environ(USERNAME='online-judge-tools', PASSWORD='password'):
            with temporary_cookie() as cookie_path:
                actual = main(['--cookie', str(cookie_path), 'login-service', url], debug=True)
>       self.assertEqual(expected, actual)
E       AssertionError: {'sta[24 chars]': ['onlinejudge.type.LoginError: Invalid hand[29 chars]None} != {'sta[24 chars]': ['AssertionError'], 'result': None}
E       + {'messages': ['AssertionError'], 'result': None, 'status': 'error'}
E       - {'messages': ['onlinejudge.type.LoginError: Invalid handle or password.'],
E       -  'result': None,
E       -  'status': 'error'}

tests/login_service.py:80: AssertionError

`oj-api login-service --check` crashes

To reproduce

  1. run $ oj-api login-service --check https://codeforces.com

Actual behavior

An exception is thrown at

assert username is None

Error log

$ oj-api login-service --check https://codeforces.com
INFO:root:sleep 1.000000 sec
INFO:onlinejudge._implementation.logging:[-] unknown problem: https://codeforces.com
INFO:onlinejudge._implementation.logging:[-] unknown contest: https://codeforces.com
INFO:onlinejudge._implementation.logging:[x] service recognized: CodeforcesService.from_url('https://codeforces.com/'): https://codeforces.com
INFO:onlinejudge._implementation.logging:[x] load cookie from: /home/user/.local/share/online-judge-tools/cookie.jar
ERROR:root:
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.6/site-packages/onlinejudge_api/main.py", line 257, in main
    result = login_service.main(service, username=parsed.username, password=parsed.password, check_only=parsed.check, session=session)
  File "/home/user/.local/lib/python3.6/site-packages/onlinejudge_api/login_service.py", line 28, in main
    assert username is None
AssertionError
{"status": "error", "messages": ["AssertionError"], "result": null}

Remove trailing spaces from sample cases of Codeforces

example: https://codeforces.com/contest/1334/problem/D

$ oj --version                      
online-judge-tools 9.2.0

$ oj d -n https://codeforces.com/contest/1334/problem/D
[x] problem recognized: CodeforcesProblem.from_url('https://codeforces.com/contest/1334/problem/D'): https://codeforces.com/contest/1334/problem/D
[x] load cookie from: /home/user/.local/share/online-judge-tools/cookie.jar
[x] GET: https://codeforces.com/contest/1334/problem/D
[x] 200 OK
[x] save cookie to: /home/user/.local/share/online-judge-tools/cookie.jar

[*] sample 0
[x] input: sample-1
3
2 1 3
3 3 6
99995 9998900031 9998900031

[x] output: sample-1
1 2 1_(trailing spaces)
1 3 2 3_(trailing spaces)
1_(trailing spaces)

Downloading system cases of AtCoder from the Dropbox

Description

We can download system cases of some problems of AtCoder, because AtCoder distributes some test cases via Dropbox.
This feature is desired by users. However, while technically possible, I'm not sure if it's acceptable in terms of morality and maintainability.

Possible implementations and the analysis

選択肢:

  1. Dropbox の App として正規の API を使ってダウンロードする
  2. Dropbox を scraping する

過去の議論:

しかし、しんどいので私は実装もメンテもあまりしたくないです。(ユーザからは望まれてるので、よく書けたプルリクが来てしまった場合はマージせざるをえないみたいなところはありますが……)
これは以下のふたつの課題のためです:

  1. Dropbox の操作が難しい
    • Dropbox は scraping を歓迎していません。サイトが AtCoder などと違って複雑かつ頻繁に変更が入るので、scraping は開発もメンテもしんどいはずです。ちなみにユーザにとっては楽です
      • 追記: 規約違反の可能性がある (ない可能性もある) ので確認が必要
    • API の利用は推奨されていますが、開発者にとってもユーザにとってもしんどいです。開発者側は OAuth 用のサーバを立てて維持管理をする必要がありますし、ユーザ側は Dropbox のアカウントを作って認証その他をする必要があります (たぶん)
    • 参考: https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/download-file-using-python-API-frpm-another-user/td-p/191824
  2. AtCoder はテストケースのダウンロードの自動化を歓迎していない
    • 「望まれているのは分かるけど、機械的にダウンロードできるようにしちゃうと他のオンラインジャッジに問題を丸パクリされるのでつらい」みたいな話があります。実装前に chokudai さんに話を通す必要があります

Originally posted by @kmyk in online-judge-tools/verification-helper#74 (comment)

Other notes

AtCoder's submission result pages are updated

Description

The submission result page https://atcoder.jp/contests/abc168/submissions/13347505 seems to be updated.
This breaks our code.

This doesn't affect most users, but at least breaks our CI.
Remove the feature to retrieve the detailed info from submissions?

Error log

https://github.com/online-judge-tools/api-client/runs/1178238041

=========================== short test summary info ============================
FAILED tests/get_problem_topcoder.py::GetProblemTopcoderTest::test_10760 - As...
FAILED tests/dispatch.py::DispatchAtCoderTest::test_submission_from_url - Ass...
FAILED tests/type.py::TypeTest::test_download_contest - AssertionError
FAILED tests/dispatch.py::DispatchAtCoderTest::test_submission_from_url - Ass...
FAILED tests/service_atcoder.py::AtCoderContestTest::test_iterate_submissions
FAILED tests/service_atcoder.py::AtCoderProblemTest::test_iterate_submissions
FAILED tests/service_atcoder.py::AtCoderSubmissionTest::test_get_score_float
FAILED tests/service_atcoder.py::AtCoderSubmissionTest::test_get_source_code
FAILED tests/service_atcoder.py::AtCoderSubmissionTest::test_get_test_cases
FAILED tests/service_atcoder.py::AtCoderSubmissionTest::test_get_test_sets - ...
FAILED tests/service_atcoder.py::AtCoderSubmissionTest::test_submission_info
FAILED tests/service_atcoder.py::AtCoderSubmissionTest::test_submission_info_compile_warnings
= 12 failed, 142 passed, 5 skipped, 2 xfailed, 108 warnings in 149.93s (0:02:29) =
__________________ AtCoderSubmissionTest.test_submission_info __________________

self = <tests.service_atcoder.AtCoderSubmissionTest testMethod=test_submission_info>

    def test_submission_info(self):
        submission = AtCoderSubmission.from_url('https://atcoder.jp/contests/agc030/submissions/3904911')
>       data = submission.download_data()

tests/service_atcoder.py:209: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = AtCoderSubmission.from_url('https://atcoder.jp/contests/agc030/submissions/3904911')

    def download_data(self, *, session: Optional[requests.Session] = None) -> AtCoderSubmissionDetailedData:
        """
        :note: `Exec Time` is undefined when the status is `RE` or `TLE`
        :note: `Memory` is undefined when the status is `RE` or `TLE`
        """
        session = session or utils.get_default_session()
        resp = _request('GET', self.get_url(type='beta', lang='en'), session=session)
        soup = bs4.BeautifulSoup(resp.content.decode(resp.encoding), utils.html_parser)
        timestamp = datetime.datetime.now(datetime.timezone.utc).astimezone()
    
        # Submission #N
        id_, = soup.find_all('span', class_='h2')
        assert id_.text == 'Submission #{}'.format(self.submission_id)
    
        # Source Code
        source_code = soup.find(id='submission-code')
        source_code = source_code.text.encode()
    
        # get tables
        tables = soup.find_all('table')
        if len(tables) == 3:
            submission_info, test_cases_summary, test_cases_data = tables
        elif len(tables) == 1:
            submission_info, = tables
            test_cases_summary = None
            test_cases_data = None
        else:
>           assert False
E           AssertionError

onlinejudge/service/atcoder.py:1158: AssertionError

Receiving `403 Forbidden` when submitting to Codeforces

Description

When I try to submit to Codeforces, it receives 403 Forbidden error message.

Error log

$ oj s https://codeforces.com/contest/1400/problem/E main.cpp
[INFO] online-judge-tools 10.1.2 (+ online-judge-api-client 10.3.2)
[INFO] read history from: /Users/morio/Library/Caches/online-judge-tools/download-history.jsonl
[INFO] found urls in history:
https://codeforces.com/contest/1400/problem/E
[INFO] problem recognized: CodeforcesProblem.from_url('https://codeforces.com/contest/1400/problem/E'): https://codeforces.com/contest/1400/problem/E
[INFO] code (9087 byte):

** MY SOLUTION **

[INFO] load cookie from: /Users/morio/Library/Application Support/online-judge-tools/cookie.jar
[NETWORK] GET: https://codeforces.com/contest/1400/problem/E
[NETWORK] 200 OK
[INFO] both GCC and Clang are available for C++ compiler
[INFO] use: GCC
[INFO] chosen language: 61 (GNU G++17 9.2.0 (64 bit, msys 2))
[INFO] sleep(3.00)
Are you sure? [y/N] y
[NETWORK] GET: https://codeforces.com/contest/1400/problem/E
[NETWORK] 200 OK
[NETWORK] POST: https://codeforces.com/contest/1400/problem/E
[NETWORK] 403 Forbidden
[ERROR] 403 Client Error:  for url: https://codeforces.com/contest/1400/problem/E
Traceback (most recent call last):
  File "/Users/morio/.pyenv/versions/3.8.2/lib/python3.8/site-packages/onlinejudge_command/main.py", line 242, in main
    run_program(parsed, parser=parser)
  File "/Users/morio/.pyenv/versions/3.8.2/lib/python3.8/site-packages/onlinejudge_command/main.py", line 212, in run_program
    submit(args)
  File "/Users/morio/.pyenv/versions/3.8.2/lib/python3.8/site-packages/onlinejudge_command/subcommand/submit.py", line 131, in submit
    submission = problem.submit_code(code, language_id=LanguageId(args.language), session=sess)
  File "/Users/morio/.pyenv/versions/3.8.2/lib/python3.8/site-packages/onlinejudge/service/codeforces.py", line 378, in submit_code
    resp = form.request(session=session)
  File "/Users/morio/.pyenv/versions/3.8.2/lib/python3.8/site-packages/onlinejudge/_implementation/utils.py", line 107, in request
    return request(method, url, session=session, raise_for_status=raise_for_status, data=self.payload, files=self.files, **kwargs)
  File "/Users/morio/.pyenv/versions/3.8.2/lib/python3.8/site-packages/onlinejudge/_implementation/utils.py", line 172, in request
    resp.raise_for_status()
  File "/Users/morio/.pyenv/versions/3.8.2/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error:  for url: https://codeforces.com/contest/1400/problem/E

Environment

macOS Catalina 10.15.6 (19G2021)

$ python3 --version
Python 3.8.2

$ oj --version
[INFO] GET: https://pypi.org/pypi/online-judge-tools/json
[INFO] 200 OK
[INFO] GET: https://pypi.org/pypi/online-judge-api-client/json
[INFO] 200 OK
online-judge-tools 10.1.2 (+ online-judge-api-client 10.3.2)

Other notes

  • Will you try to create a pull request?
    • no

Downloading from Facebook Hacker Cup doesn't work

Description

The platform is updated and this brakes our program.
The new URLs are like https://www.facebook.com/codingcompetitions/hacker-cup/2020/qualification-round/problems/A

Also, now the old URLs like https://www.facebook.com/hackercup/problem/448364075989193/ are unavailable (without appropriate redirection).

Error log

$ oj d https://www.facebook.com/codingcompetitions/hacker-cup/2020/qualification-round/problems/A
[INFO] online-judge-tools 10.0.5 (+ online-judge-api-client 10.2.5)
[INFO] unknown problem: https://www.facebook.com/codingcompetitions/hacker-cup/2020/qualification-round/problems/A
[ERROR] The contest "https://www.facebook.com/codingcompetitions/hacker-cup/2020/qualification-round/problems/A" is not supported
Traceback (most recent call last):
  File "/home/user/GitHub/online-judge-tools/onlinejudge_command/main.py", line 242, in main
    run_program(parsed, parser=parser)
  File "/home/user/GitHub/online-judge-tools/onlinejudge_command/main.py", line 208, in run_program
    download(args)
  File "/home/user/GitHub/online-judge-tools/onlinejudge_command/subcommand/download.py", line 36, in download
    raise requests.exceptions.InvalidURL('The contest "%s" is not supported' % args.url)
requests.exceptions.InvalidURL: The contest "https://www.facebook.com/codingcompetitions/hacker-cup/2020/qualification-round/problems/A" is not supported

Import tests from online-judge-tools/oj

  • command_download_anarchygolf.py
  • command_download_atcoder.py
  • command_download_csacademy.py
  • command_download_hackerrank.py
  • command_download_invalid.py
  • command_download_kattis.py
  • command_download_library_checker.py
  • command_download_toph.py
  • command_download_yukicoder.py

oj can't download tests from Codeforces Educational (EDU)

Description

Codeforces have recently added an educational section (see https://codeforces.com/edu/courses) containing videos and associated practice problems. The problems have URLs such as https://codeforces.com/edu/course/2/lesson/2/1/practice/contest/269100/problem/A which oj doesn't recognise as valid problem URLs.

The source of this seems to be patterns used for Codeforces URLs in onlinejudge/service/codeforces.py.

Error log

oj -v download https://codeforces.com/edu/course/2/lesson/4/1/practice/contest/273169/problem/A
[DEBUG] load the cache for update checking: /home/anthony/.cache/online-judge-tools/pypi.json
[DEBUG] load the cache for update checking: /home/anthony/.cache/online-judge-tools/pypi.json
[DEBUG] args: Namespace(cookie=PosixPath('/home/anthony/.local/share/online-judge-tools/cookie.jar'), directory=None, dry_run=False, format=None, json=False, silent=False, subcommand='download', system=False, url='https://codeforces.com/edu/course/2/lesson/4/1/practice/contest/273169/problem/A', verbose=True, version=False, yukicoder_token=None)
[INFO] online-judge-tools 10.1.1 (+ online-judge-api-client 10.3.0)
[ERROR] unknown problem: https://codeforces.com/edu/course/2/lesson/4/1/practice/contest/273169/problem/A
[DEBUG] 
Traceback (most recent call last):
  File "/home/anthony/.local/lib/python3.8/site-packages/onlinejudge_command/main.py", line 242, in main
    run_program(parsed, parser=parser)
  File "/home/anthony/.local/lib/python3.8/site-packages/onlinejudge_command/main.py", line 208, in run_program
    download(args)
  File "/home/anthony/.local/lib/python3.8/site-packages/onlinejudge_command/subcommand/download.py", line 36, in download
    raise requests.exceptions.InvalidURL('The contest "%s" is not supported' % args.url)
requests.exceptions.InvalidURL: The contest "https://codeforces.com/edu/course/2/lesson/4/1/practice/contest/273169/problem/A" is not supported

[ERROR] The contest "https://codeforces.com/edu/course/2/lesson/4/1/practice/contest/273169/problem/A" is not supported
Traceback (most recent call last):
  File "/home/anthony/.local/lib/python3.8/site-packages/onlinejudge_command/main.py", line 242, in main
    run_program(parsed, parser=parser)
  File "/home/anthony/.local/lib/python3.8/site-packages/onlinejudge_command/main.py", line 208, in run_program
    download(args)
  File "/home/anthony/.local/lib/python3.8/site-packages/onlinejudge_command/subcommand/download.py", line 36, in download
    raise requests.exceptions.InvalidURL('The contest "%s" is not supported' % args.url)
requests.exceptions.InvalidURL: The contest "https://codeforces.com/edu/course/2/lesson/4/1/practice/contest/273169/problem/A" is not supported

Other notes

  • Will you try to create a pull request?
    • [tbd] investigating how easy this is to fix.

An `AssertionError` happens when try to submit to atcoder

Description

When I try to submit my program file to atcoder, the below error happens.

env

> uname -a   
Linux thinkPad 5.4.0-42-generic online-judge-tools/oj#46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

> pyhton3 -V
Python 3.8.2

> pip3 freeze | grep online  
online-judge-api-client==10.2.5
online-judge-tools==10.0.5

The command I run is

oj -v s https://atcoder.jp/contests/abc174/tasks/abc174_a main.py     

Error log

[DEBUG] args: Namespace(cookie=PosixPath('/home/user/.local/share/online-judge-tools/cookie.jar'), file=PosixPath('main.py'), format_dos2unix=False, format_rstrip=False, golf=False, guess=True, guess_cxx_compiler='gcc', guess_cxx_latest=True, guess_python_interpreter='cpython', guess_python_version='auto', language=None, open=True, subcommand='s', url='https://atcoder.jp/contests/abc174/tasks/abc174_a', verbose=True, version=False, wait=3, yes=False)
[*] online-judge-tools 10.0.5 (+ online-judge-api-client 10.2.5)
[*] code (343 byte):

**My solution here**

[DEBUG] User-Agent: online-judge-tools/10.0.5 (+https://github.com/online-judge-tools/oj)
[DEBUG] 
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/onlinejudge_command/main.py", line 232, in main
    run_program(namespace, parser=parser)
  File "/home/user/.local/lib/python3.8/site-packages/onlinejudge_command/main.py", line 211, in run_program
    submit(args)
  File "/home/user/.local/lib/python3.8/site-packages/onlinejudge_command/subcommand/submit.py", line 57, in submit
    language_dict = {language.id: language.name for language in problem.get_available_languages(session=sess)}  # type: Dict[LanguageId, str]
  File "/home/user/.local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 871, in get_available_languages
    data = self.download_data(session=session)
  File "/home/user/.local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 806, in download_data
    return AtCoderProblemDetailedData.from_html(html, problem=self, session=session, response=resp, timestamp=timestamp)
  File "/home/user/.local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 765, in from_html
    data = AtCoderProblemData._from_html(html, problem=problem, session=session, response=response, timestamp=timestamp)
  File "/home/user/.local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 568, in _from_html
    assert False
AssertionError

[ERROR] 

Other notes

I found that by adding .strip() to line 563 in onlinejudge/service/atcoder.py like this, this awesome tool works fine.

- time_limit, memory_limit = h2.find_next_sibling('p').text.split(' / ')
+ time_limit, memory_limit = h2.find_next_sibling('p').text.strip().split(' / ')

imp is deprecated

Summary / 概要

imp is deprecated in favor of importlib. setup.py uses imp.

Steps to reproduce / 再現方法

Run python setup.py.

environments:

$ python setup.py --version
setup.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
10.10.0

Expected behavior / 期待される挙動

No warning are emitted.

Actual behavior / 実際の挙動

$ python setup.py 
setup.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

Other notes / その他

  • Will you try to create a pull request?
    • yes

Codeforcesの入力欄の書式変更により oj-api get-problem が誤って動作する

Summary / 概要

Codeforcesでマルチテストケース問題のサンプルの書式が変更され、装飾用の情報が入力欄に追加された。公式blog

これのせいで、そのような問題に対する oj-api get-problem がでたらめな整数列を入力として認識する。

これらの問題の入力欄は、<pre> タグで改行をそのまま反映させる代わりに、各行を <div> で囲み、block で表示している。

非マルチテストケース (1713 F)

<div class="input"><div class="title">Input</div><pre>
3
0 2 1
</pre></div>

マルチテストケース (1710 A)

<div class="input">
    <div class="title">Input</div>
    <pre>
    <div class="test-example-line test-example-line-even test-example-line-0">6</div><div class="test-example-line test-example-line-odd test-example-line-1">4 6 3</div><div class="test-example-line test-example-line-odd test-example-line-1">12 9 8</div><div class="test-example-line test-example-line-even test-example-line-2">3 3 2</div><div class="test-example-line test-example-line-even test-example-line-2">8 8</div><div class="test-example-line test-example-line-odd test-example-line-3">3 3 2</div><div class="test-example-line test-example-line-odd test-example-line-3">9 5</div><div class="test-example-line test-example-line-even test-example-line-4">4 5 2</div><div class="test-example-line test-example-line-even test-example-line-4">10 11</div><div class="test-example-line test-example-line-odd test-example-line-5">5 4 2</div><div class="test-example-line test-example-line-odd test-example-line-5">9 11</div><div class="test-example-line test-example-line-even test-example-line-6">10 10 3</div><div class="test-example-line test-example-line-even test-example-line-6">11 45 14</div><div class="test-example-line test-example-line-even test-example-line-6"></div></pre>
</div>

解決方法として、<pre> 内部の <div><div> の区切りに改行を挿入することが考えられる。

Steps to reproduce / 再現方法

  1. oj download https://codeforces.com/contest/1714/problem/A

environments:

  • version: online-judge-tools 11.5.1 (+ online-judge-api-client 10.10.0)

Actual behavior / 実際の挙動

{
    "status": "ok",
    "messages": [],
    "result": {
        "url": "https://codeforces.com/contest/1714/problem/A",
        "tests": [
            {
                "input": "31 6 138 03 6 012 3014 456 02 23 3520 1510 30\n",
                "output": "1 47\n0 0\n10 55\n"
            }
        ],
        "context": {
            "contest": {
                "url": "https://codeforces.com/contest/1714",
                "name": "Codeforces Round #811 (Div. 3)"
            },
            "alphabet": "A"
        },
        "name": "Everyone Loves to Sleep"
    }
}

Other notes / その他

  • Will you try to create a pull request?
    • yes

AtCoderのテストケースをダウンロードする際、ファイルの末尾1文字が等しい問題が複数あると失敗する

Summary / 概要

標題の通りです

Steps to reproduce / 再現方法

oj download --system https://atcoder.jp/contests/typical90/tasks/typical90_041 --dropbox-token ***

environments:

  • version: online-judge-tools 11.5.1 (+ online-judge-api-client 10.10.1)

Expected behavior / 期待される挙動

正常に 041 のテストケースを取得できること

Actual behavior / 実際の挙動

複数の問題とマッチしたというエラーが出る

Error:  two folders match the problem: {'.tag': 'folder', 'name': '041', ...} and {'.tag': 'folder', 'name': '001', ...}

Cause / 原因

def _match_system_cases_problem_folder(self, *, contest_folder_name: str, problem_folder_name: str) -> bool:
# TODO: improve this function
return self.problem_id[-1] == problem_folder_name.lower()[-1]

において末尾1文字のみで一致判定を行っていることが原因と思われる

Other notes / その他

  • Will you try to create a pull request?
    • no

Failed to get-problem with compatibility flag on due to schema error

Summary / 概要

get-problem --compatibility failed due to schema error, can be reproduced with with any service.

Steps to reproduce / 再現方法

environments: Ubuntu 20.04, Python 3.8.12

  • version: (please write the versions of related things, e.g. $ oj --version)
    online-judge-api-client 10.10.0

Expected behavior / 期待される挙動

Getting a problem JSON

Actual behavior / 実際の挙動

Failed due to schema eror.

$ oj-api get-problem --compatibility http://jag2013spring.contest.atcoder.jp/tasks/icpc2013spring_a

Got following ERROR:

Traceback (most recent call last):                                                                     
  File "/opt/miniconda3/bin/oj-api", line 8, in <module>                                               
    sys.exit(main())                                                                                   
  File "/opt/miniconda3/lib/python3.8/site-packages/onlinejudge_api/main.py", line 356, in main
    jsonschema.validate(result, schema)                                                                
  File "/opt/miniconda3/lib/python3.8/site-packages/jsonschema/validators.py", line 1018, in validate  
    cls.check_schema(schema)                                                                           
  File "/opt/miniconda3/lib/python3.8/site-packages/jsonschema/validators.py", line 190, in check_schema
    raise exceptions.SchemaError.create_from(error)                                                    
jsonschema.exceptions.SchemaError: [{'type': 'object', 'properties': {'input': {'type': 'string'}, 'output': {'type': 'string'}}, 'required': ['input', 'output'], 'additionalProperties': False}] is not of typ
e 'object', 'boolean'                                                                                  
                                                                                                       
Failed validating 'type' in metaschema['allOf'][1]['properties']['properties']['additionalProperties']['$dynamicRef']['allOf'][1]['properties']['items']['$dynamicRef']['allOf'][0]:

Other notes / その他

  • Will you try to create a pull request?
    • yes

CodeChef parser is broken

$ oj -v d -n 'https://www.codechef.com/PLIN2020/problems/CNTSET'              
[DEBUG] args: Namespace(cookie=PosixPath('/home/ubuntu/.local/share/online-judge-tools/cookie.jar'), directory=None, dry_run=True, format=None, json=False, silent=False, subcommand='d', system=False, url='https://www.codechef.com/PLIN2020/problems/CNTSET', verbose=True, version=False, yukicoder_token=None)
[DEBUG] User-Agent: online-judge-tools/10.0.3 (+https://github.com/online-judge-tools/oj)
[DEBUG] 
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/onlinejudge_command/main.py", line 227, in main
    run_program(namespace, parser=parser)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/onlinejudge_command/main.py", line 202, in run_program
    download(args)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/onlinejudge_command/subcommand/download.py", line 47, in download
    samples = problem.download_sample_cases(session=sess)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/onlinejudge_workaround_for_conflict/service/codechef.py", line 59, in download_sample_cases
    raise SampleParseError('no markdown')
onlinejudge_workaround_for_conflict.type.SampleParseError: no markdown

[ERROR] no markdown
$ oj --version
online-judge-tools 10.0.3 (+ online-judge-api-client 10.0.8)

Import the feature to guess language ids for code from oj command

Description

Most online judges provides identifiers for programming languages and they are informally described.
For example, in the current Codeforces, ID = 59 is displayed as Microsoft Visual C++ 2017 and ID = 61 is displayed as GNU G++17 9.2.0 (64 bit, msys 2) and ID = 52 is displayed as Clang++17 Diagnostics. Such mappings is different for each online judges and varies on time (i.e. this means it's difficult to embed the correct IDs).

Now oj command has the feature to guess the correct language id from list of descriptions of language ids and user's source code. We should import the feature to guess into oj-api command and provide users of oj-api command.

For Admin, memory Limit Parsing of AtCoder is broken

Description

AtCoderのadmin権限を持っている問題に対し、service.atcoder.AtCoderProblem.get_available_languages()を呼ぶと以下のエラーで落ちます。Time Limit / Memory Limitの後ろに"Edit Task"が生まれ、パースが壊れるからです。

直す場合は yosupo06@7a63489 のようにすると直るはずです。(現状だとmemory_limit.endswith(' MB')が動かなくなる。)

バグの性質上 (対象者が狭い && テスト不可能) なのですが、対応する価値がありそうならば↑の修正をpull requestで送るつもりです。

Error log

Traceback (most recent call last):
  File "run_test.py", line 99, in <module>
    atcoder_test(source, problem)
  File "run_test.py", line 72, in atcoder_test
    atcoder_lang = problem.get_available_languages()
  File "/usr/local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 877, in get_available_languages
    data = self.download_data(session=session)
  File "/usr/local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 812, in download_data
    return AtCoderProblemDetailedData.from_html(html, problem=self, session=session, response=resp, timestamp=timestamp)
  File "/usr/local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 771, in from_html
    data = AtCoderProblemData._from_html(html, problem=problem, session=session, response=response, timestamp=timestamp)
  File "/usr/local/lib/python3.8/site-packages/onlinejudge/service/atcoder.py", line 592, in _from_html
    assert False

Other notes

  • Will you try to create a pull request?
    • yes

Improve logging: use colorlog

Now we are using logging similar to pwntools.
I think we should use to colorlog, because:

  1. Our users don't know the meaning of the signs like [*], [+], [!], ...
    • in contrast, about pwntools, users can easily know the meanings because they write pwntools as a library
  2. We want to customize logs to develop programs which depend on online-judge-tools.
    • about pwntools, users use it as a library but they write only one-time scripts

request for comments: public JSON API

@fukatani @Tatamo @kjnh10
online-judge-tools のライブラリ部分を分離して (もとからある Python API の他に) それを利用してツールを開発する人向けのコマンドを用意しました。JSON を吐きます。online-judge-tools/oj#687 で面倒だからやらないとか言ってたやつです。
JSON API は README に書いた仕様のものを予定していますが、なにかやばそうなところないか確認するのを手伝ってほしいです。後から気づくと互換性の問題が起きるので先に見つけておきたい。よろしくお願いします。

Codeforces login from terminal is broken and download samples requires login

Description

A few user from my project ai-virtual-assistant inform me they tried to login into codeforces that is not working. Then i looked it into this it seems like oj-api login for codeforces not working. i tried to login manually even tried to login using oj-tools still it is not working. I hope you will check it.

Error log

INFO:root:online-judge-api-client 10.2.1
INFO:root:sleep 1.000000 sec
INFO:onlinejudge._implementation.logging:[-] unknown problem: https://codeforces.com/
INFO:onlinejudge._implementation.logging:[-] unknown contest: https://codeforces.com/
INFO:onlinejudge._implementation.logging:[x] service recognized: CodeforcesService.from_url('https://codeforces.com/'): https://codeforces.com/
INFO:onlinejudge._implementation.logging:[x] load cookie from: /home/saurav/.local/share/online-judge-tools/cookie.jar
INFO:onlinejudge._implementation.logging:[x] GET: https://codeforces.com/enter
INFO:onlinejudge._implementation.logging:[x] 200 OK
ERROR:root:
Traceback (most recent call last):
  File "/home/saurav/.local/lib/python3.8/site-packages/onlinejudge_api/main.py", line 279, in main
    result = login_service.main(service, username=parsed.username, password=parsed.password, check_only=parsed.check, session=session)
  File "/home/saurav/.local/lib/python3.8/site-packages/onlinejudge_api/login_service.py", line 40, in main
    service.login(get_credentials=get_credentials, session=session)
  File "/home/saurav/.local/lib/python3.8/site-packages/onlinejudge/service/codeforces.py", line 44, in login
    form = utils.FormSender(form, url=resp.url)
  File "/home/saurav/.local/lib/python3.8/site-packages/onlinejudge/_implementation/utils.py", line 61, in __init__
    assert isinstance(form, bs4.Tag)
AssertionError

Add JSON API for submissions

I think the below is a example of the minimal interface.

{
  "url": "https://atcoder.jp/contests/abc160/submissions/11991846",
  "problem": {
    "url": "https://atcoder.jp/contests/abc160/tasks/abc160_a",
    "name": "Coffee"
  },
  "verdict": "AC",
  "code": "#!/usr/bin/env python3_, _, c, d, e, f = input()\nprint(c == d and e == f and 'Yes' or 'No')\n"
}

memo:

  • When we add the field for users, we should make a user object like "user": { "url": "https://atcoder.jp/users/kimiyuki", "name": "kimiyuki" }
  • When we add the field for language, we should use the language object like "language": { "id": null, "description": "Python3 (3.4.3)" }
  • If we add a field for memory usage, I think we should use MB for memory usage to keep the compatibility with get-problem, which also maintain the compatibility with competitive-companion. Or, we should use bytes for all fields and break the compatibility with competitive-companion.
  • Some people want this API to show verdicts in real-time.

TODO

  • decide the public API
  • add more contest parsers
  • add more problem metadata parsers
  • add tests for oj-api command (for command itself, not for parsers)
  • add tests for parsers modules (import tests which are once removed due to the dependency to oj)
  • upload to PyPI

'403 Client Error' while submitting to codeforces

Description

while submitting to codeforces I'm receiving 403 Client Error message.

Error log

$ oj s https://codeforces.com/contest/1451/problem/A E:\CP\CF\A\A.cpp --wait=0 --yes --no-open -l 54
[INFO] online-judge-tools 11.1.1 (+ online-judge-api-client 10.5.0)
[WARNING] cannot guess URL since the given file is not in the current directory
[INFO] problem recognized: CodeforcesProblem.from_url('https://codeforces.com/contest/1451/problem/A'): https://codeforces.com/contest/1451/problem/A
[INFO] code (1865 byte):

** MY SOLUTION **

[INFO] load cookie from: C:\Users\nebir\AppData\Local\online-judge-tools\online-judge-tools\cookie.jar
[NETWORK] GET: https://codeforces.com/enter
[NETWORK] 302 Found
[INFO] You are logged in.
[NETWORK] GET: https://codeforces.com/contest/1451/problem/A
[NETWORK] 200 OK
[INFO] chosen language: 54 (GNU G++17 7.3.0)
[WARNING] the problem "https://codeforces.com/contest/1451/problem/A" is specified to submit, but no samples were downloaded in this directory. this may be mis-operation
[NETWORK] GET: https://codeforces.com/contest/1451/problem/A
[NETWORK] 200 OK
[NETWORK] POST: https://codeforces.com/contest/1451/problem/A
[NETWORK] 403 Forbidden
[ERROR] 403 Client Error:  for url: https://codeforces.com/contest/1451/problem/A
Traceback (most recent call last):
  File "C:\Users\nebir\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\onlinejudge\service\codeforces.py", line 389, in submit_code
    resp.raise_for_status()
  File "C:\Users\nebir\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\requests\models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error:  for url: https://codeforces.com/contest/1451/problem/A
[WARNING] You may use wrong User-Agent: 'online-judge-tools/11.1.1 (+https://github.com/online-judge-tools/oj)'
[FAILURE] submission failed

Environment

Windows 10 Pro
Build: 19042.630.

$ python --version
Python 3.8.6
$ oj --version
online-judge-tools 11.1.1 (+ online-judge-api-client 10.5.0)

`oj-api get-problem https://judge.yosupo.jp/problem/xxx` fails if it's executed in parallel

あまり想定されない使い方かもしれませんが、一応。
Library-Checkerの問題を並列に取得しようとするとエラーが発生します。

OS: Windos 10
Python: 3.8
online-judge-tools: 10.0.8
oj-api get-problem https://judge.yosupo.jp/problem/aplusb を同時に実行するとエラーが出る
以下実行結果

$ oj-api get-problem https://judge.yosupo.jp/problem/aplusb
INFO:root:sleep 1.000000 sec
INFO:onlinejudge._implementation.logging:[x] problem recognized: LibraryCheckerProblem.from_url('https://judge.yosupo.jp/problem/aplusb'): https://judge.yosupo.jp/problem/aplusb
INFO:onlinejudge._implementation.logging:[-] unknown contest: https://judge.yosupo.jp/problem/aplusb
INFO:onlinejudge._implementation.logging:[x] service recognized: LibraryCheckerService.from_url('https://judge.yosupo.jp/'): https://judge.yosupo.jp/problem/aplusb
INFO:onlinejudge._implementation.logging:[x] load cookie from: C:\Users\username\AppData\Local\online-judge-tools\online-judge-tools\cookie.jar
git version 2.21.0.windows.1
INFO:onlinejudge._implementation.logging:[x] $ git -C C:\Users\username\AppData\Local\online-judge-tools\online-judge-tools\Cache\library-checker-problems pull
error: cannot lock ref 'refs/remotes/origin/master': is at 2ae8b8d8f84069bd8a1b216f0d6a960f0b3d9859 but expected 95f8a5cbbf51de68e67c3662672d6e34a585fb6f
From https://github.com/yosupo06/library-checker-problems
 ! 95f8a5c..2ae8b8d  master     -> origin/master  (unable to update local ref)
ERROR:root:Command '['git', '-C', 'C:\\Users\\username\\AppData\\Local\\online-judge-tools\\online-judge-tools\\Cache\\library-checker-problems', 'pull']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "c:\path-to-python-dir\lib\site-packages\onlinejudge_api\main.py", line 236, in main
    result = get_problem.main(problem, is_system=parsed.system, is_full=parsed.full, is_compatibility=parsed.compatibility, session=session)
  File "c:\path-to-python-dir\lib\site-packages\onlinejudge_api\get_problem.py", line 294, in main
    tests = problem.download_system_cases(session=session)
  File "c:\path-to-python-dir\lib\site-packages\onlinejudge\service\library_checker.py", line 84, in download_system_cases
    self._generate_test_cases_in_cloned_repository()
  File "c:\path-to-python-dir\lib\site-packages\onlinejudge\service\library_checker.py", line 92, in _generate_test_cases_in_cloned_repository
LibraryCheckerService._update_cloned_repository()
  File "c:\path-to-python-dir\lib\site-packages\onlinejudge\service\library_checker.py", line 66, in _update_cloned_repository
    subprocess.check_call(['git', '-C', str(path), 'pull'], stdout=sys.stderr, stderr=sys.stderr)
  File "c:\path-to-python-dir\lib\subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', '-C', 'C:\\Users\\username\\AppData\\Local\\online-judge-tools\\online-judge-tools\\Cache\\library-checker-problems', 'pull']' returned non-zero exit status 1.

ちなみに何度か試しましたがほとんど再現しませんでした
(代わりにLibrary-Checkerのgenerate.py呼び出し中に衝突して死にましたが)

Remove the feature to submit code to Codeforces

I decided to remove the feature to submit code to Codeforces. It's already broken (#117), and I will not fix it.

There is three reasons. The feature is:

  1. ethically unrecommended
  2. technically difficult
  3. not so beneficial for our users

I don't want to maintain such a feature. Rather, I would like to focus more beneficial and interesting tasks, e.g. automatically analyzing problems and generating solutions (online-judge-tools/template-generator, kmyk/Jikka).

Of course I'll merge it if someone implement the feature appropriately and send it as a pull request. However I, @kmyk, won't implement it.

1. ethics

The submission feature is basically not welcomed by online judges, because such a feature can easily cause trouble.
For example, malicious attackers (or just foolish kiddie) may use the feature to make servers down.

Observing the behavior of Codeforces servers, I feel that Codeforces dislike automating submissions.
I know this fact, so I should not do such thing.

2. difficulty

Codeforces servers have some tricks to prevent automated submissions, because they don't welcome such submissions.
Cheating such servers is difficult, freaky, and uninteresting.

3. benefits

The feature to automate submissions gives users just convenience. It saves only few seconds. That's all.

Considering other features, this is too little benefits.
For example, the features for random testing will save a lot of time and sometimes allows users to solve problems which they cannot solve without such features.
However, automated submissions doesn't have such impact. It's enough to submit by hands.

I know some users likes such convenient (and non-important) features. They might even think this submission feature is the one of main features of our tooling. Maitaining this feature may help earning more users. Howerver, I want to develop my tool for users who use my tool for increasing ratings, not for convenience.

Convert `assert` to `SampleParseError` in some situation?

online-judge-tools/oj#637

By the way, is assertion in download_sample_cases is right?
If assert len(list(case.children)) == 2 is not satified, the page contents are not supported by oj.
Therefore, raise SampleParseError is prefer to assertion?

Anyway, this assertion issue handle by other PR. This PR is mergable.

アサーションはバグ検出のために仕込むものですが、assert len(list(case.children)) == 2や他のアサーションに引っかかった時、ojのバグというよりは想定しないイレギュラーな問題ページが存在したということで、SampleParseErrorを投げるのが正しい気がしてきました。どうでしょう?

基本的に私は賛成ですが、 refactoring の複雑さと実用性の向上で天秤にかけているところです。

アルゴ式のサポート

Description / 説明

アルゴ式: https://algo-method.com/ のサポート
サンプルケースの取得だけでも可能になると、ojでテストできるので、解き進めるのがスムーズになる。

Motivation / 動機

既存のコンテストよりも丁寧な解説&体系的な順序で、アルゴリズム・競プロ・数学を学べるサービスとしてじわじわ人気を増してきている。
https://twitter.com/hashtag/%E3%82%A2%E3%83%AB%E3%82%B4%E5%BC%8F?src=hashtag_click

自分で解いていて、手動でサンプルケースのファイルを作るのが手間に感じるようになってきた。

`oj-api login-service --check` fails

Summary / 概要

oj-api login-service --check がエラーを起こします

Steps to reproduce / 再現方法

  1. oj-api login-service --check "https://atcoder.jp/contests/"

environments:

  • version: online-judge-tools 11.5.1 (+ online-judge-api-client 10.10.0)

Expected behavior / 期待される挙動

{"status": "ok", "messages": [], "result": {"loggedIn": false}}

Actual behavior / 実際の挙動

$ oj-api login-service --check "https://atcoder.jp/contests/"
INFO:onlinejudge_api.main:online-judge-api-client 10.10.0
INFO:onlinejudge_api.main:sleep 1.000000 sec
Traceback (most recent call last):
  File "/home/nahco314/.pyenv/versions/3.8.2/bin/oj-api", line 10, in <module>
    sys.exit(main())
  File "/home/nahco314/.pyenv/versions/3.8.2/lib/python3.8/site-packages/onlinejudge_api/main.py", line 268, in main
    del os.environ['PASSWORD']
  File "/home/nahco314/.pyenv/versions/3.8.2/lib/python3.8/os.py", line 691, in __delitem__
    raise KeyError(key) from None
KeyError: 'PASSWORD'

Other notes / その他

oj login --check "https://atcoder.jp/contests/"は正常に動作します
USERNAMEとPASSWORDを指定するとmain.pyの29行目でAssertionErrorになります

Support LeetCode

PyConで聞いてみると、需要はあるらしいです。

Failed to parse Topcoder SRM 478 Div 1 Hard

Summary

oj d fails on https://community.topcoder.com/stat?c=problem_statement&pm=11026

To reproduce

$ oj d https://community.topcoder.com/stat?c=problem_statement&pm=11026

Expected behavior

The output of the example 2) should be:

4 0.19685958571981937 0.24397246802233483 0.31496640865458775 0.24420153760325805

Actual behavior

I got:

{0.19685958571981937, 0.24397246802233483, 0.31496640865458775, 0.24420153760325805 }

Error log

$ oj d -n 'https://community.topcoder.com/stat?c=problem_statement&pm=11026'

[*] sample 0
[x] input: example-0
1 00
1 00
1 58

[x] output: example-0
2 0.38461538461538464 0.6153846153846154


[*] sample 1
[x] input: example-1
2 00 00
2 00 00
2 21 11

[x] output: example-1
2 0.5888888888888889 0.4111111111111111


[*] sample 2
[x] input: example-2
3 0000 0000 0000
3 2284 0966 9334
3 1090 3942 4336

[x] output: example-2

{0.19685958571981937, 0.24397246802233483, 0.31496640865458775, 0.24420153760325805 }


[*] sample 3
[x] input: example-3
5 01010110 00011000 00001000 10001010 10111110
5 22218214 32244284 68402430 18140323 29043145
5 87688689 36101317 69474068 29337374 87255881

[x] output: example-3

{0.11930766223754977, 0.14033271060661345, 0.0652282589028571, 0.14448118133046356, 0.1981894622733832, 0.10743462836879789, 0.16411823601857622, 0.06090786026175882 }


[*] sample 4
[x] input: example-4
1 10
1 00
1 00

[x] output: example-4
2 1.0 0.0

Receiving `You may use wrong User-Agent` when submitting to Codeforces

Description

When I try to submit to Codeforces, it receives wrong User-Agent error message and this submission failed.

[WARNING] You may use wrong User-Agent: 'online-judge-tools/11.1.1 (+https://github.com/online-judge-tools/oj)'
[FAILURE] submission failed

Error log

I have already signed into Codeforces using CUI-based login.
$ oj login https://codeforces.com/contest/1447/problem/A
[INFO] online-judge-tools 11.1.1 (+ online-judge-api-client 10.5.0)
[INFO] service recognized: CodeforcesService.from_url('https://codeforces.com/'): https://codeforces.com/contest/1447/problem/A
[NETWORK] GET: https://codeforces.com/enter
[NETWORK] 200 OK
[FAILURE] You are not signed in.
[INFO] Trying to open Chrome via WebDriver...
[ERROR] Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

[INFO] Trying to open Firefox via WebDriver...
[ERROR] Message: 'geckodriver' executable needs to be in PATH.

[INFO] Trying to open Edge via WebDriver...
[ERROR] Message: 'MicrosoftWebDriver.exe' executable needs to be in PATH. Please download from http://go.microsoft.com/fwlink/?LinkId=619687

[INFO] Trying to open Internet Explorer via WebDriver...
[ERROR] Message: 'IEDriverServer.exe' executable needs to be in PATH. Please download from http://selenium-release.storage.googleapis.com/index.html and read up at https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver

[INFO] Trying to open Safari via WebDriver...
[ERROR] SafariDriver was not found; are you running Safari 10 or later? You can download Safari at https://developer.apple.com/safari/download/.
[INFO] Trying to open Opera via WebDriver...
[ERROR] Message: 'operadriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

[ERROR] No WebDriver is available.
[HINT] Please install a WebDriver.
See https://www.selenium.dev/documentation/en/webdriver/driver_requirements/

Detailed instructions:
    If you use Ubuntu:
        1. Run $ sudo apt install chromium-chromedriver firefox-geckodriver
    If you use Ubuntu under Windows Subsystem for Linux:
        1. Make a symbolic link from cookie.jar in WSL to cookie.jar out of WSL. For example, run $ ln -s /mnt/c/Users/%USERNAME%/AppData/Local/online-judge-tools/online-judge-tools/cookie.jar /home/ubuntu/.local/share/online-judge-tools/cookie.jar
        2. Use `oj login` outside of WSL
    If you use Windows:
        1. Install Chocolatey. See https://chocolatey.org/
        2. Run > choco install selenium-all-drivers

[WARNING] Switch to use CUI-based login instead of Selenium
[NETWORK] GET: https://codeforces.com/enter
[NETWORK] 200 OK
Username: seiran
Password:
[NETWORK] POST: https://codeforces.com/enter
[NETWORK] redirected to: https://codeforces.com/
[NETWORK] 200 OK
[INFO] Welcome, seiran.
[NETWORK] GET: https://codeforces.com/enter
[NETWORK] 302 Found
[SUCCESS] You have already signed in.
[INFO] save cookie to: /home/hotaru/.local/share/online-judge-tools/cookie.jar
$ oj s submit.cpp
[INFO] online-judge-tools 11.1.1 (+ online-judge-api-client 10.5.0)
[INFO] read history from: /home/hotaru/.cache/online-judge-tools/download-history.jsonl
[INFO] found urls in history:
https://codeforces.com/contest/1447/problem/A
[INFO] guessed problem: https://codeforces.com/contest/1447/problem/A
[INFO] problem recognized: CodeforcesProblem.from_url('https://codeforces.com/contest/1447/problem/A'): https://codeforces.com/contest/1447/problem/A
[INFO] code (1239 byte):

** MY SOLUTION **

[INFO] load cookie from: /home/hotaru/.local/share/online-judge-tools/cookie.jar
[NETWORK] GET: https://codeforces.com/enter
[NETWORK] 302 Found
[INFO] You are logged in.
[NETWORK] GET: https://codeforces.com/contest/1447/problem/A
[NETWORK] 200 OK
[INFO] both GCC and Clang are available for C++ compiler
[INFO] use: GCC
[INFO] chosen language: 61 (GNU G++17 9.2.0 (64 bit, msys 2))
[INFO] sleep(3.00)
Are you sure? [y/N] y
[NETWORK] GET: https://codeforces.com/contest/1447/problem/A
[NETWORK] 200 OK
[NETWORK] POST: https://codeforces.com/contest/1447/problem/A
[NETWORK] 403 Forbidden
[ERROR] 403 Client Error:  for url: https://codeforces.com/contest/1447/problem/A
Traceback (most recent call last):
  File "/home/hotaru/.local/lib/python3.8/site-packages/onlinejudge/service/codeforces.py", line 389, in submit_code
    resp.raise_for_status()
  File "/home/hotaru/.local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error:  for url: https://codeforces.com/contest/1447/problem/A
[WARNING] You may use wrong User-Agent: 'online-judge-tools/11.1.1 (+https://github.com/online-judge-tools/oj)'
[FAILURE] submission failed

Environment

Ubuntu under Windows Subsystem for Linux

Windows

  • エディション: Windows 10 Pro
  • バージョン: 20H2
  • OS ビルド: 19042.630

WSL

  • WSL version: WSL2
  • Linux distribution: Ubuntu 20.04.1 LTS

Python and oj

$ python3 --version
Python 3.8.5
$ oj --version
online-judge-tools 11.1.1 (+ online-judge-api-client 10.5.0)

Other notes

  • Will you try to create a pull request?
    • no

support Topcoder

I want use online-judge-tools for Topcoder SRM.
But there are two difficulties:

  1. Reading data from the Web Arena seems very difficult.
  2. We submit classes in SRM, not entire programs.

In terms of (1.), I have no idea how to implement it. Also it may require checking the terms of service. Of course only for practice problems, we can easily implement.
(2.) is the same problem as LeetCode #21. Should we make a new subcommand? But the subcommand requires much maintenance resources...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.