Git Product home page Git Product logo

lipgn's Introduction

DEPRECATION NOTICE

Since a couple of days, nothing get downloaded because of the instantly appearing Lichess is rate-limiting us, waiting 120s ... message (see #11).

Having check with Thibault (lichess main dev) :

that API is still supported but it's being hammered so hard, it has hit the global rate limit :("

this means that lichess has closed this path, and thus, liPGN will never work again as-is without quite a heavy code refactoring...

But Thibault also give you a new way of doing it, directly via the lichess API (thus without having to use liPGN) : https://lichess.org/api#operation/gamesExportUsername

This new API simply means that liPGN is not needed anymore ! and that anyone can use it on it's own to retrieve directly a PGN formatted file. Though, advanced filtering and local data caching, that liPGN was providing, won't be available anymore...

I am not planning on moving liPGN to the new lichess API as it is not really suited for parsing (JSON format has been replaced by a plain flat PGN format).

So Long and Thanks for all the Fish.

Lichess game downloader

Build Status

This program uses the lichess REST API to retrieve all the games of a user and transform them into a big PGN file that you can then import into your favourite game analysis tool (ie. scid)

Usage

java -jar liPGN.jar [options]

For instance, if I want to prepare against Thibault for my next correspondance game:

java -jar liPGN.jar -u thibault -p unlimited

And if I want to prep for my next standard game against John Bartholomew:

java -jar liPGN.jar -u Fins --variant standard --speed classical,blitz

And if I want to challenge JannLee:

java -jar liPGN.jar -u JannLee --variant crazyHouse --speed classical,blitz

Options

  -q, --quiet                                            Don't output messages on console
  -C, --casual                                           Also handle casual games
  -v, --variant <v1,v2...>   standard                    Handle games for the given variant
  (standard/chess960/kinfOfTheHill/threeCheck/racingKings/horde/crazyHouse/antichess/atomic)
  -p, --speed <s1,s2...>                                 Handle games for the given speed (bullet/blitz/classical/unlimited). By default: all.
  -U, --url <URL>            http://en.lichess.org/api/  URL of the API
  -s, --store <store>        plocal:db                   The store to use for keeping the data (use 'memory:tmp' for a transient run)
  -c, --color <color>                                    Handle games for the given color (white/black)
  -R, --refresh-all                                      Refresh all games, even if some are already in the local DB (useful if you break a previous import with Ctrl-C...)
  -S, --no-sync                                          Don't sync the games with the server
  -t, --with-times                                       Decorate the PGN with the move times
  -u, --username <username>                              The username for whom to retrieve the games
  -o, --output <file>                                    The file to output, use '-' for stdout. By default, output to '<username>.pgn'
      --template-pgn <file>                              A file to use for templating the PGN (markdown format).
      --template-move-pair <file>                        A file to use for templating a move pair (markdown format).
  -h, --help                                             Print this help

Build

lein uberjar

Templating

Templating is easily overridable using Markdown format and the vars returned by the Lichess API.

Default PGN template

[Event "{{id}}"]
[Site "{{speed}}"]
[Date "{{date}}"]
[Round "{{url}}"]
[White "{{players.white.name}}"]
[WhiteElo "{{players.white.elo}}"]
[Black "{{players.black.name}}"]
[BlackElo "{{players.black.elo}}"]
[Variant "{{variant}}"]
[Result "{{result}}"]

{{moves}}{{analysis}}{{result}}

Default move-pair template

{{n}}. {{white}} {{black}}

Notes

The standard behavior will create a ./db directory, hosting the local database (used to reduce exchanges with the lichess server). You can delete this directory to "flush caches".

To get around this, you can use a transient DB (in RAM, won't be persistent) using --store memory:tmp

License

Copyright © 2016 Sylvain Tedoldi

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.

lipgn's People

Contributors

lukhas avatar s-ted avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

lipgn's Issues

no games found for given color

I tried extracting games from a certain player with a specified color, but no games are found. This player had 1490 games, so I'd estimate it should return around half that number.
https://gyazo.com/6efab040ad3c90acb1517a6f4c6d39af

When leaving the "-c black" part out, it functions as expected, retrieving all games.

I have tried it on 3 other users with the same result. Someone else (Seb) also tried it on one of those users and it gave him the same result, even though he hadn't encountered the problem before. Seb had tried it on the user "noklu" shortly before without any problem, and suggested I try it with that one. I did, and retrieving his white games one also worked fine for me. I have no idea why it would work in one case and not in others...

To reproduce: java -jar liPGN.jar -u Blekinge -c black

Some users are case-sensitive

lichess.org/@/noklu ("NoKlu") works with "-u noklu", but lichess.org/@/marstem "MarStem" does not work with "-u marstem".

Getting a fatal error for all users

`
WARNING: seqable? already refers to: #'clojure.core/seqable? in namespace: clojure.core.incubator, being replaced by: #'clojure.core.incubator/seqable?
WARNING: seqable? already refers to: #'clojure.core/seqable? in namespace: clostache.parser, being replaced by: #'clojure.core.incubator/seqable?
Updating user [username] from server.
Retrieving user [username] games from server since 20180326T144334.281Z.

A fatal error occured! (clj-http: status 404)
`

I have googled around without any success.

liPGN broken with Java 9 update

$ java -version
openjdk version "9.0.4"
OpenJDK Runtime Environment (build 9.0.4+12-Debian-2)
OpenJDK 64-Bit Server VM (build 9.0.4+12-Debian-2, mixed mode)

After removing all my local leiningen install to be sure, I did a "lein uberjar" which succeeded, but then lipgn fails:

$ java -jar target/liPGN.jar --user-agent "liPGN-lukhas" -p classical,rapid,blitz,unlimited -c white -u lukhas
Exception in thread "main" java.lang.ExceptionInInitializerError
        at com.orientechnologies.common.profiler.OProfilerStub.<init>(OProfilerStub.java:38)
        at com.orientechnologies.orient.core.Orient.startup(Orient.java:199)
        at com.orientechnologies.orient.core.Orient.<clinit>(Orient.java:87)
        at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.<clinit>(ORecordSerializerStringAbstract.java:52)
        at com.orientechnologies.orient.core.serialization.serializer.record.ORecordSerializerFactory.<init>(ORecordSerializerFactory.java:47)
        at com.orientechnologies.orient.core.serialization.serializer.record.ORecordSerializerFactory.<clinit>(ORecordSerializerFactory.java:37)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.<clinit>(ODatabaseDocumentTx.java:130)
        at fr.tedoldi.lichess.game.retriever.orientdb.Dal.init(orientdb.clj:409)
        at fr.tedoldi.lichess.game.retriever.core$export_BANG_.invokeStatic(core.clj:47)
        at fr.tedoldi.lichess.game.retriever.core$export_BANG_.invoke(core.clj:31)
        at fr.tedoldi.lichess.game.retriever.main$run.invokeStatic(main.clj:60)
        at fr.tedoldi.lichess.game.retriever.main$run.invoke(main.clj:59)
        at fr.tedoldi.lichess.game.retriever.main$_main.invokeStatic(main.clj:84)
        at fr.tedoldi.lichess.game.retriever.main$_main.doInvoke(main.clj:69)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at fr.tedoldi.lichess.game.retriever.main.main(Unknown Source)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make public long com.sun.management.internal.OperatingSystemImpl.getTotalPhysicalMemorySize() accessible: module jdk.management does not "opens com.sun.management.internal" to unnamed module @44b940a2
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:337)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:281)
        at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:198)
        at java.base/java.lang.reflect.Method.setAccessible(Method.java:192)
        at com.orientechnologies.orient.core.config.OGlobalConfiguration.autoConfigDiskCacheSize(OGlobalConfiguration.java:775)
        at com.orientechnologies.orient.core.config.OGlobalConfiguration.autoConfig(OGlobalConfiguration.java:757)
        at com.orientechnologies.orient.core.config.OGlobalConfiguration.<clinit>(OGlobalConfiguration.java:660)
        ... 16 more

Something changed from a program that was perfect for me

Until 4 months ago, Lipgn works perfectly for me, now when I run the program there will appear some strange messages in the middle and then it goes into some kind of infinite loop of the following message: "Lichess is rate-limiting us, waiting 120s ... "

Here is an example of some strange messages:

C: \ Users \ x \ Desktop \ x> java -jar liPGN.jar -u robertobrunopl -c black
Picked up _JAVA_OPTIONS: -Xverify: none
WARNING: seqable? already refers to: # 'clojure.core / seqable? in namespace: clojure.core.incubator, being replaced by: # 'clojure.core.incubator / seqable?
WARNING: seqable? already refers to: # 'clojure.core / seqable? in namespace: clostache.parser, being replaced by: # 'clojure.core.incubator / seqable?
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.orientechnologies.common.directmemory.OByteBufferPool (file: / C: /Users/x/Desktop/x/liPGN.jar) to method sun.nio.ch.DirectBuffer.cleaner ()
WARNING: Please consider reporting this to the maintainers of com.orientechnologies.common.directmemory.OByteBufferPool
WARNING: Use --illegal-access = warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release


I dont have the slightest idea what this message means, but below the program continued normally and then I didnt question anything, but then an infinite loop of the message "Lichess is rate-limiting us, waiting 120s ..." and never download any page or anything. :


[34mUpdating user robertobrunopl from server.
? [0m? [34mRetrieving user robertobrunopl games from server since day one.
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...
? [0m? [36mLichess is rate-limiting us, waiting 120s ...

Yes I'm new in programming and using this stuff on windows, probably I should be making some kind of mistake, but most people who use your program should also be new and use windows, please god Sylvain Tedoldi saves us. Tell us a method of opening this using an .bat file without errors

User-agent how to use it

I have problems downloading user games, every time I download it, it sends this message "Lichess is rate-limiting us, waiting 120 seconds", and I know that the User-agent has been made available, but I have no idea how to configure it, if I should write something in the .bat file or edit some file, can the program creator help me please?

Configurable User-Agent as an option

To circumvent rate-limiting that was designed to block dumb bots and not users, liPGN could make the User-Agent header a user-settable option.

API disabled for now

Hey, I'm lichess dev.

I'm sorry I had to disable this API endpoint yesterday, because someone uses it to DDoS lichess with a botnet - bypassing the IP rate-limits.

To restore it we'll need an oauth2 server implementation on lichess, so the rate limit can be applied per user.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.