UTFGrid parsing is not working about pycaching HOT 13 OPEN

Hi all,
nowadays there is an API endpoint used on https://www.geocaching.com/play/map/ :

https://www.geocaching.com/api/proxy/web/search?box=50.256919%2C18.943434%2C50.205967%2C19.027462&take=500&asc=true&skip=0&sort=distance&origin=50.2314498%2C18.985447900000054

it returns JSON with a lot of cache information:

{
    "results": [
        {
            "id": 6358428,
            "name": "Ligota #5 Bolesław Leśmian",
            "code": "GC7A78G",
            "premiumOnly": false,
            "favoritePoints": 4,
            "geocacheType": 2,
            "containerType": 2,
            "difficulty": 1.5,
            "terrain": 1,
            "userFound": false,
            "userDidNotFind": false,
            "cacheStatus": 0,
            "postedCoordinates": {
                "latitude": 50.2316,
                "longitude": 18.983617
            },
            "detailsUrl": "/geocache/GC7A78G",
            "hasGeotour": false,
            "hasLogDraft": false,
            "placedDate": "2017-08-07T00:00:00",
            "owner": {
                "code": "PRP7QFB",
                "username": "vaski82"
            },
            "lastFoundDate": "2020-04-26T14:24:57"
        }, ...
    ],
    "total": 75
}

It could be probably used to reimplement Geocaching.search_quick method?

from pycaching.

Hi @tomasbedrich 👍 I'll do my bests!

I have stared developing this feature in https://github.com/pkubiak/pycaching/tree/feature-search-rect-via-api

As far as I have noticed, there is rate limiting after downloading 2000 cached, then you must wait for 30-60s and you can continue downloading another 2000 caches. During my experiment I was able to download 8600 cached in 3:40 min. Rate limit exceeding is signaled with HTTP Exception 429 Too many requests, I have handled it, so it is unnoticeable for users.

There is also small difference between BM/PMs. BM don't seepm_only cache coordinates, all other fields are the same for both.

from pycaching.

@mkouhia: Based on your previous great work, would you like to update the parsing code?

from pycaching.

from pycaching.

Tomas, is this functionality still relevant? Would you like someone to work on it?

from pycaching.

It makes sense for me to fix this only if it takes less than 1-2 days. But for me it is not possible to fully understand the relevant code to fix so quickly. So if you can do better, go for it. ;)

But honestly, unless you use it, I doubt that there are many people waiting for this fix.. Personally I think, that solving #29 may be more interesting.

from pycaching.

from pycaching.

@pkubiak Feel free to send a PR for this - in fact it seems like this endpoint is returning much more information than the tile-based approach. I am not completely sure about using this alternative endpoint, though.

from pycaching.

I am not completely sure about using this alternative endpoint, though.

In which sense? Are there some limitations / usage concerns (e.g. from Groundspeak) ?

from pycaching.

I do not want to decide this on my own - let's wait for @tomasbedrich and his opinion on this.

from pycaching.

Hi @pkubiak, thanks for discovering this interesting endpoint! I really like it. My only concern is that it may be more strictly rate-limited than normal search page. Do we miss something else @FriedrichFroebel? If testing proves it to be good, I would even consider replacing normal search using this endpoint.

from pycaching.

@tomasbedrich As far as I have seen, the cache data used in the grid parser is reflected in the new endpoint as well, so there does not seem to be a problem from this perspective. The only difference I could spot is that the tiles endpoint did not require an active session, while the new proxy API does - but this should not be a problem as we have an active session anyway. Rate-limiting and level of detail for BM/PM should be evaluated as well to avoid unexpected problems.

From my perspective, I would probably tend to provide this as an additional search method for now. Depending on how this turns out and how this is handled regarding rate limits, we can always make this the default search endpoint later on.

from pycaching.

Good point with the active session. This means that it is easier for Groundspeak to set and enforce some rate-limits for this endpoint.

Is there a different level of detail for BM/PMs? I haven't noticed.

@pkubiak Let's try to implement it as a new search_quick() for now. As @FriedrichFroebel says – we can make this default later.

from pycaching.