Corrupted DB download when packet loss on network about locus HOT 6 OPEN

What a bizarre issue.

I used your commands for inducing packet loss but couldn't replicate after dozens of attempts. Are you using OTP 20.3 from the website? Or did you build it directly from git? I did the latter, and performed the experiment using OTP 20.3.8.24.

I'm going to introduce a simple constraint - checking whether the size of the response body matches the value of the content-length response header (if present.) It won't solve the problem, but it might point us in the right direction.

Since you're downloading directly from MaxMind, it would also be fairly easy to automate the download of the checksum files, but I've avoided doing that due to a handful of bad edge cases, and in any case it wouldn't solve this particular problem, either.

from locus.

from locus.

I've pushed the content-length check to 75ec584edcb.

from locus.

[...] maybe allowing to gracefully handle intermittent failures.

Well, it does handle intermittent failures: if you can afford to boot your system without geolocalization being ready, I highly recommend it - download attempts will be retried every minute[1] unless you had the database already cached on the file system.

However, if geolocalization is absolutely required, then maybe a different strategy can be employed[2] - repeatedly await the database loader in a loop until it succeeds while perhaps logging details on any errors.

[1]: This particular interval is customizable through the pre_readiness_update_period loader option, in milliseconds.
[2]: According to the stracktrace of the crash, I believe this would be whatever code you've got on smlib_sup:init:22, on your application

from locus.

I ended up doing as suggested and adding busy-wait loop.

init_locus() ->
    ok = locus:start_loader(?GEODB_NAME, ?LOCUS_DB),
    case locus:wait_for_loader(?GEODB_NAME, timer:seconds(30)) of
        {ok, _DatabaseVersion} ->
            lager:info("Locus loaded database"),
            ok;
        Error ->
            locus:stop_loader(?GEODB_NAME),
            lager:error("Locus init error: ~p", [Error]),
            init_locus()
    end.

And with testing

2020-01-10 19:34:03.050 UTC [error] <0.1500.0>@smlib_sup:init_locus:30 Locus init error: {error,{body_size_mismatch,#{actual_content_length => "28691056",declared_content_length => "28704939"}}}
2020-01-10 19:34:03.050 UTC [error] <0.1501.0> [locus] geoip database failed to load (remote): {body_size_mismatch,#{actual_content_length => "28691056",declared_content_length => "28704939"}}
2020-01-10 19:34:08.054 UTC [error] <0.1512.0> [locus] geoip database download failed to start: timeout
2020-01-10 19:34:08.054 UTC [error] <0.1500.0>@smlib_sup:init_locus:30 Locus init error: {error,{timeout,waiting_stream_start}}
2020-01-10 19:34:08.054 UTC [error] <0.1512.0> [locus] geoip database failed to load (remote): {timeout,waiting_stream_start}
2020-01-10 19:34:10.408 UTC [info] <0.1500.0>@smlib_sup:init_locus:26 Locus loaded database

I still find it quite odd that httpc is returning a stream_end when it has clearly not received the fully body.

from locus.

locus 1.10.0, which was released earlier today, does somethings differently and might be of use to you in working around the packet loss issue:

• initially quick retries with exponential backoff as the amount of consecutive errors increases
• checksum verification when downloading from MaxMind
• a new function for awaiting which will block for the whole specified timeout (:await_loader)

from locus.