smutt / icann-dl Goto Github PK

board-cor broken for 2012,2013

icann_cor is by far the largest directory, and it is also the one least likely to be of interest. Maybe add an example of how to exclude it from rsync:
rsync -av --exclude="icann_cor/*" icann-pdf.depht.com::pub .

Right now we're not getting updates on OCTO publications. We catch new OCTO publications, but when already existing OCTO publications are updated and a new one published we don't register them as new. They reuse filenames and replace the old publication with the new one.

We'll need to do a HTTP HEAD and check the size. Not perfect, but the best we can do.

We'll also need to develop a naming scheme since they use the same filename for every version.

We don't want to grab stuff like this.
https://www.icann.org/board-advice-status-current.pdf

We want the actual/redirected filename, not the displayed name.

This goes for .xlsx and .pdf files the same. We need to look at Content-Disposition header to find actual filename.

https://stackoverflow.com/questions/6881034/curl-to-grab-remote-filename-after-following-location

We're missing some early SSAC publications like SAC005. Maybe just download them by hand as one offs.

The RSS feed is hopelessly broken and probably needs a complete rewrite. Depressing that there is no decent library for Atom or RSS in Python3.

For example

https://icann-pdf.depht.com/pub/icann_cor/2022/chalaby-to-drazek-04jun19-en.pdf
https://icann-pdf.depht.com/pub/icann_cor/2022/drazek-to-chalaby-30apr19-en.pdf
https://icann-pdf.depht.com/pub/icann_cor/2022/namazi-to-sataki-drazek-08apr19-en.pdf

Consider downloading blog posts from icann.org.