natlibfi / annif-corpora Goto Github PK

I can't build the jyu theses corpus, because two documents are missing:

• fulltext/jyu-theses/fin/2011-D-37180.url
• fulltext/jyu-theses/fin/2017-M-54486.url

I changed the Makefile to proceed with this step. Now, if a download fails, the source (.url) and destination (.pdf) file is deleted.

%.pdf: %.url
-   wget -q --no-use-server-timestamps -O $@ -i $< || rm -f $@
+   wget -q --tries=5 --no-use-server-timestamps -O $@ -i $< || rm -f $@ $<

Also, if the conversion from text to pdf fails the complete process stops. I fixed this with the following change (all conversion errors are ignored):

%.txt: %.pdf
-   pdftotext $<
+   pdftotext $< || true

Finally, it may be worth to mention in the documentation, that it is possible to speed up the process of downloading and converting with make -j16 pdf or make -j4 txt.

It looks like the links contained in eng-test are looking for .txts and everything in juy-theses eng is .url. This seems to be true of all of the example document corpuses. I haven't tested all of them, but it looks like all of them would break.