Re-create the trie tree about genre HOT 3 CLOSED

Leaving a comment since it might be helpful for someone in the future that wants to create their own trie from the current wikipedia version.
I used wikimapper to get all wikipedia titles since it is easier than paring the xml dump. Then I used a sql query to get all titles and redirects. As far as I can tell you only need to filter all disambiguation pages (containing "(disambiguation)") and subsections (containing "/").

select wikipedia_title from 
where wikidata_id is not NULL AND
wikipedia_title is not NULL AND
wikipedia_title != "" AND
wikipedia_title NOT LIKE '%(disambiguation)%' AND
 wikipedia_title NOT LIKE '%/%'""")

The code by HuiBinR works fine for creating the Trie.

from genre.

I used no filtering for creating the KILT trie but I used the KILT knowledge source http://dl.fbaipublicfiles.com/KILT/kilt_knowledgesource.json not the Wikipedia dump directly. Probably your piece of code is extracting many other page titles (maybe from special or deprecated pages) that should not be there (indeed 14M titles is too much as you can see from my screenshot of https://www.wikipedia.org as of today there are 6.2M pages so less than half of what you are extracting).

from genre.

Hi @bablf! It's very kind of you to share the SQL for filtering KILT titles. I am wondering if you are doing the same for entity linking usage.

from genre.