Comments (18)
ringgaard
commented on June 2, 2024
4
SLING supports 25 different languages at the moment: en, da, sv, no, de, fr, es, it, nl, pt, pl, fi, ca, eu, la, eo, cs, sh, hu, ro, el, ru, uk, sr, bg.
You can add more languages by making the following changes:
- Add the language to the language priority list here.
- Update the link_prefix map here to support the new languages. The is needed to parse image and template links correctly.
- Add a template definition for the language in
data/wiki/$LANG/templates.sling. You can use a generic template definition like this to begin with. Remember to change the template definition name, e.g. replace/wp/templates/plwith/wp/templates/kj.
Please notice that the Kwanyama Wikipedia seems to have been deprecated in 2007.
You can run --build_wiki for multiple languages in parallel by specifying a comma-separated list of languages codes, e.g. --languages en,de,es. The language-independent files (e.g. the Wikidata knowledge base) are stored in local/data/e/wiki and the language dependent files (e.g. Wikipedia articles) are stored in local/data/e/wiki/$LANG.
from sling.
ringgaard
commented on June 2, 2024
1
The alias table uses normalized (case and diacritics) phrase fingerprints to match aliases to entities.
The aliases come from a variety of different sources:
- Wikidata item names, titles, and name-like properties.
- Wikipedia anchors, redirects, disambiguations, infoboxes
- Variants over "reliable" aliases with common prefixes and small edit distance
These methods allows us to cover some morphological variations, but for morphology-rich languages this might not be enough. We are looking into Wikidata Lexemes as a way to expand the coverage of morphological variants. For languages with compounds (e.g. noun-noun [de,da] and adjective-noun [fi] etc.) we are looking into doing subtoken analysis (wordpieces), and for languages with case inflection of proper nouns (e.g. Russian), lemmatization might be useful.
from sling.
jzbjyb
commented on June 2, 2024
Hi @ringgaard, thanks a lot for your detailed comments! I followed your instructions for Afrikaans (af) and re-builded the system using tools/buildall.sh. Then I run --build_wiki and entity linking on af and there are not exceptions. However the documents-0000*-of-00010.rec files in both local/data/e/wiki/af/ and local/data/e/ner/af/ are empty (I use bazel-bin/tools/codex --frames $FILE to inspect these files). Any thoughts on why this happens?
Here is my language_priority:
const char *Wiki::language_priority[] = {
"en", "da", "sv", "no", "de", "fr", "es", "it", "nl", "pt", "pl", "fi",
"ca", "eu", "la", "eo", "cs", "sh", "hu", "ro",
"el", "ru", "uk", "sr", "bg",
"af",
nullptr,
};
Here is my link_prefix (commented with af):
static const LinkPrefixMap link_prefix = {
{"Archivo", WikiParser::IMAGE}, // es
{"Bestand", WikiParser::IMAGE}, // nl
{"Categoría", WikiParser::CATEGORY}, // es
{"Categoria", WikiParser::CATEGORY}, // it, pt, ca, la
{"Catégorie", WikiParser::CATEGORY}, // fr
{"Categorie", WikiParser::CATEGORY}, // nl, ro
{"Category", WikiParser::CATEGORY}, // en
{"Datei", WikiParser::IMAGE}, // de
{"Datoteka", WikiParser::IMAGE}, // sh
{"Dosiero", WikiParser::IMAGE}, // eo
{"Fájl", WikiParser::IMAGE}, // hu
{"Fasciculus", WikiParser::IMAGE}, // la
{"Ficheiro", WikiParser::IMAGE}, // pt
{"Fichier", WikiParser::IMAGE}, // fr
{"File", WikiParser::IMAGE}, // en, el
{"Fil", WikiParser::IMAGE}, // da, no, sv
{"Fișier", WikiParser::IMAGE}, // ro
{"Fitxategi", WikiParser::IMAGE}, // eu
{"Fitxer", WikiParser::IMAGE}, // ca
{"Image", WikiParser::IMAGE}, // en
{"Immagine", WikiParser::IMAGE}, // it
{"Kategória", WikiParser::CATEGORY}, // hu
{"Kategoria", WikiParser::CATEGORY}, // pl, eu
{"Kategorie", WikiParser::CATEGORY}, // de, cs, af
{"Kategorija", WikiParser::CATEGORY}, // sh
{"Kategorio", WikiParser::CATEGORY}, // eo
{"Kategori", WikiParser::CATEGORY}, // da, no, sv
{"Luokka", WikiParser::CATEGORY}, // fi
{"Media", WikiParser::IMAGE}, // en
{"Plik", WikiParser::IMAGE}, // pl
{"Soubor", WikiParser::IMAGE}, // cs
{"Tiedosto", WikiParser::IMAGE}, // fi
{"Κατηγορία", WikiParser::CATEGORY}, // el
{"Датотека", WikiParser::IMAGE}, // sr
{"Категорија", WikiParser::CATEGORY}, // sr
{"Категория", WikiParser::CATEGORY}, // bg, ru
{"Категорія", WikiParser::CATEGORY}, // uk
{"Файл", WikiParser::IMAGE}, // bg, ru, uk
{"Lêer", WikiParser::IMAGE}, // af
}
Here is the template:
{=/wp/templates/af
"!": {type: "text" text: "|"}
"!!": {type: "text" text: "||"}
")!": {type: "text" text: "]"}
"!((": {type: "text" text: "[["}
"))!": {type: "text" text: "]]"}
"(": {type: "text" text: "{"}
")": {type: "text" text: "}"}
"((": {type: "text" text: "{{"}
"))": {type: "text" text: "}}"}
"(((": {type: "text" text: "{{{"}
")))": {type: "text" text: "}}}"}
"·": {type: "text" text: " <b>·</b>"}
"•": {type: "text" text: " •"}
"=": {type: "text" text: "="}
}
from sling.
ringgaard
commented on June 2, 2024
What about if you try --build_wiki --language af?
from sling.
jzbjyb
commented on June 2, 2024
This is basically what I did. Since I have built the language-independent files (e.g. the Wikidata knowledge base) previously, so the commands I run for af are:
# build wiki
./run.sh --import_wikipedia --parse_wikipedia --extract_names --build_nametab --build_phrasetab --language af
# entity linking
./run.sh --build_idf --label_wiki --language af
from sling.
jzbjyb
commented on June 2, 2024
I just found that there are some unexpected errors in the log when reading the template file data/wiki/templates-af.sling in the --parse_wikipedia step:
[2020-02-07 21:42:34.120401: I sling/task/job.cc:66] Start af-wikipedia/parsing/wikipedia-documents
[2020-02-07 21:42:34.126799: I sling/nlp/wiki/wikipedia-documents.cc:61] Loading redirects from local/data/e/wiki/af/redirects.sling
[2020-02-07 21:42:34.146779: I sling/nlp/wiki/wikipedia-documents.cc:66] Loading wikimap from local/data/e/wiki/af/mapping.sling
[2020-02-07 21:42:34.149868: I sling/nlp/wiki/wikipedia-documents.cc:102] Loading template configuration
[2020-02-07 21:42:34.149884: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: !
[2020-02-07 21:42:34.149888: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: !!
[2020-02-07 21:42:34.149891: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: )!
[2020-02-07 21:42:34.149895: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: !((
[2020-02-07 21:42:34.149898: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: ))!
[2020-02-07 21:42:34.149902: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: (
[2020-02-07 21:42:34.149905: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: )
[2020-02-07 21:42:34.149908: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: ((
[2020-02-07 21:42:34.149911: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: ))
[2020-02-07 21:42:34.149914: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: (((
[2020-02-07 21:42:34.149918: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: )))
[2020-02-07 21:42:34.149921: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: ·
[2020-02-07 21:42:34.149926: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: •
[2020-02-07 21:42:34.149929: W sling/nlp/wiki/wiki-annotator.cc:167] Unknown template: =
It seems that the template file is not correctly processed.
from sling.
ringgaard
commented on June 2, 2024
The import-wikidata stage depends on the language priority settings, so you have to rerun --import_wikidata to include Wikidata-to-Wikipedia mappings for Afrikaans (af). The wikipedia-import stage only imports articles that are mapped to Wikidata.
from sling.
jzbjyb
commented on June 2, 2024
Thanks for pointing it out! I will try it and let you know how it goes. I have another question about how entity linking is performed in SLING. From the wikiflow diagram, I guess SLING builds an alias table (name/phrase table) for each entity to cover different ways of expressing the same entity and just uses a string-based matching to identify mentions in Wikipedia articles. Is the alias from Wikipedia anchor texts? If I am correct about this, can I assume that SLING can potentially link different inflections of the entity to its corresponding Wikidata entity (for morphology-rich language like Greek) because the anchor text might reflect some inflections?
from sling.
jzbjyb
commented on June 2, 2024
Hi! I run import_wikidata together with other flags mentioned above for Afrikaans (af), but I still got empty documents-0000*-of-00010.rec files and the same unknown template error. I guess import_wikidata is not the problem because I can successfully do entity linking for the 25 languages included in SLING without import_wikidata.
from sling.
ringgaard
commented on June 2, 2024
I have tried to run the pipeline with the changes to language_priority and link_prefix and it seems to work for me. You can check if af is included by looking at the wikidata item:
bazel-bin/tools/codex --frame --utf8 local/data/e/wiki/[email protected] --key Q5015
This should show the link to the af wikipedia page:
Q5015: {
=Q5015
:/w/item
name: "Mette Frederiksen"
lang: /lang/en
description: "27th and current prime minister of Denmark"
alias: {=#1
name: "Mette Frederiksen"
lang: /lang/af
sources: 2
}
...
/w/item/wikipedia: {=#36
/lang/af: /wp/af/Mette_Frederiksen
...
}
}
NB: You need to recompile the code after making the changes to the language priority (i.e. run tools/buildall.sh)
from sling.
jzbjyb
commented on June 2, 2024
I did recompile the code after making changes. There is only 20 languages in /w/item/wikipedia and /lang/af is not included in my case. Anyway, I will update the SLING repository to the latest version and rerun the pipeline (previously I was using an old version).
/w/item/wikipedia: {=#35
/lang/pl: /wp/pl/Mette_Frederiksen
/lang/pt: /wp/pt/Mette_Frederiksen
/lang/nl: /wp/nl/Mette_Frederiksen
/lang/it: /wp/it/Mette_Frederiksen
/lang/es: /wp/es/Mette_Frederiksen
/lang/fr: /wp/fr/Mette_Frederiksen
/lang/de: /wp/de/Mette_Frederiksen
/lang/no: /wp/no/Mette_Frederiksen
/lang/sv: /wp/sv/Mette_Frederiksen
/lang/da: /wp/da/Mette_Frederiksen
/lang/en: /wp/en/Mette_Frederiksen
/lang/fi: /wp/fi/Mette_Frederiksen
/lang/ca: /wp/ca/Mette_Frederiksen
/lang/la: /wp/la/Mette_Frederiksen
/lang/eo: /wp/eo/Mette_Frederiksen
/lang/cs: /wp/cs/Mette_Frederiksenová
/lang/hu: /wp/hu/Mette_Frederiksen
/lang/el: /wp/el/Μέττε_Φρεντέρικσεν
/lang/ru: /wp/ru/Фредериксен\,_Метте
/lang/uk: /wp/uk/Метте_Фредріксен
}
from sling.
jzbjyb
commented on June 2, 2024
After updating to the latest version and closely following what you did, I was able to run the whole pipeline (including entity linking, i.e., sliver annotation) on Afrikaans (af)! Thanks a bunch for your help! However, there is another (maybe minorish) issue occurs in the silver annotation stage, which seems to be related to date:
[2020-02-18 17:05:23.192865: I run.py:298] Silver-label af wikipedia
[2020-02-18 17:05:23.194649: I sling/task/job.cc:342] All systems GO
[2020-02-18 17:05:23.194663: I sling/task/job.cc:62] Starting stage #0
[2020-02-18 17:05:23.194668: I sling/task/job.cc:66] Start af-silver/record-file-writer[9/10]
[2020-02-18 17:05:23.194672: I sling/task/job.cc:66] Start af-silver/record-file-writer[8/10]
[2020-02-18 17:05:23.194675: I sling/task/job.cc:66] Start af-silver/record-file-writer[7/10]
[2020-02-18 17:05:23.194678: I sling/task/job.cc:66] Start af-silver/record-file-writer[6/10]
[2020-02-18 17:05:23.194681: I sling/task/job.cc:66] Start af-silver/record-file-writer[5/10]
[2020-02-18 17:05:23.194685: I sling/task/job.cc:66] Start af-silver/record-file-writer[4/10]
[2020-02-18 17:05:23.194692: I sling/task/job.cc:66] Start af-silver/record-file-writer[3/10]
[2020-02-18 17:05:23.194695: I sling/task/job.cc:66] Start af-silver/record-file-writer[2/10]
[2020-02-18 17:05:23.194698: I sling/task/job.cc:66] Start af-silver/record-file-writer[1/10]
[2020-02-18 17:05:23.194701: I sling/task/job.cc:66] Start af-silver/record-file-writer[0/10]
[2020-02-18 17:05:23.194704: I sling/task/job.cc:66] Start af-silver/sharder
[2020-02-18 17:05:23.194708: I sling/task/job.cc:66] Start af-silver/labeler
[2020-02-18 17:05:48.862675: I sling/task/job.cc:66] Start af-silver/record-file-reader[9/10]
[2020-02-18 17:05:48.862787: I sling/task/job.cc:66] Start af-silver/record-file-reader[8/10]
[2020-02-18 17:05:48.862831: I sling/task/job.cc:66] Start af-silver/record-file-reader[7/10]
[2020-02-18 17:05:48.862875: I sling/task/job.cc:66] Start af-silver/record-file-reader[6/10]
[2020-02-18 17:05:48.862906: I sling/task/job.cc:66] Start af-silver/record-file-reader[5/10]
[2020-02-18 17:05:48.862994: I sling/task/job.cc:66] Start af-silver/record-file-reader[4/10]
[2020-02-18 17:05:48.863055: I sling/task/job.cc:66] Start af-silver/record-file-reader[3/10]
[2020-02-18 17:05:48.863097: I sling/task/job.cc:66] Start af-silver/record-file-reader[2/10]
[2020-02-18 17:05:48.863166: I sling/task/job.cc:66] Start af-silver/record-file-reader[1/10]
[2020-02-18 17:05:48.863271: I sling/task/job.cc:66] Start af-silver/record-file-reader[0/10]
[2020-02-18 17:05:52.270750: F sling/nlp/kb/calendar.cc:41] Check failed: num >= 0
Aborted
from sling.
ringgaard
commented on June 2, 2024
I think you can fix this by changing this line in mentions.cc to:
if (span.aux.IsInt() && span.aux.AsInt() > 0) {
I have fixed this in my dev branch, but I haven't gotten around to get this submitted to the master yet.
from sling.
jzbjyb
commented on June 2, 2024
Thanks! It works for af after I fixed this. However for Chinese (zh), there is an issue related to document length, which I guess might be caused by the fact that Chinese needs word segmentation:
[2020-02-19 15:24:43.270190: I run.py:298] Silver-label zh wikipedia
[2020-02-19 15:24:43.275979: I sling/task/job.cc:342] All systems GO
[2020-02-19 15:24:43.276005: I sling/task/job.cc:62] Starting stage #0
[2020-02-19 15:24:43.276025: I sling/task/job.cc:66] Start zh-silver/record-file-writer[9/10]
[2020-02-19 15:24:43.276033: I sling/task/job.cc:66] Start zh-silver/record-file-writer[8/10]
[2020-02-19 15:24:43.276040: I sling/task/job.cc:66] Start zh-silver/record-file-writer[7/10]
[2020-02-19 15:24:43.276046: I sling/task/job.cc:66] Start zh-silver/record-file-writer[6/10]
[2020-02-19 15:24:43.276053: I sling/task/job.cc:66] Start zh-silver/record-file-writer[5/10]
[2020-02-19 15:24:43.276060: I sling/task/job.cc:66] Start zh-silver/record-file-writer[4/10]
[2020-02-19 15:24:43.276067: I sling/task/job.cc:66] Start zh-silver/record-file-writer[3/10]
[2020-02-19 15:24:43.276074: I sling/task/job.cc:66] Start zh-silver/record-file-writer[2/10]
[2020-02-19 15:24:43.276081: I sling/task/job.cc:66] Start zh-silver/record-file-writer[1/10]
[2020-02-19 15:24:43.276088: I sling/task/job.cc:66] Start zh-silver/record-file-writer[0/10]
[2020-02-19 15:24:43.276094: I sling/task/job.cc:66] Start zh-silver/sharder
[2020-02-19 15:24:43.276104: I sling/task/job.cc:66] Start zh-silver/labeler
[2020-02-19 15:25:28.683499: I sling/task/job.cc:66] Start zh-silver/record-file-reader[9/10]
[2020-02-19 15:25:28.683667: I sling/task/job.cc:66] Start zh-silver/record-file-reader[8/10]
[2020-02-19 15:25:28.683762: I sling/task/job.cc:66] Start zh-silver/record-file-reader[7/10]
[2020-02-19 15:25:28.683911: I sling/task/job.cc:66] Start zh-silver/record-file-reader[6/10]
[2020-02-19 15:25:28.684037: I sling/task/job.cc:66] Start zh-silver/record-file-reader[5/10]
[2020-02-19 15:25:28.684213: I sling/task/job.cc:66] Start zh-silver/record-file-reader[4/10]
[2020-02-19 15:25:28.684432: I sling/task/job.cc:66] Start zh-silver/record-file-reader[3/10]
[2020-02-19 15:25:28.684637: I sling/task/job.cc:66] Start zh-silver/record-file-reader[2/10]
[2020-02-19 15:25:28.684825: I sling/task/job.cc:66] Start zh-silver/record-file-reader[1/10]
[2020-02-19 15:25:28.685031: I sling/task/job.cc:66] Start zh-silver/record-file-reader[0/10]
[2020-02-19 15:25:35.806995: F sling/nlp/silver/phrases.cc:326] Check failed: document->length() >= pos + length (2 vs. 4)
Aborted
from sling.
ringgaard
commented on June 2, 2024
SLING only has a standard PTB/LDC tokenizer which assumes that words are separated by spaces or punctuation which does not work for languages like Chinese, Japanese, Korean, etc. For this you will need a language-specific segmenter.
from sling.
jzbjyb
commented on June 2, 2024
Thanks for the explanation! I tried Czech (cs) which is in the 25 languages that SLING supports and is space-separated, but I came across the same issue in silver_annotation:
[2020-02-21 14:53:00.928084: I run.py:298] Silver-label cs wikipedia
[2020-02-21 14:53:00.930988: I sling/task/job.cc:342] All systems GO
[2020-02-21 14:53:00.931009: I sling/task/job.cc:62] Starting stage #0
[2020-02-21 14:53:00.931015: I sling/task/job.cc:66] Start cs-silver/record-file-writer[9/10]
[2020-02-21 14:53:00.931020: I sling/task/job.cc:66] Start cs-silver/record-file-writer[8/10]
[2020-02-21 14:53:00.931024: I sling/task/job.cc:66] Start cs-silver/record-file-writer[7/10]
[2020-02-21 14:53:00.931028: I sling/task/job.cc:66] Start cs-silver/record-file-writer[6/10]
[2020-02-21 14:53:00.931031: I sling/task/job.cc:66] Start cs-silver/record-file-writer[5/10]
[2020-02-21 14:53:00.931035: I sling/task/job.cc:66] Start cs-silver/record-file-writer[4/10]
[2020-02-21 14:53:00.931039: I sling/task/job.cc:66] Start cs-silver/record-file-writer[3/10]
[2020-02-21 14:53:00.931043: I sling/task/job.cc:66] Start cs-silver/record-file-writer[2/10]
[2020-02-21 14:53:00.931046: I sling/task/job.cc:66] Start cs-silver/record-file-writer[1/10]
[2020-02-21 14:53:00.931050: I sling/task/job.cc:66] Start cs-silver/record-file-writer[0/10]
[2020-02-21 14:53:00.931054: I sling/task/job.cc:66] Start cs-silver/sharder
[2020-02-21 14:53:00.931059: I sling/task/job.cc:66] Start cs-silver/labeler
[2020-02-21 14:53:23.196786: I sling/task/job.cc:66] Start cs-silver/record-file-reader[9/10]
[2020-02-21 14:53:23.196892: I sling/task/job.cc:66] Start cs-silver/record-file-reader[8/10]
[2020-02-21 14:53:23.196927: I sling/task/job.cc:66] Start cs-silver/record-file-reader[7/10]
[2020-02-21 14:53:23.196963: I sling/task/job.cc:66] Start cs-silver/record-file-reader[6/10]
[2020-02-21 14:53:23.203243: I sling/task/job.cc:66] Start cs-silver/record-file-reader[5/10]
[2020-02-21 14:53:23.203483: I sling/task/job.cc:66] Start cs-silver/record-file-reader[4/10]
[2020-02-21 14:53:23.203607: I sling/task/job.cc:66] Start cs-silver/record-file-reader[3/10]
[2020-02-21 14:53:23.203704: I sling/task/job.cc:66] Start cs-silver/record-file-reader[2/10]
[2020-02-21 14:53:23.204023: I sling/task/job.cc:66] Start cs-silver/record-file-reader[1/10]
[2020-02-21 14:53:23.204299: I sling/task/job.cc:66] Start cs-silver/record-file-reader[0/10]
[2020-02-21 14:58:41.231379: F sling/nlp/silver/phrases.cc:326] Check failed: document->length() >= pos + length (2 vs. 3)
It seems that this problem is not caused by word segmentation. If word segmentation is indeed the problem, is there any workaround that I can do to integrate a Chinese segmenter in SLING? Another option: first do word segmentation on the Chinese Wikipeida dump so that hopefully SLING can take it as input as if it is already space-separated.
from sling.
ringgaard
commented on June 2, 2024
I found a problem in the phrase annotation cache that caused the check fault. I have fixed this issue in PR #439. We don't have any plan at the moment of supporting external tokenizers/segmenters. It is not enough to apply the segmentation to the Wikipedia documents. The tokenizer is used in a number other places to produce phrase fingerprints.
from sling.
jzbjyb
commented on June 2, 2024
Thank you so much for the quick fix! I updated the code and it works for me. I came across another problem for Arabic (ar) at build_phrasetab stage:
[2020-02-28 01:10:16.530546: I run.py:235] Build ar phrase table
[2020-02-28 01:10:16.535545: I sling/task/job.cc:342] All systems GO
[2020-02-28 01:10:16.535579: I sling/task/job.cc:62] Starting stage #0
[2020-02-28 01:10:16.535592: I sling/task/job.cc:66] Start phrase-table/phrase-table-builder
[2020-02-28 01:10:39.074219: I sling/task/job.cc:66] Start phrase-table/name-reader[9/10]
[2020-02-28 01:10:39.074334: I sling/task/job.cc:66] Start phrase-table/name-reader[8/10]
[2020-02-28 01:10:39.074361: I sling/task/job.cc:66] Start phrase-table/name-reader[7/10]
[2020-02-28 01:10:39.074394: I sling/task/job.cc:66] Start phrase-table/name-reader[6/10]
[2020-02-28 01:10:39.074455: I sling/task/job.cc:66] Start phrase-table/name-reader[5/10]
[2020-02-28 01:10:39.074502: I sling/task/job.cc:66] Start phrase-table/name-reader[4/10]
[2020-02-28 01:10:39.074544: I sling/task/job.cc:66] Start phrase-table/name-reader[3/10]
[2020-02-28 01:10:39.074611: I sling/task/job.cc:66] Start phrase-table/name-reader[2/10]
[2020-02-28 01:10:39.074676: I sling/task/job.cc:66] Start phrase-table/name-reader[1/10]
[2020-02-28 01:10:39.074734: I sling/task/job.cc:66] Start phrase-table/name-reader[0/10]
[2020-02-28 01:19:38.750168: I sling/task/job.cc:386] Channel 7 completed
[2020-02-28 01:19:38.750238: I sling/task/job.cc:395] Task phrase-table/name-reader[7/10] completed
[2020-02-28 01:19:38.750339: I sling/task/job.cc:400] Task phrase-table/name-reader[7/10] done
[2020-02-28 01:19:49.031820: I sling/task/job.cc:386] Channel 9 completed
[2020-02-28 01:19:49.031868: I sling/task/job.cc:395] Task phrase-table/name-reader[9/10] completed
[2020-02-28 01:19:49.031958: I sling/task/job.cc:400] Task phrase-table/name-reader[9/10] done
[2020-02-28 01:20:20.335167: I sling/task/job.cc:386] Channel 6 completed
[2020-02-28 01:20:20.335233: I sling/task/job.cc:395] Task phrase-table/name-reader[6/10] completed
[2020-02-28 01:20:20.335397: I sling/task/job.cc:400] Task phrase-table/name-reader[6/10] done
[2020-02-28 01:20:21.348222: I sling/task/job.cc:386] Channel 3 completed
[2020-02-28 01:20:21.348297: I sling/task/job.cc:395] Task phrase-table/name-reader[3/10] completed
[2020-02-28 01:20:21.348387: I sling/task/job.cc:400] Task phrase-table/name-reader[3/10] done
[2020-02-28 01:20:21.842938: I sling/task/job.cc:386] Channel 0 completed
[2020-02-28 01:20:21.842987: I sling/task/job.cc:395] Task phrase-table/name-reader[0/10] completed
[2020-02-28 01:20:21.843262: I sling/task/job.cc:400] Task phrase-table/name-reader[0/10] done
[2020-02-28 01:20:23.438207: I sling/task/job.cc:386] Channel 1 completed
[2020-02-28 01:20:23.438244: I sling/task/job.cc:395] Task phrase-table/name-reader[1/10] completed
[2020-02-28 01:20:23.438343: I sling/task/job.cc:400] Task phrase-table/name-reader[1/10] done
[2020-02-28 01:20:23.472297: I sling/task/job.cc:386] Channel 4 completed
[2020-02-28 01:20:23.472337: I sling/task/job.cc:395] Task phrase-table/name-reader[4/10] completed
[2020-02-28 01:20:23.472460: I sling/task/job.cc:400] Task phrase-table/name-reader[4/10] done
[2020-02-28 01:20:23.587317: I sling/task/job.cc:386] Channel 2 completed
[2020-02-28 01:20:23.587353: I sling/task/job.cc:395] Task phrase-table/name-reader[2/10] completed
[2020-02-28 01:20:23.587495: I sling/task/job.cc:400] Task phrase-table/name-reader[2/10] done
[2020-02-28 01:20:23.617490: I sling/task/job.cc:386] Channel 8 completed
[2020-02-28 01:20:23.617511: I sling/task/job.cc:395] Task phrase-table/name-reader[8/10] completed
[2020-02-28 01:20:23.617587: I sling/task/job.cc:400] Task phrase-table/name-reader[8/10] done
[2020-02-28 01:20:23.735916: I sling/task/job.cc:386] Channel 5 completed
[2020-02-28 01:20:23.735936: I sling/task/job.cc:395] Task phrase-table/name-reader[5/10] completed
[2020-02-28 01:20:23.736093: I sling/task/job.cc:395] Task phrase-table/phrase-table-builder completed
[2020-02-28 01:20:23.736134: I sling/task/job.cc:400] Task phrase-table/name-reader[5/10] done
[2020-02-28 01:20:23.736156: I sling/nlp/kb/phrase-table-builder.cc:114] Transfer aliases
[2020-02-28 01:20:33.298420: F sling/nlp/kb/phrase-table-builder.cc:287] Check failed: !item.IsNil() Q15566146
Aborted
I checked the existence of Q15566146 with the following command and it exists.
bazel-bin/tools/codex --frames --utf8 local/data/e/wiki/[email protected] --key Q15566146
from sling.
Related Issues (20)
- Wiki markup parsing is hard HOT 1
- Could the training data of 'caspar.flow' be released? HOT 2
- Open source projects using SLING?
- The caspar.flow appears to be un-trained HOT 9
- Problem converting Ontonotes corpus to SLING HOT 3
- Build Error: object of type 'ctx' has no field 'actions' HOT 2
- /data/wiki/templates-en.sling: No such file or directory HOT 4
- ImportError: No module named 'sling.pysling' HOT 3
- SLING for other languages HOT 1
- TypeError: 'ParserState' object is not iterable HOT 1
- SLING Frames - how can one retrieve multiple slot values? HOT 5
- What does "mapping.sling" do? HOT 4
- Training sling in Korean | How do I make a `.rec` file? HOT 4
- simple python example HOT 2
- SLING with many frames on a small dataset HOT 3
- Annotating data HOT 1
- Python API for the tokenizer HOT 5
- problem about "sling fuse_items" HOT 2
- Error: No kernel supports step features/GatherSum HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sling.