Comments (4)
Findings so far:
-
The
.vert
file does not have the utf-8 errors of thenci_cleaned
file. The.vert
file covers a wider range of characters from the "Latin" block and "μ" (U+03BC
). The cleaned version has only 19 million words. The Irish side of the NCI is supposed to have 30.2 million words. -
I found a missing
<s>
tag. Our new extractor script should use any of<s>
,<p>
,<doc>
and<file>
(and the respective closing tags) as trigger for a sentence boundary. -
Looking at 3 examples of English sentences in random locations, they seem to unexpectedly occur in bursts after Irish sentences in the same document. Maybe the corpus is a snapshot of ongoing translation with incomplete parts defaulting to the source language or English. If we can confirm this, this information can be used in language classification: When the classifier is not highly confident it should go with the class of the neighbouring sentences.
-
Some of the
<doc>
tags have atitle
attribute that has Irish text not part of the document itself. We could add this text as a separate sentence before the first sentence to get even more data. Same could be done with theauthor
attribute whenever thepubdate
field is not empty andmedium
one of "book" and "newspaper". -
Glue tags
<g/>
indicating that there was no space between the neighbouring tokens are not used. -
No occurrences of
<
or>
outside tags. -
Backslash is not used as an escape symbol, except maybe in the 7 occurrences of
\x\x13
. -
Number of
<p>
equals number of<s>
, i.e.<p>
are useless here. -
Some
</p>
and</s>
are missing. -
There are empty sentences.
-
Looking at the first 100 lines, it seems that all-caps headings and the first sentence of a section are not separated. However, re-doing the sentence splitting without the extra signals from markup in the original documents probably would produce an overall worse segmentation. For BERT this shouldn't matter as we learned from re-reading the bert and roberta papers over the last weeks but we need to keep this in mind for other work, e.g. using this data for semi-supervised training of dependency parsers with tri-training.
-
The file has no unicode utf-8 errors. It's strange then that NCI_cleaned has such errors. Since other encodings such as ISO 8859-* would be likely to produce a byte sequence that is not valid utf-8 this means that the .vert file is utf-8 encoded with very high likelihood, meaning that there was no reason to attempt conversion from one encoding to another and no opportunity to cause damage to the encoding. The fact that all 34 utf-8 errors in NCI_cleaned are at the start of a line each and use the byte '\xa4' also rules out random bit rot, e.g. on a low quality memory stick. ANyway, good news that the errors are not here.
-
There is a section where "&" is encoded as three tokens
&
,#38
and;
, i.e. on three lines. (HTML entity&
is&
in most modern character encodings.) Also, I found&
,gt
,;
split over 3 lines. -
There are plenty of
"
tokens. -
The unicode combining diaeresis character occurs 18 times. When slicing and recombining character sequences, care must be taken not to separate it from its preceding character, or at least not let it end up at the start of a token, not to fail strict unicode encoding checks.
-
Otherwise, the character table looks fine. There is indication for a small amount of foreign material but this may just be names. The fraction slash
U+2044
is used only 17 times. There are no fractions like 1/2 as a single character.
from irish-bert.
doc id="itgm0022"
, doc id="icgm1042"
and doc id="iwx00055"
have unescaped &
in the value of attributes. XML parser not happy. Implemented workaround in commit a5a27e2 line 52.
from irish-bert.
Update (with help from Teresa and Lauren):
- Tokens in tab-separated columns may contain space characters.
- Found more ampersand substitutions, including recursive
& #38 ; #38 ; #38 ;
, seeprint_sentence()
in https://github.com/jbrry/Irish-BERT/blob/master/scripts/extract_text_from_nci_vert.py - Some tokens contain unexpected hyphens, e.g.
Massa-chusetts
. Probably a problem with conversion from PDF. - The content inside some
<s>
elements is huge and spans many sentences. The longest element has 65094 tokens. The 100th longest has 5153 tokens. - There are cases of special characters replaced by spaces, e.g. in
'G idhlig
. - There are cases of words split into small pieces, e.g.
T UA R A SC Á I L B H L I A N TÚ I L A N O M B UD S MA N 1 9 9 7
.
from irish-bert.
Created separate issues for all issues mentioned above.
from irish-bert.
Related Issues (20)
- Paper: report random seeds of from scratch models HOT 1
- corpus statistics after de-duplication
- Increase number of parsers from 5 to 9
- Effect of corpus sampling on continued pre-training
- tag, branch and/or release code for reproducibility HOT 3
- report statistical power of test sets
- Include Scannell's corpus
- Reference for NCI paper Kilgarriff et al. HOT 1
- rclone is unable to find Theme A folder on Google Driver HOT 1
- Investigate gaHealth parallel corpus
- Add Irish subset of Indigenous Tweets
- Use ELRC and OPUS corpora directly
- Repair ligatures in NCI
- Add the Irish Crúbadán Web Corpus
- Add Irish subset of Indigenous Blogs
- Add the Gaois Corpus of Contemporary Irish
- Add the EduGA Corpus of Educational Materials
- Add the classical modern Irish Corpas Filíocht shiollach na Gaeilge
- Update HF model cards to refer to LREC paper HOT 1
- Repair fadas in NCI
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from irish-bert.