Comments (6)
Thanks Alan,
Excluding NCL_cleaned
that we now replace with text extracted from Teresa's .vert
file, the list gets a lot shorter:
$ find Irish_Data -type f | fgrep -v .tmx | fgrep -v NCI_cleaned | xargs -d'\n' grep -h -o -E "[&][#0-9a-zA-Z]+[;]" | sort | uniq -c
85 |
1022 [
1013 ]
2 &Dodgers;
15793 &
6496 '
9 >
17 <
2106 "
(I ran this in a snapshot taken in July before scripts and google docs documents were added.)
It is possible that some of these occurrences are correct, e.g. in technical material. However, it will be too much work to check each context. A global replace as part of the automatic pre-processing is the way forward.
from irish-bert.
Just to mention that I did a good bit of searching to try to find out what &Dodgers;
may be in September but couldn't find anything conclusive.
from irish-bert.
Great. I'm inclined to agree that a global replace is the optimal solution.
&Dodgers;
and &c;
deserved some attention as they don't appear to be valid Character Reference Entities (CRE).
TLDR;
I had a look at &Dodgers;
and it does not appear to be a valid CRE:
- It is not listed in w3.org charref table.
- It does not render in a browser i.e. >
&Dodgers;
< >&Dodgers;< whereas a valid CRE does render e.g. >&
< >&<
The 2 occurrences are in Paracrawl(v4) which is currently not included in training (it’s on the exclude list in ‘gdrive_filelist.csv’) but still worth handling it since we're aware of it.
I also had a look at &c
since it was the only other string that didn't render in the browser.
As with &Dodgers;
, points 1 and 2 above apply here also.
The 4 occurrences of &c
are in a single file "train.txt" in 4 passages of text that are almost identical. The &c
do not appear to have any meaning in the passages so they can be simply removed.
All other CREs listed above are valid and can be replaced with their equivalent character.
from irish-bert.
Thanks for throwing another pair of eyes on it.
&c
is always followed by .
and &c.
is an alternative way of writing "etc.". ("&" derives from a ligature of "e" and "t".)
Do you mean train.txt
in Irish_Data > processed_ga_files_for_BERT_runs
? That's excluded in gdrive_filelist.csv
and seems to be a file derived from other files in Irish_Data
and is from January, i.e. does not reflect various changes made since then.
Technical note: Replace "valid CRE" with "pre-defined CRE". Data providers may have had their own entities defined in custom DTD files. Such DTDs can be referenced in a document's <!DOCTYPE
element instead of a standardised public DTD.
from irish-bert.
Do you mean train.txt in Irish_Data > processed_ga_files_for_BERT_runs
This is the training file used for the first multilingual_bert
run. It uses almost all of the files in Irish_Data
(minus NCI and Paracrawl I believe) and I ran them through a similar script to text_processor.py
. I added this file for reproducibility, e.g. to compare to our first run. It is not used in any subsequent runs due to the pipeline changing.
from irish-bert.
Thanks for throwing another pair of eyes on it.
&c
is always followed by.
and&c.
is an alternative way of writing "etc.". ("&" derives from a ligature of "e" and "t".)
Sorry I should have typed &c;
I omitted the semicolon, but as you and James pointed out the file is no longer used so the four instances with the semicolons are moot.
Thanks for the note on "pre-defined CRE" :) and the '&c' explanation which I won't forg& :D
from irish-bert.
Related Issues (20)
- Paper: report random seeds of from scratch models HOT 1
- corpus statistics after de-duplication
- Increase number of parsers from 5 to 9
- Effect of corpus sampling on continued pre-training
- tag, branch and/or release code for reproducibility HOT 3
- report statistical power of test sets
- Include Scannell's corpus
- Reference for NCI paper Kilgarriff et al. HOT 1
- rclone is unable to find Theme A folder on Google Driver HOT 1
- Investigate gaHealth parallel corpus
- Add Irish subset of Indigenous Tweets
- Use ELRC and OPUS corpora directly
- Repair ligatures in NCI
- Add the Irish Crúbadán Web Corpus
- Add Irish subset of Indigenous Blogs
- Add the Gaois Corpus of Contemporary Irish
- Add the EduGA Corpus of Educational Materials
- Add the classical modern Irish Corpas Filíocht shiollach na Gaeilge
- Update HF model cards to refer to LREC paper HOT 1
- Repair fadas in NCI
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from irish-bert.