Comments (3)
jbrry
commented on June 12, 2024
All filtering is now done inside wiki-bert-pipeline on the bucketed files
so you no longer need to assign them to a single bucket for filtering. I
will remove this from the README soon.
This was done originally as an opusfilter configuration only takes in one
input file and writes a single output file. I added a script inside the
wiki-bert-pipeline which dynamically creates a filter configuration file
for each bucketed file so filtering can be done after tokenisation and
segmentation.
So to answer your question, a size of `100000` would probably be best (as
Oscar and NCI) are only ~300,000 and ~500,000 lines initially).
…On Sat 5 Dec 2020, 10:04 a.m. Joachim Wagner, ***@***.***> wrote:
Assigned #47 <#47> to @jbrry
<https://github.com/jbrry>.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#47 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFPGL6DJJQPMBCFHKDGU4OLSTIATNANCNFSM4UOOBLTQ>
.
--
__
Séanadh Ríomhphoist/_
Email Disclaimer__
**
Tá an ríomhphost seo agus
aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus
sin amháin é. Is féidir tuilleadh a léamh anseo.
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>
<https://www4.dcu.ie/iss/seanadh-riomhphoist.shtml>*
_
This e-mail and any
files transmitted with it are confidential and are intended solely for use
by the addressee. Read more here.
<https://www4.dcu.ie/iss/email-disclaimer.shtml> _
*_
from irish-bert.
jowagner
commented on June 12, 2024
Thanks. Does wikibert depend on OpusFilter? I've trouble installing variKN.
from irish-bert.
jbrry
commented on June 12, 2024
You can install OpusFilter without VariKN and eflomal as we don't use the language model or word alignment filters.
You can run the wiki-bert-pipeline without OpusFilter however. When you are on the external_data branch of our fork, there is a script external_scripts/gather_external_data.py which will copy files from this repository and place it into the wiki-bert-pipeline.
So assuming you had the NCI corpus downloaded, you would run the below to create the pre-training data for BERT with no filtering and no Wikipedia files:
python external_scripts/gather_external_data.py --datasets NCI --input-type processed --filter-type None --no-wiki
If you wanted to include basic, char and lang filters:
python external_scripts/gather_external_data.py --datasets NCI --input-type processed --filter-type basic+char-@+lang-@ --char-filter-threshold 1.0 --lang-filter-threshold 0.8 --no-wiki
from irish-bert.
Related Issues (20)
- Paper: report random seeds of from scratch models HOT 1
- corpus statistics after de-duplication
- Increase number of parsers from 5 to 9
- Effect of corpus sampling on continued pre-training
- tag, branch and/or release code for reproducibility HOT 3
- report statistical power of test sets
- Include Scannell's corpus
- Reference for NCI paper Kilgarriff et al. HOT 1
- rclone is unable to find Theme A folder on Google Driver HOT 1
- Investigate gaHealth parallel corpus
- Add Irish subset of Indigenous Tweets
- Use ELRC and OPUS corpora directly
- Repair ligatures in NCI
- Add the Irish Crúbadán Web Corpus
- Add Irish subset of Indigenous Blogs
- Add the Gaois Corpus of Contemporary Irish
- Add the EduGA Corpus of Educational Materials
- Add the classical modern Irish Corpas Filíocht shiollach na Gaeilge
- Update HF model cards to refer to LREC paper HOT 1
- Repair fadas in NCI
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from irish-bert.