cuny-cl / wikipron Goto Github PK
View Code? Open in Web Editor NEWMassively multilingual pronunciation mining
License: Apache License 2.0
Massively multilingual pronunciation mining
License: Apache License 2.0
Malay has a decent number of Arabic spellings.
Specifying the "script"
component in languages.json
and then regenerating ought to take care of it.
"script": {
"latn": "Latin",
"ara": "Arabic"
}
During the big scrape, wikipron worked on scraping Chinese for about 24 hours and the tsvs generated only contained a few entries.
Chinese entries on Wiktionary appear to have IPA information in <li>
elements within a collapsible <div>
. As mentioned by Jackson here, these entries often contain IPA transcriptions for various Chinese languages and dialects within those languages. Mandarin IPA transcriptions are often given in Sinological IPA, which may contain unofficial(?) IPA symbols.
These entries are perhaps causing problems for _LI_SELECTOR_TEMPLATE
in config.py
because of intervening <small>
elements between the <sup>
and <span class="IPA">
that we want to target?
(Certain Chinese entries will simply link to other entries.)
We noticed some duplicate entries in Icelandic such that one has the segmented pronunciation and one doesn't (example). We don't yet know where this comes from but it only seems to affect words that begin with þ, and none of these examples seem to have ended up in our Icelandic sample for the paper.
For an example, see #100. I think these are all false positives...timeout should be increased substantially.
For various reasons we get true duplicate entries (i.e., same graphemic form, same phonemic form) in the big scrape. For example I I don't think this is exactly a bug; this could arise if our processing of the graphemic and/or phonemic form causes a duplicate pair to be formed.
Example: phonemic Icelandic has 77 duplicates.
Two possible solutions:
set
of the pairs as we go. If we find a pair already in the set, we don't print it.sort | uniq
to every TSV file at the end of the big scrape (can turn this into a simple Bash script and add it to the README.I think (3) is but could be persuaded.
Both [khm] Khmer and [tha] Thai pull no data in phonetic or phonemic mode.
Both languages list their transcriptions within tables.
From https://github.com/kylebgorman/wikipron/issues/66#issuecomment-544214656 and subsequent discussion, we should remove the "require dialect label" option. We would lose the ability to do the following, neither of which I think is practically useful now:
The Wikipedia transcription guidelines say that the high front phoneme is /i/ but many transcriptions have /ɪ/ instead. We should fix this upstream (e.g., on Wiktionary itself) and rescrape.
See issue for more context.
We have found that if we construct models over Unicode codepoints, simple models will predict impossible IPA symbol combinations. (Most IPA diacritics do not exist in combining forms.) One solution @m-sean experimented with is as follows. If a codepoint is
do not split; otherwise split between codepoints. Sample logic here: https://github.com/yeonju123/pair_ngram_model/blob/master/build_sym.py#L13
I propose that, as a feature enhancement, we apply this logic in WikiPron, putting spaces between "symbols" so defined); this could also be made optional and triggered by flag.
(Then we will have to modify the baseline model tools to handle data chunked in this fashion.)
Thai IPA transcriptions are within tables. Example
This is related to the issue in #70 regarding Khmer in that the solution offered for scraping Khmer (creating a new selector template) looks like it would work for scraping Thai.
Thai IPA transcriptions also include tones. Does this present a problem? Are there pre- or post-processing steps that we are considering for handling languages that use IPA tones?
It looks like there are "North Wales" and "South Wales" entries.
Running codes.py
turns the following section in languages.js
:
"hanoi": "Hà Nội",
"hcmc": "Hồ Chí Minh City",
"hue": "Huế",
"tc": "Vinh, Thanh Chương",
"ht":"Hà Tĩnh"
into
"hanoi": "H\u00e0 N\u1ed9i",
"hcmc": "H\u1ed3 Ch\u00ed Minh City",
"hue": "Hu\u1ebf",
"tc": "Vinh, Thanh Ch\u01b0\u01a1ng",
"ht": "H\u00e0 T\u0129nh"
The Latin "big scrape" files are missing macrons (vertical bars above vowels indicating contrastive length) on the grapheme side. This is because, in Wiktionary backend, Latin words WITHOUT macrons are used as headwords. Example: https://en.wiktionary.org/wiki/malus#Etymology_2_2
Note that that page does pair the macroned form mālus with thhe pronunciation maːlus
like we want it to. It seems to me that we need to switch from using the backend headword to the word rendered on the page.
(As it happens, this is exactly the same issue that we discuss for the CoNLL-SIGMORPHON 2017 shared task in http://wellformedness.com/papers/gorman-etal-2019.pdf, which also scraped data from Wiktionary.)
While working on Japanese I noticed that lat.py
in the extract
directory does not call config.casefold
on the extracted words. I checked our most recent Latin data and indeed, despite casefold
being set to true in languages.json, the data is not all lower case.
This should be an easy fix but I'm submitting the issue because we will need to run Latin again.
Wiktionary sometimes does not provide IPAs, in which case, only graphemes are scraped.
I propose that the big scrape, instead of removing empty files, removes any file with less than, say, 100 entries. It is relatively clear to me such files will never be any use for modeling.
I would entertain suggestions that the threshold should be 1k instead.
Languages with less than 100 entries after remove_duplicates.sh
should be removed. We should perhaps move the logic which removes small files out of scrape.py
and into generate_summary.py
.
Once #152 is merged we should merge the whitelist functionality into languages.json
. Presumably it should simply be the case that for any language or dialect if a whitelist
value is specified, we load it and generate open two TSV files, one for the total data, one for the filtered data. This would also in theory make whitelist.py
obsolete.
I would suggest that we actually call the unfiltered one a special name or put it somewhere different.
Bringing over the discussion on Japanese from #62...
@lfashby @kylebgorman Please edit or comment as you see fit.
Taking this online after discussion with @lfashby...
One annoying thing about the big scrape scripts is that the TSV directory has a huge list of files only then followed by the README table below. One way to work around this is as follows:
wikipron/languages/wikipron/README.md
, which details how to re-run the big scrape, should be moved to the src
subdirectory, which doesn't have a README yetwikipron/languages/wikipron/tsv/README.md
, the autogenerated language table, should be moved to wikipron/languages/wikipron
instead.Since this will screw with the paths we should wait until after the next submission deadline to do it...
From Wiktionary:
Lü is a Southwestern Tai language spoken mainly in Yunnan province, China.
Lü has 301 entries on Wiktionary but we were only able to scrape one. This look to be another problem with our _LI_SELECTOR_TEMPLATE
. It seems all entries but the one we scraped have title set as title="Appendix:Lü pronunciation (page does not exist)"
. Here is an example. The sole entry we scraped has title set as title="wikipedia:Lü phonology"
I'd imagine fixing this is a very low priority, but I figured it is something we should be aware of.
Khmer IPA transcriptions are within tables. Example
An obvious, although perhaps not ideal, solution would simply be to create a new selector template in config.py
that targets <tr>
or <td>
elements containing a span[@class = "IPA"]
. You'd then have to have some logic in the yield_phn
function that checks if the language is Khmer or Thai and switches the selector template.
Afrikaans
pulls no data in both phonemic and phonetic modes.
As mentioned in #146, certain of our extraction functions do not support dialects.
Currently, dialects are handled by the _get_pron_xpath_selector
method in config.py
, the results of which are stored in pron_xpath_selector
. _yield_phn
in our default extraction function makes use of config.pron_xpath_selector
, as does Japanese (because its extraction function is used for targeting orthographic entries).
Our Chinese, Khmer, Latin, and Thai extraction functions do not use the default pron selector and skip over _yield_phn
by calling yield_pron
directly - meaning dialect information is ignored.
This lexicon really has to come in two parts.
Trying to scrape for Brunei (ISO 639 name; Brunei Malay is the Wiktionary name) led to the following error.
File ".../wikipron/config.py", line 128, in _get_language
language = iso639.to_name(key).split(";")[0]
File ".../lib/python3.7/site-packages/iso639/init.py", line 115, in to_name
raise NonExistentLanguageError('Language does not exist.')
iso639.NonExistentLanguageError: Language does not exist.
Brunei Malay's code is "kxd" which is ISO 639-3 only, meaning it isn't covered by the iso639 package we are using.
For the time being I'll just go ahead and add "kxd": "Brunei Malay"
myself to languagecodes.py. Perhaps it should be added in a separate pull request from the one I'll eventually submit with all the new data?
I'm not sure how this happened, shouldn't this have been caught by test_languagecodes.py? (Unless someone added 228 entries to Brunei Malay since test_languagecodes.py was last run.)
In the current checked-in languages.json
there is no dialect specification for either English or Spanish, whereas the earlier Bash script had one (and we want one).
For eng
: --dialect="US | General American"
For spa
: --dialect="Latin America"
The scraping should support all config options including cut-off date, and then next big scrape run should use, IDK, 11/1/2019 as a cutoff date?
Our English UK phonetic data has ~
separating alternate pronunciations similar to -
which we previously removed.
Ex: dance d æ n s ~ d a n s
Lucas has reported the following in https://github.com/kylebgorman/wikipron/issues/68#issuecomment-544209039:
Specifying an output config option prompts Wikipron to create a file that doesn't get written to (when using Wikipron from python).
This is a bug, as it means the output
option (for specifying whether scraped data is in a text file or in stdout) has inconsistent behavior being CLI and API usage:
Two options:
scrape
function call use the output
option somehow to match the CLI behavior, i.e., if output
is used, pipe the results to the specified text file and return `None, but if not used, then just yield the <word, pron> pairs (= current behavior).output
option completely. For CLI usage, if a user wants data in a text file, they could redirect stdout by >
, like $ wikipron fra > fra.tsv
(which we would document in the readme). From this perspective, the output
option seems unnecessary.Relatedly, I'm pretty open to improving WikiPron by making breaking changes at this early stage, as I'm suggesting another breaking change now, see https://github.com/kylebgorman/wikipron/issues/66#issuecomment-544214656). For this current ticket, I have a slight preference for just removing the output
option, since this would mean keeping the scrape
function behave in only one way (i.e., for always yielding the <word, pron> pairs, and not having an alternative behavior of returning None
).
Often we want to test changes to one or a small number of languages by running a partial "big scrape" and it would be nice to make this easier to do than making copies of languages.json
. Maybe something like:
./scrape.py --restriction=lit
Would run the scrape for Lithuanian (all dialects and scripts) but would use the appropriate metadata in languages.json
, and
./scrape.py
would run it for all languages in languages.json
.
There are lingering TSV files in wikipron/languages/wikipron/
made obsolete by the Big Scrape.
Once #152 is in we should add some kind of logging to the whitelister so we can get a list of "bad" words and the phones that go with them.
Perhaps it should also produce a some kind of structured report (JSON?) instead of just logging.
I started the second big scrape and while scraping for phonetic data from Albanian, Wikipron threw an error, the last line of which I'll reproduce below:
File ".../wikipron/config.py", line 73, in _parse_combining_modifiers
last_char = chars.pop()
IndexError: pop from empty list
The final line in the Albanian phonetic tsv is herë h ɛː ɾ
meaning the scrape likely failed on this entry which contains what looks like word initial aspiration.
I guess for words like the one that caused this error we would want to combine with next char ʰi d r ɔ ɟ ɛ n
?
Running the big scrape has revealed that the Japanese Wiktionary community decided to change the format of Japanese entries.
Our Japanese extraction function no longer works and the Japanese data gathered in this run of the big scrape needs to be discarded. We'll need to re-write the extraction function at some point, though for the time being I believe they are still ironing out the details of how the Japanese entry pages should look.
K'iche' has 117 words with IPA pronunciations but
wikipron quc | wc -l
only gives us 65 entries. I believe this is because many of the entries have whitespace in the graphemic side and that we just need to enable
"no_skip_spaces_word": true,
"no_skip_spaces_pron": true,
for this language.
IPA segmentation doesn't appear to correctly handle the tie bar for affricates. For instance, from https://github.com/kylebgorman/wikipron/pull/76#discussion_r339324885, I saw the following for Brazilian Portuguese:
adição a d͡ ʒ i s ɐ̃ w
Instead, we would want the following for this example:
adição a d͡ʒ i s ɐ̃ w
Flags should be renamed to be positive statements rather than negative ones, if possible.
[yue] is ISO 693-3 only, but should be handled by languagecodes.py in the same way that [nci] for Classical Nahuatl and [hbs] for Serbo-Croatian are.
Nonetheless, [yue] pulls nothing in phonetic and phonemic modes and does not even seem to begin scraping.
I notice a few obvious problems with Vietnamese:
languages.json
--no-skip-space
) here, or we could make a language specific-extractor, I suppose.Add "Classical", "Ecclesiastical", and "Vulgar" specifications for Latin.
Somewhat like Latin (#80) Serbo-Croatian headwords largely don't match the word given on the page. The headwords are not given with accent diacritics.
Cyrillic example
Latin example
Here's a quote from the Serbo-Croatian phonology page:
Accent diacritics are not used in the ordinary orthography, but only in the linguistic or language-learning literature (e.g. dictionaries, orthography and grammar books). However, there are very few minimal pairs where an error in accent can lead to misunderstanding.
Is it important to us to grab the words with accents, or can we get away with just grabbing the headword? (The people putting together the Serbo-Croatian Wiktionary IPA category elected to omit accents from the headword.)
Maltese lexicon has one entry in Arabic script. I suspect it's ended there from an etymology section somewhere. Is there a way to blacklist scripts during scraping?
As reported here there are some inconsistencies with /l/ and the dental stops. As [discussed here](https://en.wiktionary.org/wiki/Wiktionary:Information_desk/2020/April#Performing_bulk_edits, there is a pronunciation module and pron template for Bulgarian on Wiktionary; we may be just bulk-migrate Bulgarian to that template and re-scrape.
Wiktionary only lists Serbo-Croatian and does not have separate categories for Serbian, Croatian, Montenegrin, and Bosnian - the languages that the ISO 693-3 [hbs] code encapsulates.
Wikipron can handle scraping if given [hbs] but the resulting tsv files may not be of much use. [hbs] tsv files are therefore not included in pull request #61 .
:Add note to contributors of whitelists about what to do. Here is the basic guidance:
data/wikipron/tsv
../postprocess && ./generate_summary.py
.git add
the whitelist, the newly-filtered TSV files, and the README and summary TSVs modified, then commit, push, and file the PR. This will also allow the reviewer to quickly see how many forms are removed by whitelisting.Comment style is Python-like: phone/phoneme, two spaces, #
, space, then a title-cased expression with period at end. E.g., t̪ # Allophone of /t/.
The English data contains roughly 300-400 entries where the pronunciation is given incompletely, with a dash at the beginning, middle, or end of the entry (usually indicating that we're supposed to fill in the rest of the pronunciation ourselves). Some examples:
arabica ə - ɹ ă b ĭ - k ə
observer ɒ b -
fornicatory - t o ʊ ɹ i
I would propose that we should remove all these punctuations and can do so on a language-independent basis.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.