The site ''thanlwintimes.com'' is one of the sites used by Corpus Crawler. However, the site no longer seems to be available; when I visit, I see an error saying that the domain expired:

http://thanlwintimes.com/

The site should be replaced with another site using the same encoding.

We've been referred to the following sources for corpora in additional Myanmar-script laguages.

1. Pali (Tri Pitaka) [pi-Mymr]
   1. https://tipitaka.org/mymr/
2. Mon [mnw]
   1. http://mon.monnews.org/
   2. https://mnw.wikipedia.org/wiki/မုက်လိက်တမ်
3. Shan [shn] -- already included
   1. https://shannews.org/
   2. https://shn.wikipedia.org/wiki/ၼႃႈႁူဝ်ႁႅၵ်ႈ
4. Karen [kar]
   1. http://karen.kicnews.org/
   2. https://wol.jw.org/ksw/wol/h/r350/lp-kr (Bible, Publications...)

CC @sven-oly

This site is in be-tarask: https://www.svaboda.org/

Norwegian language is currently not present.
It would be great to have it added too.

@brawer how to use this script?

Hi Sascha,

Nice work! Here's the output of what roozbeh did for HarfBuzz testing by extracting Wikipedia:
https://github.com/behdad/harfbuzz-testing-wikipedia

Don't know if it's of much use. That one included all talk pages of Wikipedia as well, so the word distribution is skewed, for example the word for "User" is over-represented. Anyway, thought I share here for the record.

The Universal Dependencies project has corpora in a set of languages; consider incorporating them.

I looked through the readme; is there a list of what sites are crawled by this script? Is there documentation for how to add additional sites?

Is there any work being done regarding any Arabic dialects?

We can start with http://www.dw.com/ar/, which is Modern Standard Arabic. I think MSA is a good start, and we can add regional dialects later.

Please list here any source you think we should add, for MSA or regional dialects.

Related to #80. Suggestion. Mainly, move the core codes up so it is more visible.
The crawlers are kept into their own folder.

• Reoganize project structure from :

corpuscrawler
├─ README.md
├─ LICENSE
├─ LICENSE.md
├─ CONTRIBUTING.md
├─ corpuscrawler
└─ Lib
   └─ corpuscrawler
      ├─ *.py : utilities
      └─ crawl_{iso}.py : crawlers

to

corpuscrawler
├─ README.md
├─ LICENSE
├─ LICENSE.md
├─ CONTRIBUTING.md
├─ corpuscrawler
├─ *.py : utilities
└─ crawlers
   └─ crawl_{iso}.py : crawlers

Would such changes disturb some complementary toolchain ?

Not sure what is going on here:

fran@ipek:~/source/corpuscrawler$ ./corpuscrawler --output ~/corpora/languages/kaqchikel/corpcrawl/ --language cak
Downloading:    http://listen.bible.is/robots.txt
Downloading:    http://listen.bible.is/CAKSBG/Matt/1
Traceback (most recent call last):
  File "./corpuscrawler", line 28, in <module>
    sys.exit(corpuscrawler.main.main())
  File "/home/fran/source/corpuscrawler/Lib/corpuscrawler/main.py", line 1249, in main
    crawls[args.language](crawler)
  File "/home/fran/source/corpuscrawler/Lib/corpuscrawler/crawl_cak.py", line 21, in crawl
    crawl_bibleis(crawler, out, bible='CAKSBG')
  File "/home/fran/source/corpuscrawler/Lib/corpuscrawler/util.py", line 718, in crawl_bibleis
    jsonraw = json.loads(content.split('var chaptersByBook = ')[1].split(';\n')[0])
IndexError: list index out of range

When attempting to run corpuscrawler in Python 3, I get the error:

ImportError: No module named 'mimetools'

It would be good to support Python 3.

When running in Python 2, I get this other error:

AttributeError: 'module' object has no attribute 'SSLContext'

which apparently happens because that function was backported very recently to Python 2.

For Scottish Gaelic, https://dasg.ac.uk/text/ now contains plaintext files which makes it easier to crawl than before. Some material is multilingual, but it’s already language-tagged with a custom tagging scheme using tags such as <eng> and <gai>. For example, https://dasg.ac.uk/text/68.txt has English sections that are marked up like this — a trivial regexp subsitution should be able to remove the English sections:

Dh’fhosgail e i; is léugh e:
<eng>The Queen, who is lying very ill, urges your immediate attendance.
(Signed) Eveleyn Marlborough.<gai>
“Ma thig am Prionnsa,” thuirt e, ...

/cc @jimregan

The crawled Maori language corpus contains some English text. By filtering out every paragraph that contains the string the, we’d remove 1902 lines from the corpus which are mostly English-language quotes. I believe this would be an improvement. @jimregan, would you be fine with this change?

Here’s a few sample paragraphs that would be removed:

Hei tā Whakaruru, “That's what the King believed his mother signed was to give us the means to help. We want to put on the table lands, resources and people.”

The doting parents of Whakatōhea descendant Kayla Imrie have won a free ticket to the Olympics.

Hei tā Kiripatea, “This is a primary model of one of the backyard gardens that we have done and we did it over a weekend.”

Thanks to Volkswagen, they will get a free ticket to watch their daughter perform with the NZ K4 Women's Kayaking crew at Rio.

The first-time Olympian is currently in Portugal training alongside her K4 team, Aimee Fisher, Caitlin Ryan and Jamie Lovett, for the upcoming world cup in Germany this month.

Ko tā Marama Davidson, mema o te Pāti Kākāriki, “We are still in decline when it comes to those who can speak te reo. So I think it's only 3.7% of people in Aotearoa can have a conversation in te reo. We know that over 77% of children are not enrolled in any subject at schools at the moment and we know that half of all schools in Aotearoa have absolutely no student learning te reo or taking a te reo subject.”

In order to crawl my-t-d0-zawgyi.txt, you have to wait for my.txt to finish. It would be nicer if the two files could be crawling in parallel. For example, one could initiate my.txt, my-t-d0-zawgyi.txt, and shn.txt crawling, and have data after just a few minutes; the data files can keep growing as more pages get crawled.

There are ready-to-download open licence Wikipedia corpora available.

This /CONTRIBUTING.md is a License Agreement / Code of Conduite to sign. As far as I can see, this very cherishable project has no actual tutorial.

In don't have Python and code knowledge to fix this documentation issue but I can map the road so it become easier for the next person to do so.

Wanted

If an user want to add a language such as Catalan from Barcelona (ca, cat : missing). What do he needs to jump in quickly ? What should he provide ?

• What isn the local structure :
  • util.py : store functions uses by multiple languages crawlers
  • main.py : stores the 1000+ crawlers calls, run them all.
  • crawl_{iso}.py : stores language-specific copora's source url and processing functions.
    • crawl_ca_valencia.py
• What tools I have :
  • lists of available modules
  • API of key functions
• What input(s) : python list of url ?
• What are the classic parts of a crawler function ?
• What output format : raw text ? html is fine because a html balise wiper is then used ?
• Example of easily hackable base-code.

API (to complete)

Defined functions within util.py, by order of apparition as of 2021/02/26. If you have some relevant knowledge, please help for a sub-section or one item.

Some tools

• daterange(start, end): __
• urlpath(url): __
• urlencode(url): __

Main element

• class Crawler(object):
  • __init__(self, language, output_dir, cache_dir, crawldelay): __
  • get_output(self, language=None): __
  • close(self): __
  • fetch(self, url, redirections=None, fetch_encoding='utf-8'): __
  • fetch_content(self, url, allow_404=False): __
  • fetch_sitemap(self, url, processed=set(), subsitemap_filter=lambda x: True): __
  • is_fetch_allowed_by_robots_txt(self, url): __
  • crawl_pngscriptures_org(self, out, language): __
  • _find_urls_on_pngscriptures_org(self, language): __
  • crawl_abc_net_au(self, out, program_id): __
  • crawl_churchio(self, out, bible_id): __
  • crawl_aps_dz(self, out, prefix): __
  • crawl_sverigesradio(self, out, program_id): __
  • crawl_voice_of_america(self, out, host, ignore_ascii=False): __
  • set_context(self, context): __

Some crawlers for multi-languages sites

• crawl_bbc_news(crawler, out, urlprefix): __
• crawl_korero_html(crawler, out, project, genre, filepath): __
• write_paragraphs(et, out): __
• crawl_deutsche_welle(crawler, out, prefix, need_percent_in_url=False): __
• crawl_radio_free_asia(crawler, out, edition, start_year=1998): __
• crawl_sputnik_news(crawler, out, host): __
• crawl_udhr(crawler, out, filename): __
• crawl_voice_of_nigeria(crawler, out, urlprefix): __
• crawl_bibleis(crawler, out, bible): __
• crawl_tipitaka(crawler, out, script): __
• find_wordpress_urls(crawler, site, **kwargs): __

Some cleaners

• unichar(i): __
• replace_html_entities(html): __
• cleantext(html): __
• clean_paragraphs(html): __
• extract(before, after, html): __
• fixquotes(s): __

Shorter way to do so

In code comments can do a lot. Pointing to wisely chosen sections too. If you have the required know how, please add comments onto a chosen, existing crawler and point to it as an in-code tutorial.

@sffc, @brawer : anyone could help on that ?

Hi,

Can we fetch data from URLs not mentioned in the existing code by adding custom functions? Also, does it not support the English language('en' not mentioned anywhere in the list of supported languages)?

Thanks

I ran ./corpuscrawler --language=my-t-d0-zawgyi --output=./corpus (with python 2.7 on Ubuntu 18.04) and the program crashed while downloading from some url. The output is shown below.

Downloading:    http://thanlwintimes.com/robots.txt
Downloading:    http://thanlwintimes.com/
Downloading:    http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8/
Cache-Hit:      http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8/
Downloading:    http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8/page/2/
Downloading:    http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8/page/3/
Downloading:    http://thanlwintimes.com/category/%e1%80%b1%e1%80%86%e1%80%ac%e1%80%84%e1%80%b9%e1%80%b8%e1%80%95%e1%80%ab%e1%80%b8/
Cache-Hit:      http://thanlwintimes.com/category/%e1%80%b1%e1%80%86%e1%80%ac%e1%80%84%e1%80%b9%e1%80%b8%e1%80%95%e1%80%ab%e1%80%b8/
Downloading:    http://thanlwintimes.com/category/%e1%80%b1%e1%80%86%e1%80%ac%e1%80%84%e1%80%b9%e1%80%b8%e1%80%95%e1%80%ab%e1%80%b8/page/2/
Downloading:    http://thanlwintimes.com/category/%e1%80%b1%e1%80%86%e1%80%ac%e1%80%84%e1%80%b9%e1%80%b8%e1%80%95%e1%80%ab%e1%80%b8/page/3/
Downloading:    http://thanlwintimes.com/category/%e1%80%b1%e1%80%86%e1%80%ac%e1%80%84%e1%80%b9%e1%80%b8%e1%80%95%e1%80%ab%e1%80%b8/page/4/
Downloading:    http://thanlwintimes.com/category/%e1%80%b1%e1%80%86%e1%80%ac%e1%80%84%e1%80%b9%e1%80%b8%e1%80%95%e1%80%ab%e1%80%b8/page/5/
Downloading:    http://thanlwintimes.com/category/%e1%80%b1%e1%80%86%e1%80%ac%e1%80%84%e1%80%b9%e1%80%b8%e1%80%95%e1%80%ab%e1%80%b8/page/6/
Downloading:    http://thanlwintimes.com/category/%e1%80%a1%e1%80%84%e1%80%b9%e1%80%90%e1%80%ac%e1%80%97%e1%80%ba%e1%80%b4%e1%80%b8/
Cache-Hit:      http://thanlwintimes.com/category/%e1%80%a1%e1%80%84%e1%80%b9%e1%80%90%e1%80%ac%e1%80%97%e1%80%ba%e1%80%b4%e1%80%b8/
Downloading:    http://thanlwintimes.com/category/%e1%80%a1%e1%80%84%e1%80%b9%e1%80%90%e1%80%ac%e1%80%97%e1%80%ba%e1%80%b4%e1%80%b8/page/2/
Downloading:    http://thanlwintimes.com/category/%e1%80%a1%e1%80%84%e1%80%b9%e1%80%90%e1%80%ac%e1%80%97%e1%80%ba%e1%80%b4%e1%80%b8/page/3/
Downloading:    http://thanlwintimes.com/category/%e1%80%a1%e1%80%84%e1%80%b9%e1%80%90%e1%80%ac%e1%80%97%e1%80%ba%e1%80%b4%e1%80%b8/page/4/
Downloading:    http://thanlwintimes.com/category/%e1%80%a1%e1%80%9a%e1%80%b9%e1%80%92%e1%80%ae%e1%80%90%e1%80%ac%e1%80%b7-%e1%80%a1%e1%80%ac%e1%80%b1%e1%80%98%e1%80%ac%e1%80%b9/
Cache-Hit:      http://thanlwintimes.com/category/%e1%80%a1%e1%80%9a%e1%80%b9%e1%80%92%e1%80%ae%e1%80%90%e1%80%ac%e1%80%b7-%e1%80%a1%e1%80%ac%e1%80%b1%e1%80%98%e1%80%ac%e1%80%b9/
Downloading:    http://thanlwintimes.com/category/%e1%80%a1%e1%80%9a%e1%80%b9%e1%80%92%e1%80%ae%e1%80%90%e1%80%ac%e1%80%b7-%e1%80%a1%e1%80%ac%e1%80%b1%e1%80%98%e1%80%ac%e1%80%b9/page/2/
Downloading:    http://thanlwintimes.com/category/%e1%80%80%e1%80%ac%e1%80%90%e1%80%bc%e1%80%94%e1%80%b9%e1%80%b8/
Cache-Hit:      http://thanlwintimes.com/category/%e1%80%80%e1%80%ac%e1%80%90%e1%80%bc%e1%80%94%e1%80%b9%e1%80%b8/
Downloading:    http://thanlwintimes.com/category/%e1%80%80%e1%80%ac%e1%80%90%e1%80%bc%e1%80%94%e1%80%b9%e1%80%b8/page/2/
Downloading:    http://thanlwintimes.com/category/%e1%80%80%e1%80%ac%e1%80%90%e1%80%bc%e1%80%94%e1%80%b9%e1%80%b8/page/3/
Downloading:    http://thanlwintimes.com/category/%e1%80%80%e1%80%ac%e1%80%90%e1%80%bc%e1%80%94%e1%80%b9%e1%80%b8/page/4/
Downloading:    http://thanlwintimes.com/category/%e1%80%80%e1%80%ac%e1%80%90%e1%80%bc%e1%80%94%e1%80%b9%e1%80%b8/page/5/
Downloading:    http://thanlwintimes.com/category/%e1%80%80%e1%80%ac%e1%80%90%e1%80%bc%e1%80%94%e1%80%b9%e1%80%b8/page/6/
Downloading:    http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8%e1%80%93%e1%80%ab%e1%80%90%e1%80%b9%e1%80%95%e1%80%af%e1%80%b6/
Cache-Hit:      http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8%e1%80%93%e1%80%ab%e1%80%90%e1%80%b9%e1%80%95%e1%80%af%e1%80%b6/
Downloading:    http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8%e1%80%93%e1%80%ab%e1%80%90%e1%80%b9%e1%80%95%e1%80%af%e1%80%b6/page/2/
Traceback (most recent call last):
  File "./corpuscrawler", line 28, in <module>
    sys.exit(corpuscrawler.main.main())
  File "/home/dell/nlp/corpuscrawler/Lib/corpuscrawler/main.py", line 1249, in main
    crawls[args.language](crawler)
  File "/home/dell/nlp/corpuscrawler/Lib/corpuscrawler/crawl_my_t_d0_zawgyi.py", line 24, in crawl
    _crawl_than_lwin_times(crawler, out)
  File "/home/dell/nlp/corpuscrawler/Lib/corpuscrawler/crawl_my_t_d0_zawgyi.py", line 28, in _crawl_than_lwin_times
    urls = find_wordpress_urls(crawler, 'http://thanlwintimes.com/')
  File "/home/dell/nlp/corpuscrawler/Lib/corpuscrawler/util.py", line 767, in find_wordpress_urls
    assert pgdoc.status == 200, (pgdoc.status, pgurl)
AssertionError: (404, u'http://thanlwintimes.com/category/%e1%80%9e%e1%80%90%e1%80%84%e1%80%b9%e1%80%b8%e1%80%93%e1%80%ab%e1%80%90%e1%80%b9%e1%80%95%e1%80%af%e1%80%b6/page/2/')

It would be nice to add crawlers for https://www.tipitaka.org to Corpus Crawler. This is the Buddhist Tipitaka in the Pali language, written in various scripts. The crawled corpora will be useful for testing Pali transliteration rules, contributed to Unicode CLDR by @mjansche.

CLDR is proposing the script code Qaag to use for Zawgyi text:

Qaag is a special script code for identifying the non-standard use of Myanmar characters for display with the Zawgyi font. The purpose of the code is to enable migration to standard, interoperable use of Unicode by providing an identifier for Zawgyi for tagging text, applications, input methods, font tables, transformations, and other mechanisms used for migration.

corpuscrawler should be updated to use the new script code instead of the -u-s0-zawgyi workaround. Myanmar Tools will need to also be updated to consume the new script code.

A quick search shows you that CorpusCrawler does not crawl or use Wikipedia. I don't know Python but it seems feasible, either from scratch on Wikipedia API (1) or using existing server-side tools (2).

Assess interest

1. Assess how many Wikipedia languages are not in UNILEX. See unicode-org/unilex#14 .
2. Assess quality of wikipedia raw text data in minority languages.
3. Compare gain to other available public corpora such Tatoeba (358 languages).

Crawling via API

By using and loading available list of articles per wikipedia, then scrap the sites. If too large, could be limited to max=n articles.

Given an iso code such as Ndonga's ng :

• download List of page titles in main namespace archive (see below)
• get the articles into a python list variable (python)
• code a crawler in /Lib/corpuscrawler/util.py, following other crawler as examples 1, which query Wikipedia API, extract the valuable text, save the text. (python)
• Update relevant crawlers /Lib/corpuscrawler/

Wikipedia API provides text

• en + wikipedia + Dragon + xmlfm
• ca + wikipedia + Drac + jsonfm

Various formats available:

• format : The format of the output.
  • jsont : Output data in JSON format.
  • jsonfmt : Output data in JSON format (pretty-print in HTML).
  • nonet : Output nothing.
  • phpt : Output data in serialised PHP format.
  • phpfmt : Output data in serialised PHP format (pretty-print in HTML).
  • rawfmt : Output data, including debugging elements, in JSON format (pretty-print in HTML).
  • xmlt : Output data in XML format.
  • xmlfmt : Output data in XML format (pretty-print in HTML).

List of Wikipedia (~300)

• List_of_Wikipedias
• List of dumps - Wikipedia and others wiki projects.

List of articles per Wikipedia

For convenience, I use the tiny Ndonga (ng) Wikipedia (8 articles), easier to explore by hand.

• List of all page titles
• List of page titles in main namespace

For larger demo, you could also inspect similar URLs with the iso of :

Namespaces

On all wikis. See also here

• 0: (main)
• 1: Talk:
• 2: User:
• 3: User_talk:

Dumps' & paths

• List of dumps
  • /ngwiki/20200220 - manual (change the date)
  • /ngwiki/latest - directory
    • /ngwiki-latest-all-titles.gz
    • /ngwiki-latest-all-titles-in-ns0.gz - articles only

Using Wikipedia extractors ?

• https://github.com/attardi/wikiextractor#installation
• https://github.com/hugolpz/induction-wikipedia-corpus / https://www.kdnuggets.com/2017/11/building-wikipedia-text-corpus-nlp.html
• https://github.com/telecoms-intelligence/induction-wikipedia-corpus

Hybrid approach

• ISO: get the list of all local wiki's iso codes.
• Downloads: loop over each language code, download the dump.
• Extract: use extractor above, zip each language
• Cloud: put text result online.
• Crawl: in util.py, code a simple crawler which get just that .zip, convert back to txt content, add to the corpora.

cc: @brawer

I was analyzing the exit file and I realized the text for each "news" is only the title, the headline, and the 1st paragraph. It must be correct?
I'm using the crawler for "pt" language.

% flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

./corpuscrawler/Lib/corpuscrawler/crawl_mi.py:62:39: F821 undefined name 'sitemap'
        if pubdate is None: pubdate = sitemap[url]
                                      ^
./corpuscrawler/Lib/corpuscrawler/crawl_kab.py:53:48: F821 undefined name 'url'
        assert doc.status == 200, (doc.status, url)
                                               ^
./corpuscrawler/Lib/corpuscrawler/crawl_tpi.py:48:48: F821 undefined name 'url'
        assert doc.status == 200, (doc.status, url)
                                               ^
./corpuscrawler/Lib/corpuscrawler/crawl_shn.py:90:30: F821 undefined name 'striptags'
                p = ' '.join(striptags(replace_html_entities(p)).split())
                             ^
./corpuscrawler/Lib/corpuscrawler/crawl_shn.py:90:40: F821 undefined name 'replace_html_entities'
                p = ' '.join(striptags(replace_html_entities(p)).split())
                                       ^
./corpuscrawler/Lib/corpuscrawler/crawl_ga.py:147:39: F821 undefined name 'fetchresult'
        if pubdate is None: pubdate = fetchresult.headers.get('Last-Modified')
                                      ^
./corpuscrawler/Lib/corpuscrawler/crawl_th.py:25:5: F821 undefined name 'crawl_bibleis'
    crawl_bibleis(crawler, out, bible='THATSV')
    ^
./corpuscrawler/Lib/corpuscrawler/crawl_vec.py:43:48: F821 undefined name 'start_url'
        assert doc.status == 200, (doc.status, start_url)
                                               ^
8     F821 undefined name 'fetchresult'
8

https://flake8.pycqa.org/en/latest/user/error-codes.html

On the flake8 test selection, this PR does not focus on "style violations" (the majority of flake8 error codes that psf/black can autocorrect). Instead, these tests are focus on runtime safety and correctness:

• E9 tests are about Python syntax errors usually raised because flake8 can not build an Abstract Syntax Tree (AST). Often these issues are a sign of unused code or code that has not been ported to Python 3. These would be compile-time errors in a compiled language but in a dynamic language like Python, they result in the script halting/crashing on the user.
• F63 tests are usually about the confusion between identity and equality in Python. Use ==/!= to compare str, bytes, and int literals is the classic case. These are areas where a == b is True but a is b is False (or vice versa). Python >= 3.8 will raise SyntaxWarnings on these instances.
• F7 tests logic errors and syntax errors in type hints
• F82 tests are almost always undefined names which are usually a sign of a typo, missing imports, or code that has not been ported to Python 3. These also would be compile-time errors in a compiled language but in Python, a NameError is raised which will halt/crash the script on the user.

I am trying to use this crawler to build an Urdu corpus. I am running Ubuntu 18.04 inside a VMWare virtual machine. The crawler will start and successfully download a few links but will eventually get permanently hung up. Nothing happens until I ctrl-c to exit the script. I can kill the script, start it again and it will successfully get the link it got hung up on the previous run, it will then successfully crawl a few more until getting hung up again. The below copied text is an example of what I get when I kill the script with ctrl-c

...(the crawler has successfully downloaded several links so far)...
Downloading: https://www.bbc.com/urdu/entertainment-37527961
Downloading: https://www.bbc.com/urdu/entertainment-37529481
Downloading: https://www.bbc.com/urdu/entertainment-37531642
Downloading: https://www.bbc.com/urdu/entertainment-37532975
^CTraceback (most recent call last):
File "./corpuscrawler", line 28, in
sys.exit(corpuscrawler.main.main())
File "/home/thebucketmouse/Desktop/corpuscrawler-master/Lib/corpuscrawler/main.py", line 1249, in main
crawlsargs.language
File "/home/thebucketmouse/Desktop/corpuscrawler-master/Lib/corpuscrawler/crawl_ur.py", line 22, in crawl
crawl_bbc_news(crawler, out, urlprefix='/urdu/')
File "/home/thebucketmouse/Desktop/corpuscrawler-master/Lib/corpuscrawler/util.py", line 475, in crawl_bbc_news
fetchresult = crawler.fetch(url)
File "/home/thebucketmouse/Desktop/corpuscrawler-master/Lib/corpuscrawler/util.py", line 150, in fetch
content = response.read()
File "/usr/lib/python2.7/socket.py", line 355, in read
data = self._sock.recv(rbufsize)
File "/usr/lib/python2.7/httplib.py", line 597, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 384, in read
data = self._sock.recv(left)
File "/usr/lib/python2.7/ssl.py", line 772, in recv
return self.read(buflen)
File "/usr/lib/python2.7/ssl.py", line 659, in read
v = self._sslobj.read(len)
KeyboardInterrupt

The script doesn't run with Python 3.

Shows error :

For solving this I have tried changing this:

to: checker.parse(robots_txt) as it is already decoded in Python 3 and it worked for me.

After this I am getting error:

• By running main.py in am getting error:

ModuleNotFoundError: No module named 'corpuscrawler'

Can anybody help solving this?

Tiny issue : add a reference to the language code system you use.

I may have missed it.

See also : https://en.wikipedia.org/wiki/Language_code

The language which we currently call taq is actually Kabyle (BCP47: kab). Our use of taq came from the website taq.tamurt.info, but their taq stands for the word “taqbaylit,” which means Kabyle in Berber.

Research

• J. Tiedemann, 2016, Finding Alternative Translations in a Large Corpus of Movie Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016)

Gain

Closest of natural oral corpora.

Links

• Portal
  • bre.txt.gz -- Bretonl corpus.
  • 60+ languages available.
  • List: af,ar,bg,bn,br,bs,ca,cs,da,de,el,en,eo,es,et,eu,fa,fi,fr,gl,he,hi,hr,hu,hy,id,is,it,ja,ka,kk,ko,lt,lv,mk,ml,ms,nl,no,pl,pt,pt_br,ro,ru,si,sk,sl,sq,sr,sv,ta,te,th,tl,tr,uk,ur,vi,ze_en,ze_zh,zh_cn,zh_tw

There are ready-to-download open licence Wikipedia corpora available.

Related to #80. This is the core documentation which can help open source contributions.