adbar Goto Github PK

adbar

archiveis

A simple Python wrapper for the archive.is capturing service

awesome-crawler

A collection of awesome web crawler,spider in different languages

awesome-web-scraper

A collection of awesome web scaper, crawler.

btw21

Visualization of the most frequent words in the German federal election in 2021

cchardet

universal character encoding detector

coronakorpus

Material zum Aufbau eines deutschsprachigen COVID-19-Webkorpus / Building a corpus in German dedicated to coronavirus

corpus-visualizer

Explore, visualize and publish corpora as CSS/XHTML documents

courlan

Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

dateparser

python parser for human readable dates

dwdsmor

SFST/SMOR/DWDS-based German Morphology

equipe-crawler

Automatically exported from code.google.com/p/equipe-crawler

flux-toolchain

Filtering and Language-identification for URL Crawling Seeds (FLUCS) a.k.a. FLUX-Toolchain

geokelone

integrates spatial and textual data processing tools into a modular software package which features preprocessing, geocoding, disambiguation and visualization

german-nlp

Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

german-reddit

Extraction of a German Reddit Corpus

gps-corpus-builder

Automatically exported from code.google.com/p/gps-corpus-builder

htmldate

Fast and robust date extraction from web pages, with Python or on the command-line

jlcl-style

Experiments to modernize the LaTeX class of the JLCL

jparser

A readability parser which can extract title, content, images from html pages

justext

Heuristic based boilerplate removal tool

laclos

LAnguage-CLassified OpenSubtitles

microblog-explorer

Perform crawls of social networks (identi.ca, reddit, friendfeed) to gather internal and external links and identify their language

py3langid

Faster, modernized fork of the language identification tool langid.py

python-readability

fast python port of arc90's readability tool, updated to match latest readability.js!

shoten

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

toponyms

Old prototype for toponym extraction in historical texts written in German

trafilatura

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments