What is a corpus? A corpus is a large collection of texts. It is a body of written or spoken material, such as articles, books, conversations, etc. The plural form of corpus is corpora.

Why would you need a corpus? Instead of teaching a computer how to spell or the rules of grammar for a particular language, you'd use a large collection of texts (a corpus), to test against, in order to determine whether a sample of text is properly spelled, has valid grammar, to interpolate missing words, etc.

By testing against existing texts, a computer can determine how a word SHOULD be spelled as well as where and how a word SHOULD be used.

The way you would actually go about teaching a computer to process text is the basis of Natural Language Processing/Understanding and is not covered here. This is a simple utility, to filter and clean up large collections of texts, for later processing.

You can find a few sample texts in the folder samples, they were taken from textfiles.com.

original

APPLE II HISTORY
===== == =======

Compiled and written by Steven Weyhrich
(C) Copyright 1991, Zonker Software

 (PART 6 -- THE APPLE II PLUS)
[v1.1 :: 12 Dec 91]

processed

APPLE II HISTORY.

Compiled and written by Steven Weyhrich
Copyright 1991, Zonker Software.

PART 6 - THE APPLE II PLUS
v1.1 12 Dec 91.

original

This should be
a single sentence!

processed Sentence Stitching

This should be a single sentence!

There is only a single required argument you must provide, that is the input filename or directory. If a folder is specified it will be recursively searched for files to process.

The output argument should be a filename to output the resulting corpus after processing all input files. If no output is specified, results will be sent to the console (stdout).

var Corpora = require('./lib/corpora.js')
var corpora = new Corpora()

corpora.input = process.argv[2]
corpora.output = process.argv[3]
corpora.process(function(){
	console.log('FINITO')
})

By default a series of processing steps are done. They are defined in default_steps.js. You can specify your own steps:

NOTE Steps are fed directly to String.prototype.replace as arguments.

var corpora = new Corpora()

// Step - uppercase
corpora.steps.push(
	[
		/[a-z]+/gi,
		function UPPERCASE(string)
		{
			return string.toUpperCase()
		}
	]
)

// Replace Special Characters with NULL character
corpora.steps.push(
	[
		/[@#$%]+/g,
		0x00
	]
)

// Replace a specific string
corpora.steps.push(
	[
		'John Smith',
		'Adam Smith'
	]
)