iconv-chunks - Process huge files with iconv

iconv-chunks [iconv-options]

The standard iconv program reads the entire input file into memory, which doesn't work for large files (such as database exports).

This script is just a wrapper that processes the input file in manageable chunks and writes it to standard output.

The first argument is the input filename (use - to specify standard input). Anything else is passed through to iconv.

The real iconv needs to be somewhere in your PATH.

# Convert latin1 to utf-8:
./iconv-chunks database.txt -f latin1 -t utf-8 > out.txt

# Input filename of - means standard input:
./iconv-chunks - -f iso8859-1 -t utf8 < database.txt > out.txt

# More complex example, using compressed input/output to minimize disk use:
zcat database.txt.gz | ./iconv-chunks - -f iso8859-1 -t utf8 | \
gzip - > database-utf.dump.gz

Maurice Aubrey [email protected]

MIT