Light

deepinthebuild / dedup Goto Github PK

View Code? Open in Web Editor NEW

15.0 3.0 0.0 112 KB

A command-line tool for deduplicating entries in a file or stream.

Rust 73.28% PowerShell 1.31% Shell 3.41% Python 22.00%

rust deduplication command-line-tool text-processing deduplicator uniq

dedup's Introduction

A better deduplicator written in Rust.

Basic usage: dedup <INPUT> [-o <OUTPUTFILE>]

Run dedup --help to see:

USAGE:
    dedup.exe [FLAGS] [OPTIONS] [INPUT]

FLAGS:
    -l, --count-lines        If flag is set only print the number of unique entries found.
        --mmap               Enables use of memory mapped files. This is enabled by default.
        --no-mmap            Prohibits usage of memory mapped files. This will slow down the deduplication process
                             significantly!
    -z, --zero-terminated    Specifies that entries should be intepreted as being separated by a null byte rather than a
                             newline.
    -h, --help               Prints help information
    -V, --version            Prints version information

OPTIONS:
    -o, --output <OUTPUT>
        --terminator <TERMINATOR>    Specifies the single-byte pattern to separate entries by. Default is newline.
                                     [default: \n]

ARGS:
    <INPUT>    Specifies the input file to read from. Omit or supply '-' to read from stdin.

To run the benchmark run python benchsuite/benchrunner. This will download a large (400MB+) text file to use as a benchmark case.

Feature requests and bug reports are always welcome! Please raise them as an issue in this Github repository.

dedup's People

Contributors

Stargazers

Watchers

dedup's Issues

How much memory does it use?

I need to dedupe line-separated records in a very large file (hundreds of gigs, each record a few KB). This looks like it could be useful for me, but I don't know how much RAM it uses. For stream mode it has to store the actual lines, which is understandable, but what about mmap mode?

benchmark performance

`$ python3 benchrunner
Downloading test corpus...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 124M 100 124M 0 0 3425k 0 0:00:37 0:00:37 --:--:-- 4143k
Unpacking test corpus...
Concatenating files in to single blob...
Building dedup...
Finished release [optimized] target(s) in 0.0 secs

awk : mean: 4.988 median: 4.803

dedup : mean: 2.337 median: 2.284

sort : mean: 17.03 median: 17.32
`

benchmark fails to run

After running 'Cargo install' which places the dedup executable in ~/.cargo/bin/dedup the benchrunner fails to correctly set the path/current working directory.

FileNotFoundError: [Errno 2] No such file or directory: 'sort -u europarl.txt'
FileNotFoundError: [Errno 2] No such file or directory: 'dedup europarl.txt'

benchmark results

`python3 benchrunner

Downloading test corpus...

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 124M 100 124M 0 0 1956k 0 0:01:04 0:01:04 --:--:-- 2747k

Unpacking test corpus...

Concatenating files in to single blob...

/bin/sh: dedup: command not found

/bin/sh: dedup: command not found

/bin/sh: dedup: command not found

/bin/sh: dedup: command not found

sort | uniq: [19.610267162322998, 19.310322523117065, 19.74545121192932] mean: 19.555346965789795

dedup: [0.0023207664489746094, 0.0020852088928222656, 0.0018146038055419922] mean: 0.002073526382446289
`

warning on cargo build

warning: variant is never constructed: `Other`
 --> src/error.rs:9:5
  |
9 |     Other,
  |     ^^^^^
  |
  = note: #[warn(dead_code)] on by default

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.