prelurn
Prelurn is a package and CLI for exploring tabular data.
Prelurn describes CSV datasets, augmenting the describe function of pandas.
For each column, it identifies:
- data type
- (numeric, boolean, timestamp, categorical)
- descriptive statistics (depending on the data type)
- mean
- standard deviation
- max
- min
- quantiles or deciles
- categories
- most frequent value and its number of occurrences
- data integrity
- propertion of observations which are missing
This enables users to do some basic determination of which variables may be useful in a machine learning model. Particularly in linear parametric models (e.g. LDA, logistic regression), which are sensitive to the distribution of features.
(venv) root@b9241715e1c2:/opt/prelurn# prelurn describe tests/data/data.csv
(venv) root@b9241715e1c2:/opt/prelurn# cat data_describe.csv
| type | categories | missing_proportion | count | unique | top | freq | mean | std | min | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | max | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UID | numeric | 0.0 | 64.0 | 257.09375 | 79.36168335988272 | 106.0 | 145.9 | 184.6 | 211.9 | 234.60000000000002 | 257.5 | 281.4 | 309.19999999999993 | 346.6 | 359.7 | 373.0 | ||||
| VAR2 | numeric | 0.0 | 64.0 | 0.201531991390625 | 0.07087738465807435 | 0.108853503 | 0.13695698950000001 | 0.15321989400000002 | 0.16363511890000002 | 0.1732312022 | 0.1840266945 | 0.1953042124 | 0.21185764439999996 | 0.2354144256 | 0.2847120188 | 0.5441622070000001 | ||||
| VAR3 | numeric | 0.0 | 64.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ||||
| VAR4 | numeric | 0.0 | 64.0 | 0.421875 | 0.4977628523130841 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | ||||
| VAR5 | numeric | 0.0 | 64.0 | 0.234375 | 0.42695628191498325 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | ||||
| VAR6 | numeric | 0.0 | 64.0 | 0.109375 | 0.3145764348029479 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.7000000000000028 | 1.0 | ||||
| VAR7 | numeric | 0.0 | 64.0 | 0.078125 | 0.27048970875004197 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | ||||
| VAR8 | numeric | 0.953125 | 3.0 | 1.0 | 1.0 | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 1.2 | 1.4 | 1.6 | 1.8 | 2.0 | ||||
| VAR9 | numeric | 0.0 | 64.0 | 0.03125 | 0.17536809360305042 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | ||||
| VAR10 | categorical | "'0','1'" | 0.0 | 64 | 2 | '0' | 61 |
Installation
Docker and (host machine) virtualenv installation are incompatible!
Virtualenv
git clone [email protected]:tesera/prelurn.git
cd prelurn
virtualenv venv
. venv/bin/activate
pip install . # regular users
Docker
Build the prelurn image as follows
docker build -t prelurn .
That creates an isolated environment where prelurn can be run
Start a container using the prelurn image
docker run -t -i prelurn /bin/bash
. venv/bin/activate
Run prelurn in the container
root@de6cccb1a217:/opt/prelurn# prelurn 10
False
False
False
False
False
False
False
False
False
False
Usage
(venv) root@8248c14c4ffc:/opt/prelurn# prelurn --help
Usage: prelurn [OPTIONS] COMMAND [ARGS]...
Machine learning preprocessing steps
Options:
-v, --verbose
-h, --help Show this message and exit.
Commands:
describe Summarize data in a csv file T
(venv) root@8248c14c4ffc:/opt/prelurn# prelurn describe --help
Usage: prelurn describe [OPTIONS] TABLE
Summarize data in a csv file
TABLE is the path to a csv file, as per http://pandas.pydata.org/pandas-
docs/stable/io.html#basic
Options:
-j, --json write output to json file
-q, --quantile-type [decile|quartile]
percentiles to use
-o, --outfile PATH path to use for output data, excluding file
extension. default is to write to working directory
-h, --help Show this message and exit.
Development
dev.env
A file, dev.env is required for some tasks
It should have the following
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
BOTO_CONFIG=
S3_TEST_DATA_URL=
BOTO_CONFIG in dev.env will override the path specified in Dockerfile.
docker-compose
Several development tasks are defined in docker-compose.yml
Note - the base directory is mounted to a volume in the container - /opt/prelurn. This way you do not need to rebuild the image every time you change a file or requirement.
Start the dev container and setup a venv; the venv will be persisted to your local storage but will probably not run on your system!
docker-compose run dev
virtualenv venv -p python2.7
. venv/bin/activate
pip install -e .[dev]
docker-compose run test # run tests
docker-compose run docs # build docs
docker-compose run dev # run bash
Tests
Any functional changes or additions should be tested. Add a test for your changes and update old tests, if required.
Run the tests as follows
docker-compose run test
# or, if set up with virtualenv outside of container
py.test tests/
Documentation
Documentation is build automatically from docstrings
See http://www.sphinx-doc.org/en/stable/rest.html for the syntax
If you change function arguments, or the docstrings are otherwise updated, you should rebuild the docs as follows:
docker-compose run docs
# or, if set up with virtualenv outside of container
cd docs
sphinx-apidoc -o ./source ../prelurn
make html
make coverage
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent