garysieling / deeplens-experiments Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 1.0 82 KB

AWS DeepLens Experiments

Shell 21.87% Python 77.89% Batchfile 0.24%

deeplens-experiments's People

Contributors

Stargazers

Watchers

Forkers

rmbryan71

deeplens-experiments's Issues

Script to watch for overfitting

What is the affect of the "neighbors" parameters on the performance of k-means?

Build yolo docker image #11 failed

Build 'Build yolo docker image' is failing!

Last 50 lines of build output:

[...truncated 649 B...]
 > git fetch --tags --progress https://github.com/garysieling/deeplens-experiments.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 279e0cc23994c840b1f1485ca23364b75b4db3a3 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 279e0cc23994c840b1f1485ca23364b75b4db3a3
Commit message: "Update run_jenkins.sh"
 > git rev-list --no-walk 279e0cc23994c840b1f1485ca23364b75b4db3a3 # timeout=10
[Build yolo docker image] $ /bin/sh -xe /tmp/jenkins2042701121386287469.sh
+ cd yolo
+ docker build -t yolo .
Sending build context to Docker daemon  2.048kB
Step 1/9 : FROM ubuntu:16.04
 ---> 0b1edfbffd27
Step 2/9 : RUN mkdir /darknet
 ---> Using cache
 ---> 5047b1384d0a
Step 3/9 : WORKDIR /darknet
 ---> Using cache
 ---> eeb2d354fc12
Step 4/9 : RUN 	apt-get update && apt-get install -y 	autoconf         automake 	libtool 	build-essential 	git
 ---> Using cache
 ---> c0db58d9d814
Step 5/9 : RUN 	apt-get install -y 	wget
 ---> Using cache
 ---> b3ec15074aa4
Step 6/9 : RUN 	git clone https://github.com/pjreddie/darknet && 	cd darknet && 	make
 ---> Using cache
 ---> 37acc951ce55
Step 7/9 : RUN 	wget -q https://pjreddie.com/media/files/yolov3.weights
 ---> Using cache
 ---> d7bce45b94f4
Step 8/9 : RUN 	cd darknet/ && 	./darknet
 ---> Using cache
 ---> 61e7b74cab49
Step 9/9 : CMD ["bash"]
 ---> Using cache
 ---> 35c2d7918d58
Successfully built 35c2d7918d58
Successfully tagged yolo:latest
+ docker tag yolo garysieling/yolo:latest
+ docker push
"docker push" requires exactly 1 argument.
See 'docker push --help'.

Usage:  docker push [OPTIONS] NAME[:TAG] [flags]

Push an image or a repository to a registry
Build step 'Execute shell' marked build as failure

Changes since last successful build:
No changes

View full output

Number of issues processed should be saved off for reporting

This is different from species, because some species have a lot more images than others

Does k-means give the same accuracy, if you run it several times in a row? -- Data Request

Gary - I have the data from this experiment, but I don't have a way to tell one run from another, except by guessing on label and duration fields.
Is there a way I could get a unique incrementing number per "run" or, I guess, even the run start time would be enough?

Script to output test performance

Build an ELK stack, expose to accept incoming data

Experiments need to track "power" of the result

From https://irb.research.chop.edu/writing-protocol

Sample Size and Power: All studies require a justification for the chosen sample size. Often the sample size is based on a formal power calculation. When this is the case, the planned sample size should be large enough to have a high probability (power) of detecting a true effect of a given magnitude, should it exist. When a power calculation is performed the protocol should include the following:

the primary endpoint (outcome);
the values assumed for the outcome in each study group (eg, proportion with event, or mean and standard deviation);
the planned statistical test;
alpha (type 1 error) level;
power (usually at least 80%); and
the calculated sample size per group - both assuming no loss of data and, if relevant, after any inflation for anticipated missing data modified from SPIRIT 2012 explanation and elaboration: guidance for protocols of clinical trials

Set up a file format for an "experiment" that the protocol for the experiment

E.g.:
on the input side, combinations of hyperparameters
how many images used, sizes, etc
max time alloted?

Script to list test images

If a container runs on AWS, add a dollar cost estimate to the data saved in elasticsearch

Script to create docker image of built model

Change docker image running to execute the containers on AWS instead of a local jenkins

Extract Hyperparameters to Splunk Fields

Models need to be saved off when experiment completes

Set up a file format to determine "success" for an experiment

Check experiments into git

Screen to track model performance over time

Write some sort of automated tests that can verify experiments are written correctly

Script to list training images

How does image size correlate to accuracy with k-means?

Build yolo docker image #9 failed

Build 'Build yolo docker image' is failing!

Last 50 lines of build output:

Started by user admin
Building in workspace /var/jenkins_home/workspace/Build yolo docker image
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/garysieling/deeplens-experiments.git # timeout=10
Fetching upstream changes from https://github.com/garysieling/deeplens-experiments.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/garysieling/deeplens-experiments.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 279e0cc23994c840b1f1485ca23364b75b4db3a3 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 279e0cc23994c840b1f1485ca23364b75b4db3a3
Commit message: "Update run_jenkins.sh"
 > git rev-list --no-walk 279e0cc23994c840b1f1485ca23364b75b4db3a3 # timeout=10
[Build yolo docker image] $ /bin/sh -xe /tmp/jenkins1416403038488051095.sh
+ cd yolo
+ docker build --name yolo .
unknown flag: --name
See 'docker build --help'.
Build step 'Execute shell' marked build as failure

Changes since last successful build:
No changes

[noreply] 279e0cc - Update run_jenkins.sh

No changes
No changes
No changes
No changes
No changes
No changes

View full output

Display training progress in jenkins

Write a script to capture experiment results, i.e. data + screenshots of charts

Take a screenshot of chart from a completed experiment, upload screenshot to github

What I would do is the following:

Manually make a chart that is filtered to a specific experiment, plots n_neighbors vs average accuracy, with one chart per species.
Get the URL from #1 and put it into a script so it can be parameterized, i.e. on n_neighbors, and the Kibana URL
Take a screenshot of this in code (it might make sense to use Selenium for this.)
Upload the screenshot to a github ticket
I would make the Elasticsearch URL come from environment variables, and the chart configuration come from a JSON file.

The future goal would be to make a series of JSON files that are "standard" reports for specific types of experiment, i.e. so they don't have to be manually built in the UI.