armandpl / wandb-jetracer Goto Github PK

Training a racecar with W&B!

Python 53.43% Jupyter Notebook 46.57%

jetson-nano machine-learning object-detection weights-and-biases jetracer nvidia self-driving-car

wandb-jetracer's Introduction

🏁🏎️💨 = W&B ➕ NVIDIA Jetracer

A picture of the car along with it's POV

This project builds on top of the NVIDIA Jetracer project to instrument it with Weights&Biases, making it easier to train, evaluate and refine models.
It features a full pipeline to effortlessly collect_data, label it, train/optimize models and finally drive the car while monitoring how the model is doing.

Weights&Biases Artifacts' graph showing the whole pipeline.

Pipeline overview

The repo is meant to be used by running (and modifying!) scripts under src/wandb_jetracer/scripts. These allow to train a model to detect the center of the racetrack which we can then feed to a control policy to drive the car.

collect_data.py will take pictures using the car's camera and upload them to Weights&Biases. It should be ran while manually driving the car around.
label.py is a labelling utiliy. It will download the images from the previous step to a computer to annotate them with the relevant labels. The labels will then be added to the dataset stored on Weights&Biases servers.
wandb_jetracer_training.ipynb is used to download the same dataset, train a model and upload it's weights to WandB.
trt_optim is meant to be ran on the car. It will convert the latest trained model to TensorRT for inference.
drive.py will take the optimized model and use it to drive the car. It will also log sensor data (IMU, Camera), system metrics (jetson stats, inference time) as well as the control signal to WandB. This helps with monitoring the model's perfomances in production.

Building the car

Check out NVIDIA Jetracer.

Setup and dependencies

These scripts are ran on three different types of machine: the actual embedded jetson nano computer on the car, a machine used for labelling and a colab instance used for training.
They all relie on different dependencies:

Training dependencies are installed in the Colab notebook so you don't need to worry about those.
Labelling dependencies can be installed in a conda env using conda create -f labelling_env.yml
Dependencies for the car are slightly trickier to get right, you'll find instructions here. Feel free to open issues if you run into troubles!

Disclaimer about default throttle values:

Even though default throttle values are set in the scripts under /src/scripts I would recommend testing those while the car is on a stand and it's wheels are not touching the ground. Depending on how your ESC was calibrated a throttle value of 0.0002 might mean going full reverse and your car might fly off into a wall.

Testing

After installing the labelling dependencies run pytest

Footnote

Feel free to open GitHub issues if you have any questions!

Ressources

wandb-jetracer's People

Contributors

Stargazers

Watchers

wandb-jetracer's Issues

Really good candidate for another arg from CLI

https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L35-L37

Decompose this block into constituent functions

wandb logging is its own function (maybe?)
image logging is also a function (https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L68-L74)

https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L46-L83

Add in a flag to set whether artifacts are used

Rather than commenting/uncommenting. Add to argparser.

Move the drive function to the top

And consider renaming from inference.py to drive.py

https://github.com/Armandpl/wandb_jetracer/blob/d49b32498423e037fff05eea3f0ad6b93732278f/inference.py#L82-L128

Make use of the logging lib

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/collect_data.py#L40

Integrate kineto

https://github.com/pytorch/kineto

Encapsulate argparse in function

Encapsulate these lines in a new function that returns the parsed args.

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/collect_data.py#L10-L26

Rewrite this as image -> proposed throttle + steering

Also, consider how you might generalize the setup so that a pure RL approach could be used -- x and y.

Idea: OpenAI gym?

https://github.com/Armandpl/wandb_jetracer/blob/d49b32498423e037fff05eea3f0ad6b93732278f/inference.py#L99-L100

Create and call a main function

Structure the code like:

if __name__ == "__main__":
    args = parse_args()
    main(args)

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/collect_data.py#L9

Make improvements to argparser

use f-strings for the help strings and put in the default values e.g.

default_debug_time = 120
parser.add_argument(...
    help=f"how long should it run for in seconds. Default is {default_debug_time}.")

use action=store_true for boolean flags: https://docs.python.org/3/library/argparse.html#action. Fixes bug with --debug boolean handling.
add help strings for all arguments. Use consistent formatting.
run python --help and look at what gets printed, make changes as needed. Ordering/aesthetics/clarity.

Add requirements.txt for labelling env/rc car/training env

Find a way to set debug level 'locally'

I'm using logging.info/debug/error to log informations. When setting the debug level it sets it not only for my messages but for the all the libraries.
This is not what I meant to do. Maybe instanciate a logger object and setup the debug level for this object? But then I don't wan't to pass an object around?

https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/utils/utils.py#L4

Log the dataset as an artifact table for exploration

-along with each item log x, y coordinates, mean, std, entropy, what else ?

https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/EDA.ipynb

Encapsulate in function

https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/trt_optim.py#L27-L41

Add in use of logging library in Python

from stdlib: https://docs.python.org/3/library/logging.html

Set to debug level/set verbosity of logging (separate from wandb logging).

Make this easier to read

https://github.com/Armandpl/wandb_jetracer/blob/1d7dbc11ae9dd3ab3cacbb800069c1fa6733fb2f/inference.py#L80-L115

Add an option to re-label a dataset

display the old labels
change how to count the number of (re)labeled image
https://github.com/Armandpl/wandb_jetracer/blob/c08bf31f8e18971fb4a9158b2efd40d45d2e57a9/label.py#L119

Evaluate the optimized model after converting it to TensorRT

https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/trt_optim.py#L44

Move this block to the end and call a main() function

and define main() earlier on.

example:

def main(*args, **kwargs):
   foo = setup(arg1, kwarg=kwarg3)
   bar = process(foo, kwarg=kwarg1)
   do_thing(bar, kwarg=kwarg2)

...

if __name__ == "__main__":
    args = do_arg_parse_stuff()  # WARNING: this part here is very pseudocode
    main(args, kwargs)

https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L12

Add the possibility to delete an image from the dataset

Listen for a specific key and if pressed delete the image

Think of a way to five feeback to the user that the images is indeed deleted (maybe in the cli?)
maybe ask for confirmation?

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/label.py#L73

Encapsulate this into a function

A function called parse_args or a function called make_argparser and then using .parse_args in the if name == main.

Add more data augmentation

https://github.com/Armandpl/wandb_jetracer/blob/1d7dbc11ae9dd3ab3cacbb800069c1fa6733fb2f/utils/xy_dataset.py#L17

Name this boolean (and maybe others)

Booleans often correspond to propositions/logical statements, so rather than directly evaluating them in an if, name the Boolean after its logical statement (is_done, is_divisible_5, is_downloaded, directory_is_present and pass to the if.

https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L76

Log the learning rate

https://github.com/Armandpl/wandb_jetracer/blob/2655ca21b7017a7d1011cfe988c9c457a0d880e8/wandb_jetracer_training.ipynb

Add setup instructions

Typo in gyroscope metric name

Only in x direction:

Encapsulate this in a setup function

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/collect_data.py#L38-L45

Download current version of the dataset before adding to it

In the current code we write images to an output directory and then add the directory to a wandb artifact.

The goal of this script is to incrementally increase the size of the dataset, thus we need to add to it each time it is run, not create a new one including only the new images. It must include the old images AND the new images.
To do so, first download the latest dataset version (in the setup function?) and set that as the directory in which to write new images.
Once the new images are written, add this dir as a wandb artifact.

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/collect_data.py#L37

Log accelerometer data

https://github.com/Armandpl/wandb_jetracer/blob/b3502a88ddcbdaa5bb8b9787ecfab61d0e94a593/inference.py#L63

Make the labeling tool full screen

https://github.com/Armandpl/wandb_jetracer/blob/d49b32498423e037fff05eea3f0ad6b93732278f/label.py#L45

Add flake8 and/or black to your Python dev env

Linting is super crucial to good team Python coding practice. Interacts with version control. flake8 inside your editor gives you hints as you're working (on saves, tells you where there's errors/warnings). black will auto-resolve these errors. can add it to pre-commit hooks if you want (for git).

Investigate interaction with local artifacts

If I create an artifact, and log it to wandb I end up with an artifact on wandb servers and on my local machine.
If I then delete this artifact from wandb server (but not my local machine), then instanciate a new artifact with the same name, add files to it and log it to wandb it will contain the content from the version that was deleted on wandb servers but not locally.
This a behavior that might be confusing for users. When I delete the artifact from wandb servers I feel like it's gone forever.

https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/collect_data.py#L68
https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/label.py#L113

Improve the argument parsing

provide help string for each arg
include default value in the help string

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/collect_data.py#L10-L26

Rewrite inference with OpenAI gym

If this is a good idea, use hooks/callbacks to do logging, and

watch out for state that needs to be passed around
package logging up very tightly -- separate e.g. jetson logging from image logging from accelerometer logging (as separate callbacks?) -- rather than combining things that are coincidentally logged at the same time/with the same hook

should set the artifact id from the CLI

https://github.com/Armandpl/wandb_jetracer/blob/d49b32498423e037fff05eea3f0ad6b93732278f/inference.py#L28

Move this block below main()

It seems clearer to me to read main first

https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/collect_data.py#L24-L47

Make sure copy() is necessary?

https://github.com/Armandpl/wandb_jetracer/blob/d49b32498423e037fff05eea3f0ad6b93732278f/inference.py#L96

Add option to log inference/frames as a video

https://github.com/Armandpl/wandb_jetracer/blob/c08bf31f8e18971fb4a9158b2efd40d45d2e57a9/inference.py#L86

Set some variables using tuple unpacking

Could here do:

x, y = float(output)

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/inference.py#L59-L60

There's a few other places where you similar set short variable names to short expressions -- key candidates for tuple unpacking

Display message if all images are already labeled and do not create new artifact version

At the moment if all images are already labeled the program quit and creates a new version of the artifact that is the same as the previous ones. This is confusing.

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/label.py#L59

Improve README.md

Write a short intro to the project
Add an image of the car
https://github.com/Armandpl/wandb_jetracer/blob/1d7dbc11ae9dd3ab3cacbb800069c1fa6733fb2f/README.md

Log more metrics during training

-log more metrics (absolute error for example)
-log the test set as a table with predicted label, ground truth and loss; This will allow to view the hardest examples later.

https://github.com/Armandpl/wandb_jetracer/blob/75cefd6e2a8bfe992719d51c3782802e1d299597/wandb_jetracer_training.ipynb

Rename system metrics to avoid short hand and include units

https://github.com/Armandpl/wandb_jetracer/blob/d49b32498423e037fff05eea3f0ad6b93732278f/inference.py#L76

Use an f-string here (and possible in more places)

Look up f-strings and the details of how to use them.

https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L77

Organize the imports to conform to PEP8

https://github.com/Armandpl/wandb_jetracer/blob/d99f98194ae2e0ab0d970b1337748bc2458f7a05/collect_data.py#L1-L7

Ordering of imports to conform to PEP8

3 blocks: stdlib, pip/3rd party, project-specific
alphabetization

See PEP8 spec for details.

https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L1-L10

Decide how to handle magic numbers/constants/config and implement

Option 1: make these command line arguments (adds extra complexity that might not be needed)
Option 2: set the values in code to "global" variables (with all-caps names) at top of the file. Right after imports, do STEERING_GAIN = -1 etc. etc. (very very simple, allow re-use, but complicates scope [especially bad for mutable values like lists/dicts] and is hard to change)
Option 3: use configparser to configure these values from files, add a config pointer argument to argparse (with sensible default). (adds some weight/complexity, adds dependence on an additional file, adds another library that's relatively new to python [i think?], but is most flexible/sensible and is a good skill to learn).

https://github.com/Armandpl/wandb_jetracer/blob/dc4fdbbf3a00ae7f0b773016a8c673132beb3eac/inference.py#L41-L44