rmdlo / cocopen-opencv Goto Github PK

Because the edge pixels of every object may have been lighter than the color threshold (some mix of the color of the object and the color of the background - black), the edge pixels were not filtered out. Every generated object image in the dataset includes black outlines around the object. We could use alpha matting or a similar technique to remove this outline - similar to this post and this post.

Device images show holes in the ethernet ports since these ports are shadowed and appear black. We should fill these holes as part of the auto-labeling process.

This link details how to do this:
https://learnopencv.com/filling-holes-in-an-image-using-opencv-python-c/

Image dimensions are currently hard-coded throughout cocopen.py (height = 1080, width = 1920). Need to replace these with a class variable self.height and self.width. It would be helpful to have a function that checks to make sure all images used to generate the dataset are the same dimensions before beginning the image combination - should throw an exception if image dimensions are not consistent. This function could be included in a dataset preprocessing script (i.e., not in coconut.py).

Note: This affects the scale_image, random_operations, and color_augmentation functions in coconut.py where image dimensions are hard-coded

Train/val split is currently hard-coded as 0.8 in two locations. It would be better to let the user define their train/val split (some users may want a 0.9/0.1 split, for example). Assign this to a class variable and/or read it in from run.py.

if 0 <= rand < 0.8:

• Collect images, upload to "device" folder on azure
• Rename "single-wire" folder on azure to "wire"
• Read object category from its storage folder/container on azure
• Tune device segmentation (color threshold values)

Purpose: generate the method can scale to N object categories by demonstrating it works for two.

Competition website

We are currently building on a downloaded model configuration in the src/train.py script rather than loading it in from a training configuration .yaml file which we set up. It would be better for usability and python syntax to load the config from a .yaml file and perform cfg.merge_from_file(), deleting all the other cfg.X calls in src/train.py.

• Generate a dataset of 10,000 training images
• Train detectron2 on generated dataset
• Start with getting the pipeline in place for single wire images; after pipeline is debugged, add device data to entire process.
• Push existing detectron2 repository to github and configure it for COCOpen repository

This is not related to structure of code, but is related to user-friendliness: it would be helpful to aggregate total computation time of each function to see which portions of dataset creation take the most time and to target dataset generation optimization.

We have a dependency on pillow, hsv_img = Image.fromarray(rgb_img). It would be better to remove pillow dependencies and only use opencv dependencies. If we use opencv for this, we need to make sure to conda uninstall pillow and upload new environment.yml files.

Add a dictionary key to the label .json file which, for each generated image, indicates which raw images were combined to produce the image (for example, train/1.png comprises single_wire/1099.png, single_wire/2351.png, and single_wire/4312.png)

This information will be added with a (key, value) pair to the coco_new_obj_seg_sem and coco_new_obj_sem dictionaries. Value = a list of image names, e.g., [“wire/2351.png”, “wire/2362.png”...]
***Note: add the folder the image was pulled from (which should correspond to the object’s category ID) to directory structure of this list
***Will need to set up reading the category name for the object from the folder name where the image is stored (so need to rename folder of “single_wire” images to “wire” images)

• The initial copy should be really short and punchy to grab people’s attention before they lose interest and move on. You should be able to explain most of what it does using example images, which I think will get the point across really rapidly.
• We need getting started and installation instructions sections (for example, see the detectron2 README.md)
• We have nonstandard stuff in repository; to extent possible, organize the README.md in the same manner as some of the other open source libraries that we respect and are widely used
• You should describe what environment the installation instructions are expecting to run in.
• The install instructions should finish with “how to run it” instructions that are a concrete example of actually running it with a data set that’s available to the novice user, so everything works end-to-end out of the box and they can try tweaking it from there. It looks like your current run.py script needs to be run with your personal Azure environment configured so it can access the images, so that won’t work for an arbitrary user (right?). Is there some way you can expose your data to them using a read-only globally accessible URL so it will work?
• "Running API" - not immediately clear that we provide instructions on how to use the library
• Need to provide assumptions that are made about the set of images, maybe other stuff that has to be provided
• Provide the smallest, simplest version of that stuff included also made publicly available (as part of repo), so someone could clone repo and follow instructions to get it to generate a labeled dataset from small, simple example. this would be a very small but complete example of all stuff that must be input to library that allows user to get started with generating the set of images (the hello-world run-it instructions should probably do something quick and make it easy for users to inspect the results, not regenerate the whole UIUC Wires data set). It would be a weird limiter to make this really depend on Azure in any way - maybe include a small set of simple example images in the repository.
• Not getting much from the figure at the moment
• Some part of the repository should include instructions on how to collect your own input images and create your own data set.
• Getting started+Run instructions should correspond to diagram in README.md. The proposed solution describes what happens when run the API, so it would be nice if all of these were combined
• Don't set apart the note in the anaconda environment. Either: (a) assume people know how to create a conda environment in README.md (minimize the stuff that you say) and simplify installation instructions, OR (b) Provide very detailed step-by-step instructions (e.g., https://tbretl.github.io/ae353-sp22/setup). Providing too many instructions is probably overkill for our repo.

The 'wire' dataset was pruned one last time. We need to upload the final version of this dataset to Azure. The final wire dataset is here.

Note: The submission is the COCOpen-OpenCV github repository. The code submission does not include training detectron2. The code submission should include a demos/visualize.py file which visualizes a subset of the generated dataset so that users can see for themselves the annotations produced by COCOpen. This file will be based on the Dataset_Verification.ipynb, but it will generate the mask visualization images in a python script and save images to a folder (make sure to add the folder to the .gitignore!). Will also need to update the environment.yml file to install detectron2 dependencies.

The demo includes training detectron2, generating a model, and qualitatively (and quantitatively?) demonstrating the performance of the model. All code for this demo is not part of the COCOpen-OpenCV repository.

The demo video should be ~5 minutes in duration and must:

1. Explain the algorithm. The explanation will include how it works, the sequential flow of data (steps of the process), approximate run time of each step, and memory requirements (time and space complexity).
2. Demonstrate the impact of the algorithm. We could include a demonstration of wire segmentation from a point cloud in real time, and a reach goal could include some type of real-time perception+wire manipulation with the robot.

In the combine function, we have several augmentation methods all in one function. It would be helpful to set the boolean for these as a class variable (e.g., self.standard_scale_jittering=False) rather than hard-coded within the function (lines 391-401 in commit 42782ee

Explain functionality, variables, and I/O associated with modules, classes, and functions in cocopen.py.

       self.class_dict = {"device": 1,
                        "ethernet": 2,
                        "phone": 3,
                        "power": 4,
                        "hdmi": 5,
                        "coaxial": 6}
        self.categories = [{"supercategory": "device", "id": self.class_dict["device"], "name": "device"},
                                {"supercategory": "cable", "id": self.class_dict["ethernet"], "name": "ethernet"},
                                {"supercategory": "cable", "id": self.class_dict["phone"], "name": "phone"},
                                {"supercategory": "cable", "id": self.class_dict["power"], "name": "power"},
                                {"supercategory": "cable", "id": self.class_dict["hdmi"], "name": "hdmi"},
                                {"supercategory": "cable", "id": self.class_dict["coaxial"], "name": "coaxial"}]

These categories correspond to data on labelbox. For our current submission, we should only deal with single wire images. We should rename the folder on azure from "single-wire" to "wire" and assign the folder name to the category id. A reach improvement would be to collect images of single devices and set up auto-labeling for them so that we can demonstrate the functionality of this method on more than one object type.

I have recently noticed that hue jittering can mess up the color:

I remember encountering similar issue in February and fixing it. We should fix this before the final submission.