jakubsochor / boxcars Goto Github PK

Source code related to BoxCars publication

Python 100.00%

boxcars's Introduction

BoxCars Fine-Grained Recognition of Vehicles

This is Keras+Tensorflow re-implementation of our method for fine-grained classification of vehicles decribed in BoxCars: Improving Vehicle Fine-Grained Recognition using 3D Bounding Boxes in Traffic Surveillance (link). The numerical results are slightly different, but similar. This code is for research only purposes. If you use the code, please cite our paper:

@ARTICLE{Sochor2018, 
author={J. Sochor and J. Špaňhel and A. Herout}, 
journal={IEEE Transactions on Intelligent Transportation Systems}, 
title={BoxCars: Improving Fine-Grained Recognition of Vehicles Using 3-D Bounding Boxes in Traffic Surveillance}, 
year={2018}, 
volume={PP}, 
number={99}, 
pages={1-12}, 
doi={10.1109/TITS.2018.2799228}, 
ISSN={1524-9050}
}

Installation

Clone the repository and cd to it.

git clone https://github.com/JakubSochor/BoxCars.git BoxCars
cd BoxCars

(Optional, but recommended) Create virtual environment for this project - you can use virtualenvwrapper or following commands. IMPORTANT NOTE: this project is using Python3.

virtuenv -p /usr/bin/python3 boxcars_venv
source boxcars_venv/bin/activate

Install required packages:

pip3 install -r requirements.txt

Manually download dataset https://medusa.fit.vutbr.cz/traffic/data/BoxCars116k.zip and unzip it.
Modify scripts/config.py and change BOXCARS_DATASET_ROOT to directory where is the unzipped dataset.
(Optional) Download trained models using scripts/download_models.py. To download all models to default location (./models) run following command (or use -h for help):

python3 scripts/download_models.py --all

Usage

Fine-tuning of the Models

To fine-tune a model use scripts/train_eval.py (use -h for help). Example for ResNet50:

python3 scripts/train_eval.py --train-net ResNet50

It is also possible to resume training using --resume argument for train_eval.py.

Evaluation

The model is evaluated when the training is finished, however it is possible to evaluate saved model by running:

python3 scripts/train_eval.py --eval path-to-model.h5

Trained models

We provide numerical results of models distributed with this code (use scripts/download_models.py). The processing time was measured on GTX1080 with CUDNN. The accuracy results are always shown as single image accuracy/whole track accuracy (in percents). We have also evaluated the method with estimated 3D bounding boxes (see paper for details) and included the results here. The estimated bounding boxes are in data/estimated_3DBB.pkl. In order to use the estimated bounding boxes, use --estimated-3DBB path-to-pkl argument for train_eval.py script. The models which were trained with the estimated bounding boxes have suffix _estimated3DBB.

Net	Original 3DBBs	Estimated 3DBBs	Image Processing Time
ResNet50	84.29/91.61	81.78/90.79	5.8ms
VGG16	84.10/92.09	81.43/90.68	5.4ms
VGG19	83.35/91.23	81.93/91.48	5.4ms
InceptionV3	81.51/89.86	79.89/89.92	6.1ms

BoxCars116k dataset

The dataset was created for the paper and it is possible to download it from our website The dataset contains 116k of images of vehicles with fine-grained labels taken from surveillance cameras under various viewpoints. See the paper BoxCars: Improving Vehicle Fine-Grained Recognition using 3D Bounding Boxes in Traffic Surveillance for more statistics and information about dataset acquisition. The dataset contains tracked vehicles with the same label and multiple images per track. The track is uniquely identified by its id vehicle_id, while each image is uniquely identified by vehicle_id and instance_id. It is possible to use class BoxCarsDataset from lib/boxcars_dataset.py for working with the dataset; however, for convenience, we describe the structure of the dataset also here. The dataset contains several files and folders:

images - dataset images and masks
atlas.pkl - BIG structure with jpeg encoded images, which can be convenient as the whole structure fits the memory and it is possible to get the images on the fly. To load the atlas (or any other pkl file), you can use function load_cache from lib/utils.py. To decode the image (in RGB channel order), use the following statement.

atlas = load_cache(path_to_atlas_file)
image = cv2.cvtColor(cv2.imdecode(atlas[vehicle_id][instance_id], 1), cv2.COLOR_BGR2RGB)

dataset.pkl - contains dictionary with following fields:

cameras: information about used cameras (vanishing points, principal point)
samples: list of vehicles (index correspons to vehicle id). 
		 The structure contains several fields which should understandable. 
		 It also contains field instances with list of of dictionaries 
		 with information about images of the vehicle track. 
		 The flag to_camera defines whether the vehicle is going towards camera or not.

classification_splits.pkl - different splits (hard, medium from paper and additional body and make split). Each split contains structure types_mapping definig mapping from textual labels to integer labels. It also contains fields train, test, and validation which are lists and each element contains tuple (vehicle_id, class_id).
verification_splits.pkl - similar to classification splits; however, the elements in train, test are triplets (vehicle_id1, vehicle_id2, class_id).
json_data and matlab_data - converted pkl file

boxcars's People

Contributors

Stargazers

Watchers

boxcars's Issues

Use trained model for single image prediction

Hi JakubSochor, thanks for interesting research and this repo. It is easily to setup and run the training. I used the model (fine-tune on ResNet50) I trainied with BoxCars116l Dataset on my computer to predict on single image:

`from keras.models import load_model
from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.applications.resnet50 import preprocess_input
from keras.applications.resnet50 import decode_predictions

path_to_model = "/home/hung/source/python/read_model/models/final_model.h5"
model= load_model(path_to_model)

image = load_img("car.png", target_size=(224, 224))
image = img_to_array(image)
image = image.reshape(1, image.shape[0], image.shape[1], image.shape[2])
image = preprocess_input(image)

predict = model.predict(image);
label = decode_predictions(predict);
print(label)`

But it got error like this ValueError: decode_predictions expects a batch of predictions (i.e. a 2D array of shape (samples, 1000)). Found array with shape: (1, 107)
I understand that because number of class is different from origin imagenet classes. Can you prove me a way to make prediction, sorry I am new to Keras and not understand your code at all.
Thank you.

list of 107 classes from model

can you give a json information for 107 output classes from model?

Bounding Box Estimator

Are you planning to provide the 3D Bounding Box estimator model? It would be interesting to try that model for training purposes and in other applications.

this command error : python3 scripts/train_eval.py --eval path-to-model.h5

Excused me,I follow your step but show the error follow as:

how can I fix this error

Dataset link is down

Hi,
the link to the dataset (https://medusa.fit.vutbr.cz/traffic/data/BoxCars116k.zip) is down. Can you put that one up again or provide a mirror? Thanks!

AttributeError: 'NoneType' object has no attribute 'TF_DeleteStatus'

Where to find the NET that estimates 3D bb ?

Hello , Thank you for this interesting research. So far I have been playing with this repository and I was able to reproduce the results in the paper.

I am more intersted in the part that you are estimating 3D bounding boxes, However in the paper and in the repository I couldnt find any clue about how to use that independently , Can you please summuarize or list some link that could help me to understand 3D bb estimation part

Thank you in advance,

Link to trained model is down

Hi, the link for downloading trained model doesn't work anymore, can you please provide link for alternative download?

Is there any full image corresponding to the patch

Thank you for your interesting and practical research.
I wonder if there is any full image corresponding to the vehicle patch, or I can transform the 2/3D coordinates to the full image, which help me validate full image do help 3D bounding box estimate. If I miss this important information, please inform me, I'll appreciate much.

Trained Model fails in evaluation and single image classification

Hi Jakub,

I tried to train new ResNet50 model, and it seemed quite good on the result.

However, when I tried to evaluate using
python3 train_eval.py --eval path-to-model.h5
the evaluation result on 'test' part is bad

Also when I tried to use the model to predict single image class, it also went bad.
Here is my code for single image prediction.(quote some lines from your work)
Where did I do wrong in this thanks!

from keras.models import load_model                          
from keras.preprocessing.image import load_img               
from keras.preprocessing.image import img_to_array           
from keras.applications.resnet50 import preprocess_input     
from keras.applications.resnet50 import decode_predictions   
import cv2           
import numpy as np   

path_to_model = ....
model= load_model(path_to_model)      
image = cv2.imread("027491_007.png")                          
image = cv2.resize(image,(224,224))                           
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)                
image = (image.astype(np.float32) - 116)/128.                 
x = np.empty([1] + list((224,224)) + [3], dtype=np.float32)   
x[0, ...] = image                                             
                                                              
predict = model.predict(x);                                   
pred_idx = np.argmax(predict)                                 
print(predict,pred_idx)

Is it possible to derived the orientation from existing label?

Hi, @JakubSochor,
Is it possible to derived the orientation from existing label (camera intrinsics, vanishing points and so on)?
Thank you~

BCNN, CBL and PCM are all using VGGNet as baselines, but achieve worse performance

Thank you for sharing the code. I admire your great work.

I am puzzled that BCNN, CBL and PCM are all using VGGNet as baselines, but achieve worse performance.
Could you eplain it for me, please?
Thank you in advance.

Best regards

requirements.txt file

Many versions are incompatible:
Failed to build numpy scipy
ERROR: yellowbrick 0.9.1 has requirement numpy>=1.13.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: yellowbrick 0.9.1 has requirement scipy>=1.0.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: xarray 0.15.1 has requirement numpy>=1.15, but you'll have numpy 1.12.0 which is incompatible.
ERROR: umap-learn 0.5.1 has requirement numpy>=1.17, but you'll have numpy 1.12.0 which is incompatible.
ERROR: umap-learn 0.5.1 has requirement scipy>=1.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: tifffile 2021.3.17 has requirement numpy>=1.15.1, but you'll have numpy 1.12.0 which is incompatible.
ERROR: textgenrnn 1.4.1 has requirement keras>=2.1.5, but you'll have keras 1.2.2 which is incompatible.
ERROR: tensorflow 2.4.1 has requirement h5py~=2.10.0, but you'll have h5py 2.6.0 which is incompatible.
ERROR: tensorflow 2.4.1 has requirement numpy~=1.19.2, but you'll have numpy 1.12.0 which is incompatible.
ERROR: tensorflow 2.4.1 has requirement protobuf>=3.9.2, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: tensorflow 2.4.1 has requirement six~=1.15.0, but you'll have six 1.10.0 which is incompatible.
ERROR: tensorflow-probability 0.12.1 has requirement numpy>=1.13.3, but you'll have numpy 1.12.0 which is incompatible.
ERROR: tensorflow-metadata 0.28.0 has requirement protobuf<4,>=3.7, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: tensorflow-hub 0.11.0 has requirement protobuf>=3.8.0, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: tensorflow-datasets 4.0.1 has requirement protobuf>=3.6.1, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: tensorboard 2.4.1 has requirement protobuf>=3.6.0, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: spacy 2.2.4 has requirement numpy>=1.15.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: seaborn 0.11.1 has requirement numpy>=1.15, but you'll have numpy 1.12.0 which is incompatible.
ERROR: seaborn 0.11.1 has requirement scipy>=1.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: scikit-image 0.16.2 has requirement scipy>=0.19.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: pywavelets 1.1.1 has requirement numpy>=1.13.3, but you'll have numpy 1.12.0 which is incompatible.
ERROR: pysndfile 1.3.8 has requirement numpy>=1.13.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: pynndescent 0.5.2 has requirement scipy>=1.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: pymc3 3.7 has requirement h5py>=2.7.0, but you'll have h5py 2.6.0 which is incompatible.
ERROR: pymc3 3.7 has requirement numpy>=1.13.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: pymc3 3.7 has requirement theano>=1.0.4, but you'll have theano 0.8.2 which is incompatible.
ERROR: pyerfa 1.7.2 has requirement numpy>=1.16, but you'll have numpy 1.12.0 which is incompatible.
ERROR: pyarrow 3.0.0 has requirement numpy>=1.16.6, but you'll have numpy 1.12.0 which is incompatible.
ERROR: plotnine 0.6.0 has requirement numpy>=1.16.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: plotnine 0.6.0 has requirement scipy>=1.2.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: pandas 1.1.5 has requirement numpy>=1.15.4, but you'll have numpy 1.12.0 which is incompatible.
ERROR: opencv-python 3.4.2.17 has requirement numpy>=1.14.5, but you'll have numpy 1.12.0 which is incompatible.
ERROR: opencv-contrib-python 4.1.2.30 has requirement numpy>=1.14.5, but you'll have numpy 1.12.0 which is incompatible.
ERROR: numba 0.51.2 has requirement numpy>=1.15, but you'll have numpy 1.12.0 which is incompatible.
ERROR: librosa 0.8.0 has requirement numpy>=1.15.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: librosa 0.8.0 has requirement scipy>=1.0.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: kapre 0.1.3.1 has requirement keras>=2.0.0, but you'll have keras 1.2.2 which is incompatible.
ERROR: jaxlib 0.1.62+cuda110 has requirement numpy>=1.16, but you'll have numpy 1.12.0 which is incompatible.
ERROR: imgaug 0.2.9 has requirement numpy>=1.15.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: googleapis-common-protos 1.53.0 has requirement protobuf>=3.12.0, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: google-colab 1.0.0 has requirement six~=1.15.0, but you'll have six 1.10.0 which is incompatible.
ERROR: google-cloud-bigquery 1.21.0 has requirement protobuf>=3.6.0, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: google-api-python-client 1.12.8 has requirement six<2dev,>=1.13.0, but you'll have six 1.10.0 which is incompatible.
ERROR: google-api-core 1.26.1 has requirement protobuf>=3.12.0, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: google-api-core 1.26.1 has requirement six>=1.13.0, but you'll have six 1.10.0 which is incompatible.
ERROR: fbprophet 0.7.1 has requirement numpy>=1.15.4, but you'll have numpy 1.12.0 which is incompatible.
ERROR: fastai 1.0.61 has requirement numpy>=1.15, but you'll have numpy 1.12.0 which is incompatible.
ERROR: fancyimpute 0.4.3 has requirement keras>=2.0.0, but you'll have keras 1.2.2 which is incompatible.
ERROR: dm-tree 0.1.5 has requirement six>=1.12.0, but you'll have six 1.10.0 which is incompatible.
ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
ERROR: cvxpy 1.0.31 has requirement numpy>=1.15, but you'll have numpy 1.12.0 which is incompatible.
ERROR: cvxpy 1.0.31 has requirement scipy>=1.1.0, but you'll have scipy 0.18.1 which is incompatible.
ERROR: blis 0.4.1 has requirement numpy>=1.15.0, but you'll have numpy 1.12.0 which is incompatible.
ERROR: astropy 4.2 has requirement numpy>=1.17, but you'll have numpy 1.12.0 which is incompatible.
ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
ERROR: tensorflow-gpu 2.4.1 has requirement h5py~=2.10.0, but you'll have h5py 2.6.0 which is incompatible.
ERROR: tensorflow-gpu 2.4.1 has requirement numpy~=1.19.2, but you'll have numpy 1.12.0 which is incompatible.
ERROR: tensorflow-gpu 2.4.1 has requirement protobuf>=3.9.2, but you'll have protobuf 3.2.0 which is incompatible.
ERROR: tensorflow-gpu 2.4.1 has requirement six~=1.15.0, but you'll have six 1.10.0 which is incompatible.

No such file :dataset.pkl

Link to dataset is down

I get an error on two different internet connections while trying to access this link to download the dataset. Is this a temporary issue?

The coordinates of vanishing point

In the dataset.pkl, every camera has three VPs' 2d coordinates, and I wonder whether those coordinates are the VPs' position in the image plane?
To validate this , I use the equation(5) :U= (ux, uy, f), V= (ux, uy, f),P= (px, py, 0),W= (U− P) × (V− P) in the article 'Fully Automatic Roadside Camera Calibration for Traffic Surveillance'.
The VPs I use are the VPs of camera videnska, I use vp1=[ 2018.41228263, 77.64316627], vp2=[ 63.71104018, 64.09228882], pp=[ 350.5, 250.5] and f=667.9198918 to compute vp3, and the result is array([-716.16197463, 48.24208282, 667.9198918 ]), it is very different to the position [ 334.37621358, 2675.97844949] given in dtaset.pkl, and it seems that the VPs' coordinates given in the dataset.pkl are not the VPs' positions in the image plane, so I want to ask what's the coordiante system of VPs given in dataset.pkl

About how to estimate the 3DBB

I am confused about how to estimate the 3D bounding box with the contour of the vehicles and the direction of the VP. I also wonder the input when using CNN for estimation of directions towards vanishing points, the whole image with several car or the cropped image with only a car?

How to show the detected image ?

hello, I was followed the README.md, and got the final_model.h5, but if I want to detect one image using the final_model.h5, how to show the results of the detection, it's like the 3D bounding box on the image(Fig.5 in your Paper). Thank you.

How to create the pickle files for my own dataset?

Hi @JakubSochor, thanks for your research on 3D bounding box estimation.
I have a problem and it's the same as the topic.
I'd like to create those pickle files for my own dataset, but I have no idea what is the format of those pickle files.
Can you give me some tips for doing this work, especially for dataset.pkl and atlas.pkl?

Thanks in advance

Project license?

Which license does this project have?

cheers,
Peter

Help in understanding dataset.pkl

Hello,

I'm trying to get the direction of VP's so that I can make a 3DBB. So, as mentioned in the paper's fig 7, I made the model to predict the direction of vanishing points by predicting a set of 3 angles.

{'098': {'pp': array([ 427.5,  240.5]),
   'vp1': array([ 948.281 ,   92.7385]),
   'vp3': array([  455.78606968,  5066.38864576]),
   'focal': 835.6732001667876,
   'vp2': array([-952.23 ,  103.878])}

This is an example of a camera's properties taken from dataset.pkl. As mentioned in the paper, I expected a set of 3 angles to be given in the dataset each corresponding to a VP. But it seems a set of numbers are given corresponding to each VP. I'm confused here.

What are those numbers?
How do I proceed from here to construct the 3DBB?

Can someone help me out in understanding this?

Thanks in advance!

how Accuracy is measured?

In this system, How Accuracy is measured? Please explain me

Confusion of contour detection

According to the paper Object Contour Detection with a Fully Convolutional Encoder-Decoder Network, I trained a network based on dataset PASCAL VOC. But unfortunately, the network always output black image.
Can you share me the code about contour detection?

And is your traffic research website down? I can't open it.

Any help will be appreciated!

what's the coordinate of the image

I see that in your article, you set the principle point be the middle of the image, and in the dataset.pkl, every camera's PP cordiante is[ 427.5, 240.5], so it confuse me with the coordinate of the image。

TypeError: new() got an unexpected keyword argument 'file'

when I tried to run command "python3 train_eval.py --train-net ResNet50 " ,,,,the following error occured -TypeError: new() got an unexpected keyword argument 'file' .
why ?

This is just a car patch classification example

I think this repo has nothing to do with 3D box prediction but just a simple and naive classification of car image patchs?

It's make no sense to do such a job and claim that can do 3D box predictions......

lots of wrong labeled images in the training boxcars116k dataset

Till now, I find following images are wrong-labeled. Would you please tell me how to automatically filter out such sort of outliers in the dataset.

wrong_labeled_image_ids = [520, 687, 8698, 1410, 4734, 5667, 33492]

` # Load image
wrong_labeled_image_id = wrong_labeled_image_ids [0]
vehicle_id, instance_id = dataset.X[wrong_labeled_image_id ]
class_ids = [dataset.Y[image_id]]
class_ids = np.array(class_ids, dtype=np.int32)

# Get corresponding data
vehicle, instance, bb3d, bb2d = dataset.get_vehicle_instance_data(vehicle_id, instance_id)
image = dataset.get_image(vehicle_id, instance_id)`

when we map the bb2d and bb3d onto the original image, we can get following image

Some more wrong-labeled images:
Image ID: 4734

Image ID: 1410

Training on Custom data

Hi @JakubSochor thanks for sharing the code and wonderful work , below i have mentioned few queries

1.How to generate the 3d bounding box for custom data, is source code available
2. Can i train on custom dataset like bikes ? is so should i createdataset similar to boxcars116k
3.Can we use this method to train two objects like car and bike in the same image
4, What is the inference time in CPU and GPU ?