Git Product home page Git Product logo

cvpr20_imcl's Introduction

Interactive Multi-Label CNN Learning with Partial Labels

Overview

This repository contains the implementation of Interactive Multi-Label CNN Learning with Partial Labels.

In this work, we address efficient end-to-end learning a multi-label CNN classifier with partial labels using an interactive dependency learning scheme.

Image

Prerequisites

  • Python 3.x
  • Tensorflow 1.x.x
  • sklearn
  • matplotlib
  • skimage
  • scipy

Data Preparation

Open Images

  1. Please download pretrained Open Images model(https://storage.googleapis.com/openimages/2017_07/oidv2-resnet_v1_101.ckpt.tar.gz) into './model/resnet' folder

  2. Please download Open Images urls and annotation into ./data/OpenImages folder according to the instructions within the folder ./data/OpenImages/2017_11.

  3. To crawl images from the web, please run the script:

python ./download_imgs/asyn_image_downloader.py 					#`data_set` == `train`: download images into `./image_data/OpenImages/train/`
python ./download_imgs/asyn_image_downloader.py 					#`data_set` == `validation`: download images into `./image_data/OpenImages/validation/`
python ./download_imgs/asyn_image_downloader.py 					#`data_set` == `test`: download images into `./image_data/OpenImages/test/`

Please change the data_set variable in the script to train, validation, and test to download different data splits.

  1. To extract features into TensorFlow storage format, please run:
python ./extract_data/extract_feature_2_TFRecords_OpenImages.py						#`data_set` == `train`: create `./TFRecords/train_feature.tfrecords`
python ./extract_data/extract_feature_2_TFRecords_OpenImages.py						#`data_set` == `validation`: create `./TFRecords/validation_feature.tfrecords`
python ./extract_data/extract_feature_2_TFRecords_OpenImages.py			        		#`data_set` == `test`:  create `./TFRecords/test_feature.tfrecords`

Please change the data_set variable in the extract_feature_2_TFRecords_OpenImages.py script to train, and validation to extract features from different data splits.

  1. Please download:

CUB

  1. Please download the following data files (https://drive.google.com/file/d/1gBQ_PQ0U8kzCaiiF7CvG92f1Ssfk8Zgq/view?usp=sharing), (https://drive.google.com/file/d/1fiNtiBj3hCj75eLHN1-02yWSDZrJ7GeN/view?usp=sharing), (https://drive.google.com/file/d/1O-0HTTFE9QpdTSQ8fdg31PsPAzJEiJga/view?usp=sharing) into ./TFRecord/ folder

MSCOCO

  1. Please download MSCOCO images and annotation into ./image_data/MSCOCO folder according to the instructions within the folders./image_data/MSCOCO/train2014,./image_data/MSCOCO/val2014,./image_data/MSCOCO/annotation, and ./data/MSCOCO_1k.

  2. To extract features into TensorFlow storage format, please run:

python ./extract_data/extract_train_img_2_TFRecords_MSCOCO.py						#create ./TFRecord/train_MSCOCO_img_ZLIB.tfrecords
python ./extract_data/extract_test_img_2_TFRecords_MSCOCO.py						#create ./TFRecord/test_MSCOCO_img_ZLIB.tfrecords
python ./extract_data/extract_dic_img_2_TFRecords_MSCOCO.py							#create ./TFRecord/dic_10_MSCOCO_img_ZLIB.tfrecords

Training and Evaluation

Open Images

  1. To pretrain the logistic backbone network, please run the script:
python ./OpenImages_experiments/baseline_logistic_OpenImages.py					# fixed feature representation
python ./OpenImages_experiments/e2e_baseline_logistic_OpenImages.py				# end-to-end training
  1. To train our method, please run the script:
python ./OpenImages_experiments/interactive_learning_OpenImages.py				# fixed feature representation
python ./OpenImages_experiments/e2e_interactive_learning_OpenImages.py			# end-to-end training
  1. To evaluate the performance, please run the script:
python ./evaluation/evaluation_interactive_learning_OpenImages.py				# fixed feature representation
python ./evaluation/evaluation_e2e_interactive_learning_OpenImages.py			# end-to-end training

CUB

  1. Please download the ImageNet ResNet backbone (http://download.tensorflow.org/models/resnet_v2_101_2017_04_14.tar.gz) into ./model/resnet_CUB

  2. To pretrain the logistic backbone network, please run the script:

python ./CUB_experiments/e2e_baseline_logistic_CUB.py
  1. To train and evaluate our method, please run the script:
python ./CUB_experiments/e2e_interactive_learning_CUB.py

MSCOCO

  1. Please download the ImageNet VGG backbone (http://download.tensorflow.org/models/vgg_16_2016_08_28.tar.gz) into ./model/vgg_ImageNet

  2. To pretrain the logistic backbone network, please run the script:

python ./MSCOCO_experiments/e2e_baseline_logistic_MSCOCO_vgg.py
  1. To train and evaluate our method, please run the script:
python ./MSCOCO_experiments/e2e_interactive_learning_MSCOCO.py

Pretrained Models

As the pretrained models are implemented with an older tensorflow version, it might not work with the current code-base.

Citation

If this code is helpful for your research, we would appreciate if you cite the work:

@article{Huynh-mll:CVPR20,
  author = {D.~Huynh and E.~Elhamifar},
  title = {Interactive Multi-Label {CNN} Learning with Partial Labels},
  journal = {{IEEE} Conference on Computer Vision and Pattern Recognition},
  year = {2020}}

cvpr20_imcl's People

Contributors

dathuynh avatar hbdat avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar

Forkers

sorrowyn zhanghaoyue cool-lzp

cvpr20_imcl's Issues

extract_data reshape ValueError

Bug when running python3 extract_data/extract_feature_2_TFRecords_OpenImages.py:

features_v.shape: (32, 1, 1, 2048)
features.shape: (?, 1, 1, 2048)
batch no. 0
Traceback (most recent call last):
  File "extract_data/extract_feature_2_TFRecords_OpenImages.py", line 181, in <module>
    feature = np.reshape(feature, [49, 2048])
  File "<__array_function__ internals>", line 6, in reshape
  File "/usr/local/lib64/python3.6/site-packages/numpy/core/fromnumeric.py", line 299, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/usr/local/lib64/python3.6/site-packages/numpy/core/fromnumeric.py", line 58, in _wrapfunc
    return bound(*args, **kwds)
ValueError: cannot reshape array of size 2048 into shape (49,2048)

How to solve this problem? Thank you!

The loss will be NAN while using the CUB scripts.

I downloaded the CUB data and models.
In the code, it uses resnet_v1, however, you provided resnet_v2 in the README file. That's not the big problem, when I run the code and I figure out that the loss always turn to nan. When I run it step by step, the "validate_Prediction_val" was nan in the line 510 .
Could you help me for this. Many thanks.

How to download OpenImages v3 (2017.11) training set fastly and completely?

Downloading the OpenImages v3 (2017.11) training set costs too much time and many images are unavailable, so how to download OpenImages v3 (2017.11) training set fastly and completely? Could you please provide a download link for your experimenting dataset since it seems that the codes doesn't take the full OpenImages dataset for training? Thank you!

Reported bugs always occur when downloading with the script (python download_imgs/asyn_image_downloader.py):

<class 'OSError'>
<class 'OSError'>
drop image
<class 'OSError'>
Traceback (most recent call last):
  File "download_imgs/asyn_image_downloader.py", line 160, in <module>
    res=parallel_process(image_infos, download,n_jobs)
  File "download_imgs/asyn_image_downloader.py", line 53, in parallel_process
    futures = [pool.submit(function, a) for a in array[front_num:]]
  File "download_imgs/asyn_image_downloader.py", line 53, in <listcomp>
    futures = [pool.submit(function, a) for a in array[front_num:]]
  File "/usr/lib64/python3.6/concurrent/futures/process.py", line 452, in submit
    raise BrokenProcessPool('A child process terminated '
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

missing files for training

These following necessary files are unavailable:

label_graph/graph_label_naive.npy
result/baseline_logistic_OpenImages.npz

Please provide these required files so as to start training. Thank you!

Reported bugs when running python OpenImages_experiments/baseline_logistic_OpenImages.py:

Traceback (most recent call last):
  File "OpenImages_experiments/baseline_logistic_OpenImages.py", line 173, in <module>
    G = np.load('./label_graph/graph_label_naive.npy').astype(np.float32)
  File "/usr/local/lib64/python3.6/site-packages/numpy/lib/npyio.py", line 416, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: './label_graph/graph_label_naive.npy'

Reported bugs when running python OpenImages_experiments/interactive_learning_OpenImages.py:

Traceback (most recent call last):
  File "OpenImages_experiments/interactive_learning_OpenImages.py", line 151, in <module>
    data = np.load(global_setting_OpenImage.saturated_Thetas_model)
  File "/usr/local/lib64/python3.6/site-packages/numpy/lib/npyio.py", line 416, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: './result/baseline_logistic_OpenImages.npz'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.