chenyanglei / automatic-video-colorization Goto Github PK

Hi, Mr.Chenyang and Mr.Qifeng.
I have a question about in your paper "Fully-Automatic-Video-Colorization-with-Self-Regularization-and-Diversity".

In your paper, you used φ in your proposed loss function for diversity.
I wonder what this φ represents.

I checked your cited paper "Interactive_Image_Segmentation_with_Latent_Diversity", but I can't find φ.
Could you tell me, what do φ represents?

Thank you.

I treid to run your code on your example dataset but it shows me nothing coud please help me to run this

I'm working on CUDA10 and how can I load the op library in flow_warp.py?
An error occurred like this

I cannot change the version of CUDA right now,and is there any solution to fix it?

Hello, I've tried to train this whole model, but I found that the training speed is very slow because the KNN calculating. Is there any other choice? Thank you!

Trying to evaluate your code.
Unfortunately the only result I've got is:

result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence

I tried:
python test.py --img_path demo_imgs/ILSVRC2012_val_00040251.JPEG
python test.py --img_path demo_imgs/
Result the same.
It only creates "teat_result" folder with empty " model_imagecolor" subfolder.
Greatly appreciate for your assistance.

hello, i can't connect https://docs.google.com/uc?export=download. Is it bad requests? error 400. Could you solve it? Thank you

Hello， your work is so interesting. I wanna know, how can i reach the eval work as you do. whats the data for eval? And how to do for eval video data？

In implementation of this code:

vgg_rawnet=scipy.io.loadmat('VGG_Model/imagenet-vgg-verydeep-19.mat')

I am getting the following error:

File "/home/akshat_suwalka/Fully-Automatic-Video-Colorization-with-Self-Regularization-and-Diversity-master/utils.py", line 273, in
vgg_rawnet=scipy.io.loadmat('VGG_Model/imagenet-vgg-verydeep-19.mat')

File "/home/akshat_suwalka/anaconda3/lib/python3.7/site-packages/scipy/io/matlab/mio.py", line 218, in loadmat
matfile_dict = MR.get_variables(variable_names)

File "/home/akshat_suwalka/anaconda3/lib/python3.7/site-packages/scipy/io/matlab/mio5.py", line 292, in get_variables
res = self.read_var_array(hdr, process)

File "/home/akshat_suwalka/anaconda3/lib/python3.7/site-packages/scipy/io/matlab/mio5.py", line 252, in read_var_array
return self._matrix_reader.array_from_header(header, process)

File "mio5_utils.pyx", line 673, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header

File "mio5_utils.pyx", line 719, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header

File "mio5_utils.pyx", line 893, in scipy.io.matlab.mio5_utils.VarReader5.read_cells

File "mio5_utils.pyx", line 671, in scipy.io.matlab.mio5_utils.VarReader5.read_mi_matrix

File "mio5_utils.pyx", line 721, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header

File "mio5_utils.pyx", line 968, in scipy.io.matlab.mio5_utils.VarReader5.read_struct

File "mio5_utils.pyx", line 671, in scipy.io.matlab.mio5_utils.VarReader5.read_mi_matrix

File "mio5_utils.pyx", line 719, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header

File "mio5_utils.pyx", line 893, in scipy.io.matlab.mio5_utils.VarReader5.read_cells

File "mio5_utils.pyx", line 671, in scipy.io.matlab.mio5_utils.VarReader5.read_mi_matrix

File "mio5_utils.pyx", line 703, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header

File "mio5_utils.pyx", line 777, in scipy.io.matlab.mio5_utils.VarReader5.read_real_complex

File "mio5_utils.pyx", line 450, in scipy.io.matlab.mio5_utils.VarReader5.read_numeric

File "mio5_utils.pyx", line 355, in scipy.io.matlab.mio5_utils.VarReader5.read_element

File "streams.pyx", line 200, in scipy.io.matlab.streams.ZlibInputStream.read_string

File "streams.pyx", line 193, in scipy.io.matlab.streams.ZlibInputStream.read_into

OSError: could not read bytes

Please help me in it.

And if you have implemented in tensorflow 2.0 , then can you please share it??

Hello, Sorry to bother you, but I really can't download the weight files "VGG_Model/ imagenet-vGG -verydeep-19.mat" and "CKpt_woflow. zip". Can you send it to my mailbox if convenient? My email address is [email protected]
thank you very much.

Hello， your work is so interesting. I wanna know, if I wanna use your mehtod to compare with my method. But my training dataset and test data are different from yours, should I use my dataset to train your model again?

Thanks for your nice work!

Could you please provide the pretrained model of 'hyper_unet' ?

It seems like flow_warp.so was not compiled with tensorflow_framework.so dynamic lib. Is there any solution to this error?

Download.py is not downloaded
Even the link provided by you i.e.ckpt is also not working please help
Please tell the steps to download.

Hi!
I wanna use our code for my course project but it is not clear how data should be arrange. Is directory JPEGImages supposed to have gray frames? If so than where should i put colored (true) frames? Also it is not clear how to "generate the optical flow by PWC-Net".

Could you please clarify that?

When I run the code for that example data you provided in test_sample0 I can't

Namespace(model='ckpt_woflow', test_dir='test_sample0', test_img='', use_gpu=1)
'grep' is not recognized as an internal or external command,
operable program or batch file.
Traceback (most recent call last):
File "main_woflow.py", line 30, in
os.environ["CUDA_VISIBLE_DEVICES"]=str(np.argmax( [int(x.split()[2]) for x in subprocess.Popen("nvidia-smi -q -d Memory | grep -A4 GPU | grep Free", shell=True, stdout=subprocess.PIPE).stdout.readlines()]))
File "<array_function internals>", line 6, in argmax
File "E:\mi\lib\site-packages\numpy\core\fromnumeric.py", line 1153, in argmax
return _wrapfunc(a, 'argmax', axis=axis, out=out)
File "E:\mi\lib\site-packages\numpy\core\fromnumeric.py", line 58, in _wrapfunc
return _wrapit(obj, method, *args, **kwds)
File "E:\mi\lib\site-packages\numpy\core\fromnumeric.py", line 47, in _wrapit
result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence

(base) E:\animegan\video attack\Fully-Automatic-Video-Colorization-with-Self-Regularization-and-Diversity-master>python main_woflow.py --model ckpt_woflow --use_gpu 1 --test_dir test_sample0

how to fix it

I try to download the VGG_19 model by run
python download_VGG.py
And the connection error happened.

I can browse Google and Youtobe. Is the download link invalid? Or may I have some wrong operation?
look forward to your reply
good wish!

Hello，thanks for your project.
I meet an error in download_models.py,line34
download_file_from_google_drive('0B_B_FOgPxgFLRjdEdE9NNTlzUWc', 'VGG_Model/imagenet-vgg-verydeep-19.mat')

FileNotFoundError: [Errno 2] No such file or directory: 'VGG_Model/imagenet-vgg-verydeep-19.mat'

I changed VGG_Model/imagenet-vgg-verydeep-19.mat with imagenet-vgg-verydeep-19.mat and download sucessful, then move it in VGG_Model. So I think if you can justify your code in download_models.py.

Hi i want to colorize my black and white video i downloaded the model and ckpt_woflow i want to ask how to use this thing. Thanks

Is it possible to have a script for the refinement network so that I can run the colourisation output through this multiple times as mentioned in your paper to gain better temporal coherence?

I tried to run your model on video followed your instruction. However, it shows the error:

OSError: cannot identify image file 'test_video/girl_video.mp4'

I have tried with other videos, but still not run. Do I need to convert it to a specific video format?

Thank you so much.

Hello, sorry to bother you! I've read your paper and source code recently. You mentioned that you train the colorization model firstly, and then the colorization model and refinement model are trained jointly. But the main_whole.py file may only include the joint training code?
If I want to reproduce the result, do I need to add a new file to train the colorization network firstly?
And in the main_whole.py file, the pre-trained model of colorization network may not be loaded. After the colorization network is trained, should I load the pre-trained model of colorization network when training the whole network?
Looking forward to your apply! Thank you!

Hi,
When I looking at the code, I notice that the bilateral loss seem to be annotated. Does it take part in the training?
Thanks!

Traceback (most recent call last):
File "main_whole.py", line 11, in
import flow_warp as flow_warp_op
File "/home/qianming/code/Fully-Automatic-Video-Colorization/flow_warp.py", line 4, in
tf.resource_loader.get_path_to_datafile("./lib/flow_warp.so"))
File "/home/qianming/anaconda3/envs/torch/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: libcudart.so.8.0: cannot open shared object file: No such file or directory

Successfully run single image colorization:

python test.py --img_path demo_imgs/ILSVRC2012_val_00040251.JPEG

however attempt to colorize the images in a folder by:

python test.py --img_path demo_imgs
throws an error:
IsADirectoryError: [Errno 21] Is a directory: 'demo_imgs'

The same error happened when running:

python test_div_video.py --use_gpu 1 --video_path demo_vid

When I run this code;

!python main_woflow.py --model ckpt_woflow --use_gpu 1 --test_img /content/frames

I'm getting this error;

File "main_woflow.py", line 129, in
im=np.float32(scipy.misc.imread(test_img, 'L'))/255.0
AttributeError: module 'scipy.misc' has no attribute 'imread'

Thank you for sharing your good code:)
I have a question about function named "Bilateral_NN".

1. What does the "division 500" mean for both color_image_X and color_image_X ??
2. Do the bold 5s as below mean (relate) K=5??

def Bilateral_NN(color_image, neigh):
h,w = color_image.shape[:2]
color_image_X = np.tile(np.arange(w),[h,1])/500.
color_image_Y = np.tile(np.arange(h),[w,1]).T/500.
color_image_all = np.concatenate([color_image,color_image_X[:,:,np.newaxis], color_image_Y[:,:,np.newaxis]],axis=2)
neigh.fit(np.reshape(color_image_all, [-1, 5]))
idxs = neigh.kneighbors(np.reshape(color_image_all, [-1, 5]), 5, return_distance=False)
return idxs

Thank you