Currently the code in git uses pre-computed tags, use Stanford POS tagger to complete that

https://nlp.stanford.edu/software/tagger.shtml#Download

I am having a problem with the prediction I have trained the model for 100 epochs but still, I got 50% accuracy and I have also tried this model to predict on the training set but the prediction accuracy is very very poor.

Happens for different videos, but always on frame 204

when runnung extract frames .sh

Hi @rohit-gupta
Thanks for your sharing~
Is this code correspond to a paper ? If so, could you tell me the name of paper? Or it's just one of your project ?
Thank you very much !

HI Rohit, I did all steps in your instructions, and what's next to create caption for a video? thanks!

video_input : Tensor("input_1:0", shape=(?, 40, 224, 224, 3), dtype=float32)
convnet_model : <keras.engine.training.Model object at 0x7f84a0613d90>

Traceback (most recent call last):
File "batched_extractor.py", line 153, in
encoded_frame_sequence = TimeDistributed(convnet_model)(video_input)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/base_layer.py", line 460, in call
output = self.call(inputs, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/layers/wrappers.py", line 248, in call
y = self.layer.call(inputs, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/network.py", line 573, in call
output_tensors, _, _ = self.run_internal_graph(inputs, masks)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/network.py", line 730, in run_internal_graph
output_tensors = to_list(layer.call(computed_tensor, **kwargs))
File "/usr/local/lib/python2.7/dist-packages/keras/layers/normalization.py", line 195, in call
self.momentum),
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1011, in moving_average_update
x, value, momentum, zero_debias=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/moving_averages.py", line 70, in assign_moving_average
update_delta = _zero_debias(variable, value, decay)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/moving_averages.py", line 180, in _zero_debias
"biased", initializer=biased_initializer, trainable=False)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1065, in get_variable
use_resource=use_resource, custom_getter=custom_getter)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 962, in get_variable
use_resource=use_resource, custom_getter=custom_getter)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 367, in get_variable
validate_shape=validate_shape, use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 352, in _true_getter
use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 664, in _get_single_variable
name, "".join(traceback.format_list(tb))))
ValueError: Variable bn_conv1/moving_mean/biased already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:

File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1011, in moving_average_update
x, value, momentum, zero_debias=True)
File "/usr/local/lib/python2.7/dist-packages/keras/layers/normalization.py", line 195, in call
self.momentum),
File "/usr/local/lib/python2.7/dist-packages/keras/engine/base_layer.py", line 460, in call
output = self.call(inputs, **kwargs)

Hi, I got the following error. I am using Tensorflow version 1.12.

Using TensorFlow backend.
Frames will be extracted for 1968 Videos
2018-11-16 23:04:06.951863: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

Layer (type) Output Shape Param #

input_1 (InputLayer) (None, 40, 224, 224, 3) 0

time_distributed_1 (TimeDist (None, 40, 2048) 23587712

lambda_1 (Lambda) (None, 2048) 0

Total params: 23,587,712
Trainable params: 0
Non-trainable params: 23,587,712

2018-11-16 23:04:23.350239: W tensorflow/core/framework/allocator.cc:122] Allocation of 2055208960 exceeds 10% of system memory.
2018-11-16 23:04:30.131324: W tensorflow/core/framework/allocator.cc:122] Allocation of 2129264640 exceeds 10% of system memory.
2018-11-16 23:04:31.361599: W tensorflow/core/framework/allocator.cc:122] Allocation of 2055208960 exceeds 10% of system memory.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
run-feature-extractor.sh: line 2: 15804 Aborted (core dumped) python2 batched_extractor.py

When i try to run the model from the scratch,got an error that ,"You don't have permission to access /~yu239/datasets/youtubeclips.zip on this server.
Server unable to read htaccess file, denying access to be safe"
How can i resolve it?
Thanks in advance...

Hi Rohit, I got the following error msg when running the extract features, do you have any idea? Thanks!

Traceback (most recent call last):
File "batched_extractor.py", line 147, in
encoded_frame_sequence = TimeDistributed(convnet_model)(video_input)
File "build/bdist.linux-x86_64/egg/keras/engine/topology.py", line 619, in call
File "build/bdist.linux-x86_64/egg/keras/layers/wrappers.py", line 211, in call
File "build/bdist.linux-x86_64/egg/keras/engine/topology.py", line 2085, in call
File "build/bdist.linux-x86_64/egg/keras/engine/topology.py", line 2235, in run_internal_graph
File "build/bdist.linux-x86_64/egg/keras/layers/normalization.py", line 193, in call
File "build/bdist.linux-x86_64/egg/keras/backend/tensorflow_backend.py", line 1004, in moving_average_update
File "/home/chikiuso/.conda/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/moving_averages.py", line 70, in assign_moving_average
update_delta = _zero_debias(variable, value, decay)
File "/home/chikiuso/.conda/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/moving_averages.py", line 180, in _zero_debias
"biased", initializer=biased_initializer, trainable=False)
File "/home/chikiuso/.conda/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1065, in get_variable
use_resource=use_resource, custom_getter=custom_getter)
File "/home/chikiuso/.conda/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 962, in get_variable
use_resource=use_resource, custom_getter=custom_getter)
File "/home/chikiuso/.conda/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 367, in get_variable
validate_shape=validate_shape, use_resource=use_resource)
File "/home/chikiuso/.conda/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 352, in _true_getter
use_resource=use_resource)
File "/home/chikiuso/.conda/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 664, in _get_single_variable
name, "".join(traceback.format_list(tb))))
ValueError: Variable bn_conv1/moving_mean/biased already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:

File "build/bdist.linux-x86_64/egg/keras/backend/tensorflow_backend.py", line 1004, in moving_average_update
x, value, momentum, zero_debias=True)
File "build/bdist.linux-x86_64/egg/keras/layers/normalization.py", line 193, in call
self.momentum),
File "build/bdist.linux-x86_64/egg/keras/engine/topology.py", line 619, in call
output = self.call(inputs, **kwargs)

swig/python detected a memory leak of type 'int64_t *', no destructor found.
swig/python detected a memory leak of type 'int64_t *', no destructor found.

FileNotFoundError: [Errno 2] No such file or directory: '../language_model/vocabulary_10.p'

Running Command: bash process-youtube-video.sh

raise ValueError(str(e))
ValueError: Dimension 0 in both shapes must be equal, but are 2617 and 2718. Shapes are [2617,256] and [2718,256]. for 'Assign_18' (op: 'Assign') with input shapes: [2617,256], [2718,256].

And, how can I use the pre-trained model to do the evaluation on all the videos? I would like to get the description results of video captioning. Thx.

IOError: [Errno 2] No such file or directory: '../language_model/vocabulary_10.p'
on runnng:
bash process-youtube-video.sh
while following:
Use a video from YouTube

could you please share the requirement.txt or the average_frame_features.pickle file.
which version of tensorflow used
tensorflow-valueerror-failed-to-convert-a-numpy-array-to-a-tensor-unsupported(np.ndarray)