vahidoox / deepcca Goto Github PK

I am trying to use this for a dataset of my own - VGG-fc8 features and GLoVE embeddings. However, when I add a regularization parameter, the loss quickly goes to nan. If I remove it, the loss grows larger (from negative to close to 0). What might I be doing wrong?

when I run deepcca.py ,error occurred about model.fit()

    model.fit([train_set_x1, train_set_x2], np.zeros(len(train_set_x1)),
              batch_size=batch_size, epochs=epoch_num, shuffle=True,
              validation_data=([valid_set_x1, valid_set_x2], np.zeros(len(valid_set_x1))),
              callbacks=[checkpointer]) 
ValueError: ('Error when checking model target: expected no data, but got:', array([ 0.,  0.,  0., ...,  0.,  0.,  0.]))

I have questions about dummy Y ,the labels ,why it says need no data ,but got np.zeors?

May I know which version of keras has been used. I would be great if I could get an earnest reply. I get different error with different versions.

Hi Vahid,
Hope you are doing well. I was wondering if you also have DCCAE implementation ( I saw that you referenced it in the main page).

Thank you very much,

Traceback (most recent call last):
File "DeepCCA.py", line 153, in
learning_rate, reg_par, outdim_size, use_all_singular_values)
File "/Users/gehuibin/Desktop/DeepCCA-master/models.py", line 22, in create_model
model.compile(loss=cca_loss(outdim_size, use_all_singular_values), optimizer=model_optimizer)
File "/Users/gehuibin/anaconda3/lib/python3.6/site-packages/keras/models.py", line 781, in compile
**kwargs)
File "/Users/gehuibin/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 840, in compile
sample_weight, mask)
File "/Users/gehuibin/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 446, in weighted
score_array = fn(y_true, y_pred)
File "/Users/gehuibin/Desktop/DeepCCA-master/objectives.py", line 22, in inner_cca_objective
H1 = y_pred[:, 0:o1].T
AttributeError: 'Tensor' object has no attribute 'T'

Hi~
It's an amazingly succinct code and it works well. But I'm wandering what's the reason for just use one eigenvalue and how can this method improve the stability?

Dear Vahid Noroozi!
Thanks for the code - very useful! I'm interested in hearing your opinion on the optimization of the network. You say that your implementation differs from the Galen Andrews (2013) original paper

"I used RMSProp (an adaptive version of gradient descent) instead of GD with momentum. "

But he uses L-BFGS because it's a full batch optimizer because:

"
Because the correlation objective is a function of the
entire training set that does not decompose into a sum
over data points, it is not clear how to use a stochastic
optimization procedure that operates on data points
one at a time[mini batches]"

doesn't this make both RMSprop and GD with momentum invalid?
From the original paper he includes how to take the derivative of the correlation function the two outputs of the two networks - how does this special derivative of the loss come into play in your code?

Thanks again for the great work i appreciate it very much!

Kind regards
Kasper

I am getting the following error. can someone help me to resolve the issue

Traceback (most recent call last):
File "/home2/anuprabha.m/kesav/DeepCCA/DeepCCA.py", line 10, in
from utils import load_data, svm_classify
File "/home2/anuprabha.m/kesav/DeepCCA/utils.py", line 6, in
from keras.utils.data_utils import get_file
ModuleNotFoundError: No module named 'keras.utils.data_utils'

Hi:
I took the code from GitHub and ran it in an Anaconda environment that has Keras 2.4.2 installed and ran into the following error:

runfile('M:/mypython/DeepCCA-master/DeepCCA.py', wdir='M:/mypython/DeepCCA-master')
Reloaded modules: utils, linear_cca
loading data ...
loading data ...
Traceback (most recent call last):

File "", line 1, in
runfile('M:/mypython/DeepCCA-master/DeepCCA.py', wdir='M:/mypython/DeepCCA-master')

File "C:\anaconda3\envs\keras\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 668, in runfile
execfile(filename, namespace)

I tried to change the line containing "Merge" to "from keras.layers import Dense, merge" and the line to calling it to "merge" - that created the error:

File "M:\mypython\DeepCCA-master\models.py", line 19, in create_model
model.add(merge([view1_model, view2_model], mode='concat'))

TypeError: 'module' object is not callable

Then I added "Concatenate" to the input and changed the line to "model.add(Concatenate([view1_model, view2_model]))". Now the data was loading OK but it failed when running "model.summary()" with:

File "C:\anaconda3\envs\keras\lib\site-packages\keras\engine\network.py", line 1252, in summary
'This model has not yet been built. '

ValueError: This model has not yet been built. Build the model first by calling build() or calling fit() with some data. Or specify input_shape or batch_input_shape in the first layer for automatic build.

Now I am at a loss how to proceed - I havn't used Keras until now.
Thanks

File "C:\anaconda3\envs\keras\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 108, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "M:/mypython/DeepCCA-master/DeepCCA.py", line 153, in
learning_rate, reg_par, outdim_size, use_all_singular_values)

File "M:\mypython\DeepCCA-master\models.py", line 19, in create_model
model.add(Merge([view1_model, view2_model], mode='concat'))

NameError: name 'Merge' is not defined

Hi there.. This is a very great code base. It all just WORKS. Good job.

I am interested in your thoughts about the objective function, however. Is such a complicated formulation really necessary, or could you just do the dot product of the projections and sum the diagonal -- e.g. Trace(f(X)^T \dot f(Y)) ? Does my question make sense?

What does the objective value represent? Is it the sum of the correlations?

Do you get similar results from the MNIST-split dataset (i.e. sum of ~39 corr coeff over 50 components) ??

With a few tries, I just found that theano=0.9.0 and keras=2.0.2 work well, and I hope this issue will help u who meet the same problems.

Hi,

I notice that the output of DCCA has two components and target classes appended to new_data. And in svm_classify, only the first component is used for classification. Could you explain the significance of these two components?

Hello :)
I hope you are doing well. I'm getting this error:

keep_dims is deprecated, use keepdims instead
Traceback (most recent call last):
  File "./DeepCCA.py", line 153, in <module>
    learning_rate, reg_par, outdim_size, use_all_singular_values)
  File "/home/enx/Downloads/DeepCCA-master2/models.py", line 19, in create_model
    model.add(concatenate([view1_model, view2_model], mode='concat'))
  File "/home/enx/Downloads/DeepCCA-master2/deepcca2_p27/local/lib/python2.7/site-packages/keras/layers/merge.py", line 384, in concatenate
    return Concatenate(axis=axis, **kwargs)(inputs)
  File "/home/enx/Downloads/DeepCCA-master2/deepcca2_p27/local/lib/python2.7/site-packages/keras/layers/merge.py", line 133, in __init__
    super(Concatenate, self).__init__(**kwargs)
  File "/home/enx/Downloads/DeepCCA-master2/deepcca2_p27/local/lib/python2.7/site-packages/keras/layers/merge.py", line 15, in __init__
    super(_Merge, self).__init__(**kwargs)
  File "/home/enx/Downloads/DeepCCA-master2/deepcca2_p27/local/lib/python2.7/site-packages/keras/engine/topology.py", line 275, in __init__
    raise TypeError('Keyword argument not understood:', kwarg)
TypeError: ('Keyword argument not understood:', 'mode')

How can i fix this ? thank you

Hi,
I was working on a PyTorch version of your keras code of the cca loss.
I was using a deep network to get features from X1, which gives me the H1 and the H2 is an unperturbed version of X2. The aim is to maximize the correlation between H1 and H2, so basically learn a transformation of X1 (using the deep network) that maximizes its correlation with X2.
However, the backward pass returned Nan in the 'SymeigBackward' function (this is in PyTorch, the backprop function for symeig function. It is supposed to work the same as the T.nlinalg.eigh function of Theano). This problem gets somewhat resolved if I decrease the regularization parameter cca_reg1. But I am not sure why this happens. Any help would be much appreciated.
Also, even if the backward doesn't throw Nans the corr loss doesn't seem to reduce. I guess this might be something else though. (Just mentioning in case it might help understand what's going on).
Thanks a bunch :)

loading data ...
loading data ...
Traceback (most recent call last):
File "DeepCCA.py", line 153, in
learning_rate, reg_par, outdim_size, use_all_singular_values)
File "X:\TestData\DeepCCA\models.py", line 15, in create_model
view1_model = build_mlp_net(layer_sizes1, input_size1, reg_par)
File "X:\TestData\DeepCCA\models.py", line 41, in build_mlp_net
kernel_regularizer=l2(reg_par)))
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\models.py", line 464, in add
layer(x)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 576, in call
self.build(input_shapes[0])
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\layers\core.py", line 830, in build
constraint=self.kernel_constraint)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 397, in add_weight
weight = K.variable(initializer(shape),
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\initializers.py", line 212, in call
dtype=dtype, seed=self.seed)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\backend\theano_backend.py", line 2249, in random_uniform
return rng.uniform(shape, low=minval, high=maxval, dtype=dtype)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\sandbox\rng_mrg.py", line 1354, in uniform
rstates = self.get_substream_rstates(nstreams, dtype)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\configparser.py", line 117, in res
return f(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\sandbox\rng_mrg.py", line 1256, in get_substream_rstates
multMatVect(rval[0], A1p72, M1, A2p72, M2)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\sandbox\rng_mrg.py", line 66, in multMatVect
[A_sym, s_sym, m_sym, A2_sym, s2_sym, m2_sym], o, profile=False)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\compile\function.py", line 326, in function
output_keys=output_keys)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\compile\pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 1795, in orig_function
defaults)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 1661, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\vm.py", line 1047, in make_all
impl=impl))
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\op.py", line 935, in make_thunk
no_recycling)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\op.py", line 839, in make_c_thunk
output_storage=node_output_storage)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1190, in make_thunk
keep_lock=keep_lock)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1131, in compile
keep_lock=keep_lock)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1586, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 1159, in module_from_key
module = lnk.compile_cmodule(location)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1489, in compile_cmodule
preargs=preargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 2325, in compile_str
return dlimport(lib_filename)
File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 302, in dlimport
rval = import(module_name, {}, {}, [module_name])
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

the function create_model raises error when evaluating customized cca objective:
model = create_model(layer_sizes1, layer_sizes2, input_shape1, input_shape2, learning_rate, reg_par, outdim_size, use_all_singular_values)
results in errors AttributeError: 'Tensor' object has no attribute 'T'. Below is the complete error message:
`---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in ()
38 # Building, training, and producing the new features by DCCA
39 model = create_model(layer_sizes1, layer_sizes2, input_shape1, input_shape2,
---> 40 learning_rate, reg_par, outdim_size, use_all_singular_values)
41 model.summary()
42 #model = train_model(model, data1, data2, epoch_num, batch_size)

~/Downloads/DeepCCA-master/models.py in create_model(layer_sizes1, layer_sizes2, input_size1, input_size2, learning_rate, reg_par, outdim_size, use_all_singular_values)
20
21 model_optimizer = RMSprop(lr=learning_rate)
---> 22 model.compile(loss=cca_loss(outdim_size, use_all_singular_values), optimizer=model_optimizer)
23
24 return model

~/anaconda2/envs/dcca/lib/python3.5/site-packages/keras/models.py in compile(self, optimizer, loss, metrics, sample_weight_mode, weighted_metrics, **kwargs)
782 sample_weight_mode=sample_weight_mode,
783 weighted_metrics=weighted_metrics,
--> 784 **kwargs)
785 self.optimizer = self.model.optimizer
786 self.loss = self.model.loss

~/anaconda2/envs/dcca/lib/python3.5/site-packages/keras/engine/training.py in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, weighted_metrics, target_tensors, **kwargs)
848 with K.name_scope(self.output_names[i] + '_loss'):
849 output_loss = weighted_loss(y_true, y_pred,
--> 850 sample_weight, mask)
851 if len(self.outputs) > 1:
852 self.metrics_tensors.append(output_loss)

~/anaconda2/envs/dcca/lib/python3.5/site-packages/keras/engine/training.py in weighted(y_true, y_pred, weights, mask)
448 """
449 # score_array has ndim >= 2
--> 450 score_array = fn(y_true, y_pred)
451 if mask is not None:
452 # Cast the mask to floatX to avoid float64 upcasting in theano

~/Downloads/DeepCCA-master/objectives.py in inner_cca_objective(y_true, y_pred)
20
21 # unpack (separate) the output of networks for view 1 and view 2
---> 22 H1 = y_pred[:, 0:o1].T
23 H2 = y_pred[:, o1:o1+o2].T
24

AttributeError: 'Tensor' object has no attribute 'T'
`

hi,
thank you for this implementation...

i was wondering - i saw that in your code you took 10 dimensions for the mnist, while in the paper they took 50 dimensions

what do you think that the val will be for 10 dimensions ?

another question- i saw that the loss is calculated with (-corr) that is for 10 dimensions and it is just the same as validation- for me it converge to 3 - i think it is not as expected - what do you think?

and for last, i had a problem using the function T.ones, it raised an exception for - it cant used with itter, so i changed it to T.ones_like and it worked - do you think this is the right implementation?

thanks,

Hi,
thank you for this implementation! While I was reading your code ,I noticed that the definition of topk loss function and all singular value loss function are not consistant.

if use_all_singular_values:
       # all singular values are used to calculate the correlation
       corr = T.sqrt(T.nlinalg.trace(T.dot(Tval.T, Tval)))
else:
       # just the top outdim_size singular values are used
       [U, V] = T.nlinalg.eigh(T.dot(Tval.T, Tval))
       U = U[T.gt(U, eps).nonzero()[0]]
       U = U.sort()
       corr = T.sum(T.sqrt(U[0:outdim_size]))

when use all singular value, first calculate the sum of singular values then do square root,
However, when use just the top k singular values ,first do square root then calculate the sum.

Am I right to think that they should both be the first case?
Thanks.