escvm / efficient-capsnet Goto Github PK

View Code? Open in Web Editor NEW

261.0 6.0 59.0 45.38 MB

Official TensorFlow code for the paper "Efficient-CapsNet: Capsule Network with Self-Attention Routing".

Home Page: https://www.nature.com/articles/s41598-021-93977-0

License: Apache License 2.0

Jupyter Notebook 28.76% Python 71.24%

deep-learning computer-vision capsule-network tensorflow

efficient-capsnet's Introduction

~ Efficient-CapsNet ~

Are you tired of over inflated and overused convolutional neural networks? You're right! It's time for CAPSULES :)

This repository has been made for two primarly reasons:

open source the code (most of) developed during our "first-stage" research on capsules, summarized by the forthcoming article "Efficient-CapsNet: Capsule Network with Self-Attention Routing". The repository let you play with Efficient-CapsNet and let you set the base for your own experiments.
be an hub and a headlight in the cyberspace to spread to the machine learning comunity the intrinsic potential and value of capsule. However, albeit remarkable results achieved by capsule networks, we're fully aware that they're only limited to toy datasets. Nevertheless, there's a lot to make us think that with the right effort and collaboration of the scientific community, capsule based networks could really make a difference in the long run. For now, feel free to dive in our work :))

1.0 Getting Started

1.1 Installation

Python3 and Tensorflow 2.x are required and should be installed on the host machine following the official guide. Good luck with it!

Clone this repository

git clone https://github.com/EscVM/Efficient-CapsNet.git

Install the required packages
```
pip3 install -r requirements.txt
```

Peek inside the requirements file if you have everything already installed. Most of the dependencies are common libraries.

2.0 Efficient-CapsNet Notebooks

The repository provides two starting notebooks to make you confortable with our architecture. They all have the information and explanations to let you dive further in new research and experiments. The first one let you test Efficient-CapsNet over three different datasets. The repository is provided with some of the weights derived by our own experiments. On the other hand, the second one let you train the network from scratch. It's a very lightweight network so you don't need "Deep Mind" TPUs arsenal to train it. However, even if a GP-GPU is not compulsory, it's strongly suggested (No GPU, no deep learning, no party).

3.0 Original CapsNet Notebooks

It goes without saying that our work has been inspiered by Geoffrey Hinton and his article "Dynamic Routing Between Capsules". It's really an honor to build on his idea. Nevertheless, when we did our first steps in the capsule world, we were pretty disappointed in finding that all repositories/implementations were ultimately wrong in some aspects. So, we implemented everything from scratch, carefully following the original Sara's repository. However, our implementation, besides beeing written for the new TensorFlow 2 version, is much more easier and practical to use. Sara's one is really overcomplicated and too mazy that you can lost pretty easily.

As for the previous section we provide two notebooks, one for testing (weights have been derived from Sara's repository) and one for training.

Nevertheless, there's a really negative note (at least for us:)); as all other repositories that you can find on the web, also our one is not capable to achieve the scores reported in their paper. We really did our best, but there is no way to make the network achieve a score greater than 99.64% on MNIST. Exactly for this reason, weights provided in this repository are derived from their repository. Anyway, it's Geoffrey so we can excuse him.

4.0 Capsules Dimensions Perturbation Notebook

The network is trained with a reconstruction regularizer that is simply a fully connected network trained in conjuction with the main one. So, we can use it to visualize the inner capsules reppresentations. In particular, we should expect that a dimension of a digit capsule should learn to span the space of variations in the way digits of that class are instantiated. We can see what the individual dimensions represent by making use of the decoder network and injecting some noise to one of the dimensions of the main digit capsule layer that is predicting the class of the input.

So, we coded a practical notebook in which you can dynamically tweak whichever dimension you want of the capsule that is making the prediction (longest one).

Finally, if you don't have the necessary resources (GP-GPU holy grail) you can still try this interesting notebook out on .

Citation

Use this bibtex if you enjoyed this repository and you want to cite it:

@article{mazzia2021efficient,
  title={Efficient-CapsNet: capsule network with self-attention routing},
  author={Mazzia, Vittorio and Salvetti, Francesco and Chiaberge, Marcello},
  year={2021},
  journal={Scientific reports},
  publisher={Nature Publishing Group},
  volume={11}
}

efficient-capsnet's People

Contributors

Stargazers

Watchers

efficient-capsnet's Issues

一些小细节

model.py里边第145行，保存名称应该是efficient_capsnet，不是original_capsnet。pre_process_multimnist.py文件中第76行应该为labels[i]，而不是label[i]

Performance issues in /utils (by P3)

Hello! I've found a performance issue in /utils: batch() should be called before map(), which could make your program more efficient. Here is the tensorflow document to support it.

Detailed description is listed below:

/pre_process_mnist.py: dataset_train.batch(batch_size)(here) should be called before dataset_train.map(image_rotate_random,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(image_shift_rand,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(image_squish_random,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(image_erase_random,num_parallel_calls=PARALLEL_INPUT_CALLS)(here) and dataset_train.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).
/pre_process_mnist.py: dataset_test.batch(batch_size)(here) should be called before dataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).
/pre_process_smallnorb.py: dataset_train.batch(batch_size)(here) should be called before dataset_train.map(random_patches,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(random_brightness,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(random_contrast,num_parallel_calls=PARALLEL_INPUT_CALLS)(here) and dataset_train.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).
/pre_process_smallnorb.py: dataset_test.batch(1)(here) should be called before dataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).

Besides, you need to check the function called in map()(e.g., generator called in dataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)) whether to be affected or not to make the changed code work properly. For example, if generator needs data with shape (x, y, z) as its input before fix, it would require data with shape (batch_size, x, y, z).

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

UnimplementedError: Fused conv implementation does not support grouped convolutions for now.

I got the same error on OS X 12.1 as well as on Google CoLab Pro.

OS X
tf.version
'2.5.0'

CoLab
tf.version
2.6.0

---------------------------------------------------------------------------
UnimplementedError                        Traceback (most recent call last)
<ipython-input-9-03c7f065910f> in <module>
----> 1 history = model_train.train(dataset, initial_epoch=0)

~/Efficient-CapsNet/models/model.py in train(self, dataset, initial_epoch)
    178         print('-'*30 + f'{self.model_name} train' + '-'*30)
    179 
--> 180         history = self.model.fit(dataset_train,
    181           epochs=self.config[f'epochs'], steps_per_epoch=steps,
    182           validation_data=(dataset_val), batch_size=self.config['batch_size'], initial_epoch=initial_epoch,

~/anaconda3/envs/ai/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1181                 _r=1):
   1182               callbacks.on_train_batch_begin(step)
-> 1183               tmp_logs = self.train_function(iterator)
   1184               if data_handler.should_sync:
   1185                 context.async_wait()

~/anaconda3/envs/ai/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    887 
    888       with OptionalXlaContext(self._jit_compile):
--> 889         result = self._call(*args, **kwds)
    890 
    891       new_tracing_count = self.experimental_get_tracing_count()

~/anaconda3/envs/ai/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    948         # Lifting succeeded, so variables are initialized and we can run the
    949         # stateless function.
--> 950         return self._stateless_fn(*args, **kwds)
    951     else:
    952       _, _, _, filtered_flat_args = \

~/anaconda3/envs/ai/lib/python3.8/site-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
   3021       (graph_function,
   3022        filtered_flat_args) = self._maybe_define_function(args, kwargs)
-> 3023     return graph_function._call_flat(
   3024         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
   3025 

~/anaconda3/envs/ai/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1958         and executing_eagerly):
   1959       # No tape is watching; skip to running the function.
-> 1960       return self._build_call_outputs(self._inference_function.call(
   1961           ctx, args, cancellation_manager=cancellation_manager))
   1962     forward_backward = self._select_forward_and_backward_functions(

~/anaconda3/envs/ai/lib/python3.8/site-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    589       with _InterpolateFunctionError(self):
    590         if cancellation_manager is None:
--> 591           outputs = execute.execute(
    592               str(self.signature.name),
    593               num_outputs=self._num_outputs,

~/anaconda3/envs/ai/lib/python3.8/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     57   try:
     58     ctx.ensure_initialized()
---> 59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:

UnimplementedError:  Fused conv implementation does not support grouped convolutions for now.
	 [[node Efficinet_CapsNet_Generator/Efficient_CapsNet/primary_caps/conv2d/BiasAdd (defined at /Users/davidlaxer/Efficient-CapsNet/utils/layers.py:129) ]] [Op:__inference_train_function_3601]

Function call stack:
train_function

I also tried running on tensorflow-metal w/GPU ... machine unresponsive ... had to reboot.

UnimplementedError: Fused conv implementation does not support grouped convolutions for now

Thanks for sharing the code of your awesome work.

I am trying to run the different .ipynb of your work. However, once I run the "1.2.1 Evaluate the model"

model_test.evaluate(mnist_dataset.X_test, mnist_dataset.y_test)

I get the following error:

`UnimplementedError: Fused conv implementation does not support grouped convolutions for now.
[[node sequential/conv2d/Relu (defined at :1) ]] [Op:__inference_train_function_1358]

Function call stack:
train_function`

(I am running on CPU)
I would be grateful for any kind of help that will solve this issue. Thank you.

how to use it for other datasets?

just build the dataset in some format?

or need to rewrite the graph?

Multiple Caps Layers

we are very impressed by your article and the improvement you implemented.
In the article you wrote:
"However, we adopt only two layers of capsules due to the relative simplicity of the dataset investigated"
we are trying to use your caps implementation with our problem. our input is 40X40X40 and a more complicated image than MNIST, therefore we want to use more caps layers.
should we simply duplicate the PrimaryCaps layers before the FCCaps layer?

Thanks

Yaniv.

Layer

Thank you for your contribution to Capsule Network
For the network structure you mentioned, the order of the dimensions of the weight w in the paper is not quite the same as in the code, and for the self-attention matrix, it seems to be dl in the paper, but why is it dl+1 in the code

Does anyone use the Efficient-CapsNet of the pytorch framework？

Overfitting

It is always overfitting on my own dataset, even I try my best to simplify the model which only contains 2000+ parameters.
I data is (3003001) pictures, 800+ trainsing samples.
Anyone can guide me this? Thanks

Visualisation notebook throws UnimplementedError: Fused conv implementation does not support grouped convolutions for now

I am getting an error when running model_test.evaluate(mnist_dataset.X_test, mnist_dataset.y_test) in the dynamic visualisations notebook. The first time I installed the requirements there were a couple of warnings, about incompatible versions. I tried installing again and everything was then OK.

The notebook runs fully on Colab, thanks, but I would like to get it working locally. Any ideas to try and fix this please?

UnimplementedError Traceback (most recent call last)
in
----> 1 model_test.evaluate(mnist_dataset.X_test, mnist_dataset.y_test)

C:\DSAI\Dissertation\Efficient-CapsNet\models\model.py in evaluate(self, X_test, y_test)
94 acc = np.mean(acc)
95 else:
---> 96 y_pred, X_gen = self.model.predict(X_test)
97 acc = np.sum(np.argmax(y_pred, 1) == np.argmax(y_test, 1))/y_test.shape[0]
98 test_error = 1 - acc

~\miniconda3\envs\dsai_py3.8\lib\site-packages\tensorflow\python\keras\engine\training.py in predict(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)
1627 for step in data_handler.steps():
1628 callbacks.on_predict_batch_begin(step)
-> 1629 tmp_batch_outputs = self.predict_function(iterator)
1630 if data_handler.should_sync:
1631 context.async_wait()

~\miniconda3\envs\dsai_py3.8\lib\site-packages\tensorflow\python\eager\def_function.py in call(self, *args, **kwds)
826 tracing_count = self.experimental_get_tracing_count()
827 with trace.Trace(self._name) as tm:
--> 828 result = self._call(*args, **kwds)
829 compiler = "xla" if self._experimental_compile else "nonXla"
830 new_tracing_count = self.experimental_get_tracing_count()

~\miniconda3\envs\dsai_py3.8\lib\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
892 *args, **kwds)
893 # If we did not create any variables the trace we have is good enough.
--> 894 return self._concrete_stateful_fn._call_flat(
895 filtered_flat_args, self._concrete_stateful_fn.captured_inputs) # pylint: disable=protected-access
896

~\miniconda3\envs\dsai_py3.8\lib\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1916 and executing_eagerly):
1917 # No tape is watching; skip to running the function.
-> 1918 return self._build_call_outputs(self._inference_function.call(
1919 ctx, args, cancellation_manager=cancellation_manager))
1920 forward_backward = self._select_forward_and_backward_functions(

~\miniconda3\envs\dsai_py3.8\lib\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager)
553 with _InterpolateFunctionError(self):
554 if cancellation_manager is None:
--> 555 outputs = execute.execute(
556 str(self.signature.name),
557 num_outputs=self._num_outputs,

~\miniconda3\envs\dsai_py3.8\lib\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
57 try:
58 ctx.ensure_initialized()
---> 59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:

UnimplementedError: Fused conv implementation does not support grouped convolutions for now.
[[node Efficinet_CapsNet_Generator/Efficient_CapsNet/primary_caps_2/conv2d/BiasAdd (defined at C:\DSAI\Dissertation\Efficient-CapsNet\utils\layers.py:129) ]] [Op:__inference_predict_function_3999]

Function call stack:
predict_function

performance problems

Is there a problem with the code?
The performance of the model may be abnormal.
There may be some problems with the accuracy.

Epoch 35/150
3750/3750 [==============================] - 142s 38ms/step - loss: 0.0191 - Original_CapsNet_loss: 0.0106 - Generator_loss: 0.0217 - Original_CapsNet_multiAccuracy: 0.5635 - val_loss: 0.0138 - val_Original_CapsNet_loss: 0.0056 - val_Generator_loss: 0.0209 - val_Original_CapsNet_multiAccuracy: 0.5458
Epoch 36/150
3750/3750 [==============================] - 141s 38ms/step - loss: 0.0184 - Original_CapsNet_loss: 0.0100 - Generator_loss: 0.0215 - Original_CapsNet_multiAccuracy: 0.5591 - val_loss: 0.0133 - val_Original_CapsNet_loss: 0.0051 - val_Generator_loss: 0.0209 - val_Original_CapsNet_multiAccuracy: 0.5900
Epoch 37/150
3750/3750 [==============================] - 142s 38ms/step - loss: 0.0181 - Original_CapsNet_loss: 0.0098 - Generator_loss: 0.0212 - Original_CapsNet_multiAccuracy: 0.5619 - val_loss: 0.0132 - val_Original_CapsNet_loss: 0.0052 - val_Generator_loss: 0.0204 - val_Original_CapsNet_multiAccuracy: 0.5344

how can I use my dataset to train this model ?

my image size is 64643

Can the model be used for NLP tasks

Depthwise separable convolution incomplete?

Hi @EscVM,

to me, it seems like the depth wise separable convolution implemented in https://github.com/EscVM/Efficient-CapsNet/blob/705449c/utils/layers.py#L123 lacks the point-wise convolution part?!

Shouldn't there be a second Conv2D layer with F filters of kernel size (1,1)?

Otherwise, you're doing a simple depth-wise convolution, right?

I am referencing https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728 here btw.
Not sure if you gave a reference for the depthwise separable convolution in your paper.

Output is always 1

I'm running the following code to test my custom images, and here is the cude:

import tensorflow as tf
from utils import Dataset, plotImages, plotWrongImages
from models import EfficientCapsNet
import cv2, numpy as np

img_path = "imgs/box_9.jpg"
img_np = cv2.bitwise_not(cv2.imread(img_path, 0)) # invert conversion

gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)

# some parameters
model_name = 'MULTIMNIST'
custom_path = None # if you've trained a new model, insert here the full graph weights path

dataset = Dataset(model_name, config_path='config.json')

model_test = EfficientCapsNet(model_name, mode='test', verbose=True, custom_path=custom_path)

model_test.load_graph_weights() # load graph weights (bin folder)

model_test.evaluate(dataset.X_test, dataset.y_test) # if "smallnorb" use X_test_patch

#not working with MultiMNIST

shape = dataset.X_test[0].shape

img_np = cv2.resize(img_np, shape[:2], interpolation=cv2.INTER_AREA)
img_np = img_np[..., None]

y_pred = model_test.predict(np.expand_dims(img_np, axis=0))[0] # if "smallnorb" use X_test_patch

print(np.argmax(y_pred))
print(y_pred)

cv2.imshow('show', img_np)
cv2.waitKey(0)

And here is the result:

[[0.3153154  0.956559   0.2567287  0.2726928  0.18593171 0.6188526
  0.14006346 0.44058013 0.36337793 0.15720315]]

As it's shown, index 1 has highest value and it means the output is 1. But I tested 13 images and all has highest value in index 1 (output is always 1). What am I doing wrong?

Error running code

Code:

from utils import AffineVisualizer, Dataset
from models import EfficientCapsNet

mnist_dataset = Dataset('MNIST', config_path='config.json') # only MNIST

model_test = EfficientCapsNet('MNIST', mode='test', verbose=False)
model_test.load_graph_weights()
model_play = EfficientCapsNet('MNIST', mode='play', verbose=False)
model_play.load_graph_weights()

model_test.evaluate(mnist_dataset.X_test, mnist_dataset.y_test)`

ERROR:
````------------------------------MNIST Evaluation------------------------------
2021-05-02 16:05:36.390373: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-05-02 16:05:36.390888: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2095074999 Hz
Traceback (most recent call last):
  File "test01.py", line 12, in <module>
    model_test.evaluate(mnist_dataset.X_test, mnist_dataset.y_test)
  File "/home/NETID/datle/css_586/Efficient-CapsNet-main/models/model.py", line 96, in evaluate
    y_pred, X_gen =  self.model.predict(X_test)
  File "/home/NETID/datle/css_586/venv/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1629, in predict
    tmp_batch_outputs = self.predict_function(iterator)
  File "/home/NETID/datle/css_586/venv/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/home/NETID/datle/css_586/venv/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 895, in _call
    filtered_flat_args, self._concrete_stateful_fn.captured_inputs)  # pylint: disable=protected-access
  File "/home/NETID/datle/css_586/venv/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/NETID/datle/css_586/venv/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/home/NETID/datle/css_586/venv/lib64/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnimplementedError:  Fused conv implementation does not support grouped convolutions for now.
	 [[node Efficinet_CapsNet_Generator/Efficient_CapsNet/primary_caps/conv2d/BiasAdd (defined at /home/NETID/datle/css_586/Efficient-CapsNet-main/utils/layers.py:129) ]] [Op:__inference_predict_function_2000]

Function call stack:
predict_function`

Performance issues in utils/pre_process_mnist.py(P2)

Hello,I found a performance issue in the definition of generate_tf_data ,
utils/pre_process_mnist.py,
dataset_train = dataset_train.map was called without num_parallel_calls.
I think it will increase the efficiency of your program if you add this.

Here is the documemtation of tensorflow to support this thing.