sascha-kirch / deepsaki Goto Github PK

DeepSaki is an add-on to TensorFlow. It provides a variaty of custom classes ranging from activation functions to entire models, helper functions to facilitate connectiong to your, compute HW and many more!

Home Page: https://sascha-kirch.github.io/DeepSaki/

License: MIT License

Python 99.62% Dockerfile 0.19% HTML 0.18%

computer-vision deep-learning distributed machine-learning mit-license python tensorflow tpu

deepsaki's Introduction

Hi there 👋

My name is Sascha and I am a PhD Student working in the field of deeplearning. I am a hardware and software engineer.

🔭 I’m currently working on generative multi-modal deeplearning for depth estimation of scenes and have been working adversarial domain adaptation for RGBD data before
👯 I’m open to collaborate on interesting new topics to meet new people and learn new things!
✌️ I hope my repos are usefull to you and would be happy for any constructive feedback and suggestion to further improve

To find out more about me check out my website: https://sascha-kirch.github.io/

since 04.01.2022

Star History

deepsaki's People

Contributors

Stargazers

Watchers

deepsaki's Issues

Support * for imports

To support "from DeepSaki import *" I need to change the init.py files and add support for all

initial implementation of a CycleGAN

Add a CycleGAN framework as used in the VoloGAN and change it to be applicable to any data not only RGB-D.

Image from Readme is not displayed on PyPi

I use the ./docs/README.me file as long description on PyPi. Since the DeepSaki logo is loaded with a relative link, it is not rendered when uploaded to PyPi.

improve imports of init.py files

Currently in the initi.py the exposed api's are imported. The issue is that then everytime everything is imported, eventhough I might only import a single API.

Pypi status in readme not rendered

Most likely some metadata is missing in pyprobect.toml

Previw image of website

Preview image on the documentation is cropped. The aspect ratio of the logo must be adapted.

add tf.keras.layers.InputSpec to custom layers layers

with tf.keras.layers.InputSpec(dim=xxx) one can specify the specification for an input when the layer is called.

Dont forget: unit test

Change refference from develop to latest before new release

DeepSaki/overrides/main.html

Line 7 in 7f7dc3e

 <meta property="og:image" content="https://sascha-kirch.github.io/DeepSaki/develop/assets/images/social_card.png" /> 

rFFTPooling2D() in fourier_pooling.py is no valid operation

The sequence length of the DFT and the iDFT must have the same length in order for the cropping operation to have the desired effect.

The cropping in the frequency domain only makes sence, if not transformed back to the spatial domain. It should stay in the frequency domain, cropped several times and then finally can either be fed to a classification head to classify in the frequency domain or to transform back into the spatial domain using the initial sequence length.

FourierConvolution2D () in fourier_layer.py does not consider circular convolution

The convolution theorem for DFTs differs from standard fourier transforms.
Solution:

Zero Pad image
DFT, multiply, iDFT
circulate result
crop to original size

Code:

DeepSaki/DeepSaki/layers/fourier_layer.py

Lines 54 to 112 in 10982b1

 def call(self, inputs): 

 if not self.built: 

 raise ValueError('This model has not yet been built.') 

 #FFT2D is calculated over last two dimensions!  

 if not self.isChannelFirst: 

 inputs = tf.einsum("bhwc->bchw",inputs) 

 outputs_F = np.ndarray(shape=self.out_shape) 

 # Pad the kernel to the shape of the input to enable element-wise multiplication 

 signal_shape = tf.shape(inputs) 

 kernel_shape = tf.shape(self.kernel) 

 x_pad = signal_shape[2] - kernel_shape[2] 

 y_pad = signal_shape[3] - kernel_shape[3] 

 # paddings shape is [2,4] because rank of inputs is 4, and 2 for height and width 

 paddings = [[0,0], 

 [0,0], 

 [0,x_pad], 

 [0,y_pad] 

 ] 

 kernels_padded = tf.pad(self.kernel, paddings) # [out_channels, inp_channel, height,width,out_channe] 

 #print("Shape: inputs {}".format(np.shape(inputs))) 

 #print("Shape: kernels_padded {}".format(np.shape(kernels_padded))) 

 # Compute DFFTs for both inputs and kernel weights 

 inputs_F = tf.signal.rfft2d(inputs) #[batch,height,width,channel] 

 kernels_F = tf.signal.rfft2d(kernels_padded) 

 #kernels_F = tf.math.conj(kernels_F) #calculate conjugate to be mathematically correct with the cross-corelation implementation. Not important, since filter is learned!  

 #print("Shape: inputs_F {}".format(np.shape(inputs_F))) 

 #print("Shape: kernels_F {}".format(np.shape(kernels_F))) 

 # Apply filters by element wise multiplications 

 for filter in range(self.filters): 

 #print("Shape: kernels_F[filter,:,:,:] {}".format(np.shape(kernels_F[filter,:,:,:]))) 

 outputs_F = tf.concat( 

 [outputs_F, 

 tf.reduce_sum( 

 inputs_F * kernels_F[filter,:,:,:], #inputs:(batch, inp_filter, height, width ), fourier_filter:(...,out_filter,inp_filter,height, width) 

 axis = -3, # sum over all applied filters 

 keepdims = True 

 )], 

 axis = -3 # is the new filter count, since channel first 

 ) 

 #print("Shape: outputs_F {}".format(np.shape(outputs_F))) 

 # Inverse rDFFT 

 output = tf.signal.irfft2d(outputs_F) 

 #output = tf.math.real(output) 

 if self.use_bias: 

 output += self.bias 

 #reverse the channel configuration to its initial config  

 if not self.isChannelFirst: 

 output = tf.einsum("bchw->bhwc",output) 

 return output

TensorFlow Addons depricated

TFA will not be supported anymore soon. Despite the anoying warning that can't be surpressed, I'd like to remove the dependency on that library and refactor the code to not depend on it anymore.

initial implementation of diffusion api

Add a new API for diffusion models and start with a DDPM

FourierLayer() has instance methods that do not act on the instance.

FourierLayer() has methods like _matrix_product() or _change_to_channel_first() that do not require the instance of the class.
They have been simply put into a base class that is then inherited from.
So it is actually not really a base class but rather a collection of functions used in the child classes.

They could be turned into static methods.

	def call(self, inputs):
	if not self.built:
	raise ValueError('This model has not yet been built.')

	#FFT2D is calculated over last two dimensions!
	if not self.isChannelFirst:
	inputs = tf.einsum("bhwc->bchw",inputs)

	outputs_F = np.ndarray(shape=self.out_shape)

	# Pad the kernel to the shape of the input to enable element-wise multiplication
	signal_shape = tf.shape(inputs)
	kernel_shape = tf.shape(self.kernel)
	x_pad = signal_shape[2] - kernel_shape[2]
	y_pad = signal_shape[3] - kernel_shape[3]
	# paddings shape is [2,4] because rank of inputs is 4, and 2 for height and width
	paddings = [[0,0],
	[0,0],
	[0,x_pad],
	[0,y_pad]
	]
	kernels_padded = tf.pad(self.kernel, paddings) # [out_channels, inp_channel, height,width,out_channe]
	#print("Shape: inputs {}".format(np.shape(inputs)))
	#print("Shape: kernels_padded {}".format(np.shape(kernels_padded)))

	# Compute DFFTs for both inputs and kernel weights
	inputs_F = tf.signal.rfft2d(inputs) #[batch,height,width,channel]
	kernels_F = tf.signal.rfft2d(kernels_padded)
	#kernels_F = tf.math.conj(kernels_F) #calculate conjugate to be mathematically correct with the cross-corelation implementation. Not important, since filter is learned!

	#print("Shape: inputs_F {}".format(np.shape(inputs_F)))
	#print("Shape: kernels_F {}".format(np.shape(kernels_F)))

	# Apply filters by element wise multiplications
	for filter in range(self.filters):
	#print("Shape: kernels_F[filter,:,:,:] {}".format(np.shape(kernels_F[filter,:,:,:])))
	outputs_F = tf.concat(
	[outputs_F,
	tf.reduce_sum(
	inputs_F * kernels_F[filter,:,:,:], #inputs:(batch, inp_filter, height, width ), fourier_filter:(...,out_filter,inp_filter,height, width)
	axis = -3, # sum over all applied filters
	keepdims = True
	)],
	axis = -3 # is the new filter count, since channel first
	)

	#print("Shape: outputs_F {}".format(np.shape(outputs_F)))
	# Inverse rDFFT
	output = tf.signal.irfft2d(outputs_F)
	#output = tf.math.real(output)

	if self.use_bias:
	output += self.bias

	#reverse the channel configuration to its initial config
	if not self.isChannelFirst:
	output = tf.einsum("bchw->bhwc",output)

	return output