mathiasgruber / pconv-keras Goto Github PK
View Code? Open in Web Editor NEWUnofficial implementation of "Image Inpainting for Irregular Holes Using Partial Convolutions". Try at: www.fixmyphoto.ai
License: MIT License
Unofficial implementation of "Image Inpainting for Irregular Holes Using Partial Convolutions". Try at: www.fixmyphoto.ai
License: MIT License
Sorry if this is a stupid question, I'm a newbie. I've tried reading the notebooks and looking at 'Issues', but I don't quite get how to get this thing to work. Here are some issues I'm facing:
Thank you
thanks alot for sharing yr code.
how can i use it to train on my own dataset ? also i have a question regarding used loss function, from wt i understand of yr code is that u only used root mean sqaure loss, is that correct?
yr help is really appropriated
Hi guys, I was wondering if Raspberry Pi supported with Intel Movidius Neural Compute Stick able to support the image inpainting process (assuming training is completed on a separate decent computer)?
thanks a lot for sharing yr code.I spent many time to use my device to train.Can you publish your pretrained models.
hello, I'm testing your code with model in readme. And I have a question in step5.
when the code deepcopy mask to chunked_masks in chunked_masks = chunker.dimension_preprocess(deepcopy(mask))
, it shows that AttributeError: 'MaskGenerator' object has no attribute 'shape'
. Could you please help me to solve the problem? Thank you
Hi Mathias,
What is an advantage of padding the mask and the image outside of K.conv2d()
function (like you do here)?
It seems to me that your original implementation (with padding done inside K.conv2d()
and using the "same" padding scheme) should give the same result. The normalisation is based on the mask padded the same way and has pixels near the edges down-weighted properly. What am I missing?
From Notebook 5, cell 2
if os.path.basename(os.getcwd()) != 'PConv-Keras':
os.chdir('..')
from libs.pconv_model import PConvUnet
from libs.util import random_mask, ImageChunker
%load_ext autoreload
%autoreload 2
No such code is found.
I was able to get it to work by using MaskGenerator, I'm assuming you cleaned up the mask generation code then didn't update the notebook.
I'm working on PR
I am attempting to use transfer learning techniques to apply this model to a similar problem in the game of battleships. I am using the model to predict the positions of ships in a partially visible board (as in the game of battleships). The board of battleships is a 10 by 10 2d array with numbers 1 through 5 indicating which ship is placed in which cell of the board and 0 for water. In a sense the board is the same as a 10 by 10 grayscale image which I believe this model will be able to handle.
Could you provide an example using grayscale images (preferably size 10x10)?
@MathiasGruber
when I wrote following code:
`
import os
from copy import deepcopy
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
import cv2
if os.path.basename(os.getcwd()) != 'PConv-Keras-new':
os.chdir('..')
from libs.pconv_model import PConvUnet
from libs.util import MaskGenerator, ImageChunker
from libs.pconv_model import PConvUnet
model = PConvUnet(vgg_weights=None, inference_only=True)
model.load(r"D:\artAI\model\weights.26-1.07.h5", train_bn=False)
sample_image_file = os.path.dirname(os.path.realpath(file)) + '\xxt\my_bike_xxt03.jpg'
mask_image_file = os.path.dirname(os.path.realpath(file)) + '\xxt\xxt03.jpg'
img = cv2.imread(sample_image_file)
mask = cv2.imread(mask_image_file)
img = np.array(img)/255
mask = np.array(mask)/255
np.random.seed(16)
chunker = ImageChunker(512, 512, 30)
chunked_images = chunker.dimension_preprocess(deepcopy(img))
chunked_masks = chunker.dimension_preprocess(deepcopy(mask))
pred_imgs = model.predict([chunked_images, chunked_masks])
reconstructed_image = chunker.dimension_postprocess(pred_imgs, img)
reconstructed_image = np.array(reconstructed_image)*255
file_name = 'result{}.jpg'.format(np.random.randint(1,100))
cv2.imwrite(file_name, reconstructed_image)
`
my_bike_xxt03.jpg:
xxt03.jpg:
but the result is:
but when i try https://www.fixmyphoto.ai/, it's correct.Why?
thanks lot
@MathiasGruber Hi, I've used your code to get pretty decent results but when ever there is a lighter background in image such as sky , some noise is added as a result. But your deployed model on www.fixmyphoto.ai works fine. is there something missing in this code base ?
Hey :)
Thanks for sharing this nice code. I was just wondering how long an epoch takes for you... (Would be very helpful to know)
@MathiasGruber Thank you very much for your excellent code! I have come up against some problems when running your code in Step 3, about training UNet on single image. I've downloaded your pretrained VGG weights and put it in model. However, I found the loss decrease very slow, and the predicted image seems not right.
Epoch 1/10
2000/2000 [==============================] - 1239s 619ms/step - loss: 5.1134 - PSNR: 11.9789
Epoch 2/10
2000/2000 [==============================] - 1226s 613ms/step - loss: 2.8002 - PSNR: 16.5682
Epoch 3/10
2000/2000 [==============================] - 1231s 616ms/step - loss: 2.2371 - PSNR: 19.1468
Is there some wrong with my implementation?
I'm trying to run "Step5 - Prediction.ipynb
" but it fails in first step:
ImportError Traceback (most recent call last)
<ipython-input-1-86b0b96f5fd9> in <module>
14
15 from libs.pconv_model import PConvUnet
---> 16 from libs.util import random_mask, ImageChunker
17
18 print(os.getcwd())
ImportError: cannot import name 'random_mask'
Hi, due to the quite long training time, I want to know how can I use the keras.utils.multi_gpu_model?
Hi! After successfully running Creating train & test data generator, the third image shown in the result is the original image?
hi, thanks for your work. i note that the code generates a random irregular mask with lines, circles and elipses. I wonder do you follow the method of generating masks in the original paper?
Hello. How can I convert your script to mlmodel file to use in the ios app?(swift)
I try to use this tutorial but I need your help please.
Not sure if it would be much of an optimization, but I noticed the current code produces masked versions of the images for training, and I'm not sure it is a needed step.
Since the partial convolution will use the mask to zero out any contribution from masked pixels, would there be any impact to using the ground truth image? And if it is possible, would there be any benefit?
hey, thanks for your codes.
Do you know about the partial convolutional layer in the paper? I have seen your architecture image showed in Readme, but I can't explain that an image which size is 5125123, and is input into a partial layer ,and the output's size is 25625664.The kernel is 77,and the strides is 2.I don't get why 512512 comes to 256*256? Do you have any idea? Thanks!
Hi! I'm trying to train the model on my dataset in notebook #4 with pictures of size 288x288. I have changed image sizes here:
`
train_datagen = AugmentingDataGenerator(
rotation_range=10,
width_shift_range=0.1,
height_shift_range=0.1,
rescale=1./255,
horizontal_flip=True
)
train_generator = train_datagen.flow_from_directory(
TRAIN_DIR,
MaskGenerator(288, 288, 3),
target_size=(288, 288),
batch_size=BATCH_SIZE
)
val_datagen = AugmentingDataGenerator(rescale=1./255)
val_generator = val_datagen.flow_from_directory(
VAL_DIR,
MaskGenerator(288, 288, 3),
target_size=(288, 288),
batch_size=20,
classes=['val'],
seed=42
)
test_datagen = AugmentingDataGenerator(rescale=1./255)
test_generator = test_datagen.flow_from_directory(
TEST_DIR,
MaskGenerator(288, 288, 3),
target_size=(288, 288),
batch_size=20,
seed=42
)`
but receiving the following error:
ValueError: Error when checking input: expected inputs_img to have shape (512, 512, 3) but got array with shape (288, 288, 3)
in cell #44 when starting to train.
Is there a way to train on a different size or only 512x512?
Hi MathiasGruber,
Firstly I am kindly to thank you for your awesome contribution, especially making them as step by step process.
I have went through the source code, and as you have said that it is going to take time to train the model, could you provide your pretrained weights such as these files:
50_weights_2018-06-01-16-41-43.h5
150_weights_2018-06-26-22-19-32.h5
170_weights_2018-06-28-15-00-38.h5
in "Step4 - Imagenet Training" file
Thank you again 👍 🥇
In the Paper, authors use the sum(M) means the sum of the conv slide window binary mask to make sure that do not use the hole pixels.
But i read your code , do you use the mean value of the whole mask? If so , you use the hole pixels' values
Hi MathiasGruber,
(and Hello guys)
At the training step, you have used 512x512 pixels images.
How can we test free size images ( lets say 640x640 pixels images)?
Thank you.
What is the use of ‘pconv_imagenet.h5’ .How do I add a mask to my training
Hi,
thank you for sharing this code. It is rather a general question than an issue.
Do not you think you over-fit if you randomly generate masks on the same images and it happens that with future iterations the net eventually will see the whole image?
I am just curios how you dealt with this problem.
PConv-Keras/libs/pconv_model.py
Line 264 in 7c8f2cb
Rebuilding the model without the init options gives error when training on multi GPU.
Error traceback:
Traceback (most recent call last):
File "train_pconvunet.py", line 270, in <module>
model.load(args.checkpoint)
File "/home/ubuntu/image-inpainting/libs/pconv_model.py", line 307, in load
self.model.load_weights(filepath)
File "/home/ubuntu/image-inpainting/env/lib/python3.6/site-packages/keras/engine/topology.py", line 2667, in load_weights
f, self.layers, reshape=reshape)
File "/home/ubuntu/image-inpainting/env/lib/python3.6/site-packages/keras/engine/topology.py", line 3365, in load_weights_from_hdf5_group
str(len(filtered_layers)) + ' layers.')
ValueError: You are trying to load a weight file containing 1 layers into a model with 31 layers.
Commenting the rebuilding and compilation statements in the load function works however.
I want to know the range of loss when training, thx
I try to fill image with holes on https://www.fixmyphoto.ai/ and it performs well. However when I try the code in notebook (including step 4 and 5), the performance can not keep up with it. Hence, I wonder the workflow between them is identical? I use the released pre-trained imagenet model.
My confusion was solved after Mathias confirms all the workflow are identical.
when i run:
python main.py --name MyDataset --train C:\Documents\Kaggle\Kaggle-imagenet\ILSVRC\Data\CLS-LOC\train --validation C:\Documents\Kaggle\Kaggle-imagenet\ILSVRC\Data\CLS-LOC -test C:\Documents\Kaggle\pexels --vgg_path 'D:\artAI\model\pytorch_to_keras_vgg16.h5'
error is ocurred:
2019-04-23 14:44:49.128950: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 153363456 totalling 146.26MiB
2019-04-23 14:44:49.134231: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 201326592 totalling 960.00MiB
2019-04-23 14:44:49.138279: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 212413696 totalling 405.15MiB
2019-04-23 14:44:49.148331: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 4.56GiB
2019-04-23 14:44:49.151895: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 4949881651
InUse: 4894695424
MaxInUse: 4895090944
NumAllocs: 1693
MaxAllocSize: 2529192960
2019-04-23 14:44:49.162031: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ****************************************************************************************************
2019-04-23 14:44:49.168876: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at conv_ops.cc:746 : Resource exhausted: OOM when allocating tensor with shape[3,256,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "c:\Users\flyon.vscode\extensions\ms-python.python-2019.3.6558\pythonFiles\ptvsd_launcher.py", line 45, in
main(ptvsdArgs)
File "c:\Users\flyon.vscode\extensions\ms-python.python-2019.3.6558\pythonFiles\lib\python\ptvsd_main_.py", line 391, in main
run()
File "c:\Users\flyon.vscode\extensions\ms-python.python-2019.3.6558\pythonFiles\lib\python\ptvsd_main_.py", line 272, in run_file
runpy.run_path(target, run_name='main')
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "d:\artAI\PConv-Keras\main.py", line 240, in
TQDMCallback()
File "d:\artAI\PConv-Keras\libs\pconv_model.py", line 254, in fit_generator
*args, **kwargs
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\keras\engine\training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\keras\engine\training_generator.py", line 217, in fit_generator
class_weight=class_weight)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch
outputs = self.train_function(ins)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in call
return self._call(inputs)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\tensorflow\python\client\session.py", line 1439, in call
run_metadata_ptr)
File "C:\Users\flyon\AppData\Local\conda\conda\envs\artai\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,256,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node loss_1/outputs_img_loss/model_1_2/vgg16/block3_conv3/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@train...kpropInput"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](loss_1/outputs_img_loss/model_1_2/vgg16/block3_conv2/Relu, block3_conv3/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node loss_1/mul/_1041}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_8425_loss_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
I'm trying to run (from step 5 ipynb):
from libs.pconv_model import PConvUnet
model = PConvUnet(vgg_weights=None, inference_only=True)
model.load(r"/media/user/PConv-Keras/pconv_imagenet.h5", train_bn=False)
But it fails with ValueError: invalid literal for int() with base 10: 'h5'
error:
ValueError Traceback (most recent call last)
<ipython-input-11-adc04f0f404a> in <module>
1 from libs.pconv_model import PConvUnet
2 model = PConvUnet(vgg_weights=None, inference_only=True)
----> 3 model.load(r"/media/user/PConv-Keras//pconv_imagenet.h5", train_bn=False)
/media/user/PConv-Keras/libs/pconv_model.py in load(self, filepath, train_bn, lr)
266
267 # Load weights into model
--> 268 epoch = int(os.path.basename(filepath).split('.')[1].split('-')[0])
269 assert epoch > 0, "Could not parse weight file. Should include the epoch"
270 self.current_epoch = epoch
ValueError: invalid literal for int() with base 10: 'h5'
Hi, i want to train my own dataset, and after that I try to set up a page to interactively put mask on image and get result. can you share your code of fixmyphoto page or would you please tell me how to do it? thanks.
You are using VGG16 with imagenet weights, should it using preprocess_input to computing VGG features?
img = Input(shape=(self.img_rows, self.img_cols, 3))
# Get the vgg network from Keras applications
processed=Lambda(preprocess_input, name='Input_Image', input_shape=(self.img_rows, self.img_cols, 3))(img)
vgg = VGG16(weights="imagenet", include_top=False)
# Output the first three pooling layers
vgg.outputs = [vgg.layers[i].output for i in self.vgg_layers]
# Create model and compile
model = Model(inputs=img, outputs=vgg(processed))
Is the model provided in the google drive on imagenet the same as the one used for the web UI(www.fixmyphoto.ai)? If yes, then is the preprocessing used in Step5 notebook similar to the one used in UI. because not getting results as good as the UI .
Hi,
I have a question regarding your implementation of partial convolution layer. Your implementation employs mean normalization instead of sum normalization as it is in the original paper. Could you explain the intuition behind that change?
Currently, I am creating a model using images in my house for learning images.
The original model has not changed.
When predicting images, it becomes white like the image below.
I changed the loss function L6 (loss_tv) from 0.1 to 1, but it was ineffective.
libs/pconv_model.py
in loss_total func
return l1 + 6*l2 + 0.05*l3 + 120*(l4+l5) + 1.0*l6
mask image
In order to avoid recognizing it as a mask when there is a pure white color in the original image, the whole is grayed out.
model_rootpath = '/mnt/PConv-Keras/'
model = PConvUnet(weight_filepath='data/model/')
model.load(
'{}/01/weight/13_weights_2019-03-09-00-20-20.h5'.format(model_rootpath),
train_bn=False,
lr=0.00005
)
SAVE_IMG_ROOTPATH = '/mnt/PConv-Keras/predicted_imgs'
CASE_NAME = 'CASE-01'
GPU_SIZE = 'GPU-1'
BATCH_SIZE = 'Batch-07'
EPOCH = 'Epoch-13'
LEARNING_TIME = 'Time-200h'
# Create dir
# ex) CASE-01_GPU-4_Batch-27_Epoch-37_Time-46h
save_filename = ('_').join((CASE_NAME, GPU_SIZE, BATCH_SIZE, EPOCH, LEARNING_TIME))
dir_name = datetime.now().strftime('%Y%m%d_%H%M%S')
save_dir_path = SAVE_IMG_ROOTPATH + '/' + dir_name
os.makedirs(save_dir_path)
start = time.time()
for i in range(6):
# Load masked image and Create mask image(black and white)
masked_img = cv2.imread('{0}/input_imgs/img_{1:02d}_mask.jpg'.format(SAVE_IMG_ROOTPATH, i))
masked_img = cv2.resize(masked_img, (cst.IMG_WIDTH, cst.IMG_HEIGHT))
masked_img = cv2.cvtColor(masked_img, cv2.COLOR_BGR2RGB)
# Create mask image
mask = np.concatenate([((np.sum(masked_img, axis = -1) > 200 * 3) * 255)[..., np.newaxis]] * 3, axis = -1)
mask_inv_img = 255 - mask
# Load original color image
ori_img = cv2.imread('{0}/input_imgs/img_{1:02d}_ori.jpg'.format(SAVE_IMG_ROOTPATH, i))
ori_img = cv2.resize(ori_img, (cst.IMG_WIDTH, cst.IMG_HEIGHT))
ori_img = cv2.cvtColor(ori_img, cv2.COLOR_BGR2RGB)
# Resize for prediction
model_input_img_tensor = ori_img[np.newaxis, ...]/255.
model_input_mask_tensor = mask_inv_img[np.newaxis, ...]/255
# Prediction
model_output = model.predict([model_input_img_tensor, model_input_mask_tensor])
save_img = Image.fromarray(np.uint8((model_output[0,:,:,:] * 1.)*255))
save_img.save('{0}/{1}_{2:02d}.jpg'.format(save_dir_path, save_filename, i))
I got a question. when input shape is (None,1,512,1) and kernel_size and stride change with (1,*) the pconv output_shape is (None,1,255,1) and conv2d is (None,1,256,1).
In Notebook 5, the model is loaded by using:
model.load(r"C:\Users\Mathias Felix Gruber\Documents\GitHub\PConv-Keras\data\logs\imagenet_phase2\weights.26-1.07.h5", train_bn=False)
But where can I download the trained model? I only want to run the test and do not want to perform training.
@MathiasGruber I am trying to test your code on my own masks. I have made the necessary changes. My doubt is what color masks does the code expect. Should the masked region be red (like in your demo) or white like shown below??
Similarly should the mask be black-over-white or white-over-black??
Traceback (most recent call last):
File "/home/irlts/nianyi/PConv-Keras-master/main.py", line 210, in
model.load(args.checkpoint)
File "/home/irlts/nianyi/PConv-Keras-master/libs/pconv_model.py", line 271, in load
self.model.load_weights(filepath)
File "/localscratch/irlts.21223938.0/env/lib/python3.7/site-packages/keras/engine/saving.py", line 492, in load_wrapper
return load_function(*args, **kwargs)
File "/localscratch/irlts.21223938.0/env/lib/python3.7/site-packages/keras/engine/network.py", line 1230, in load_weights
f, self.layers, reshape=reshape)
File "/localscratch/irlts.21223938.0/env/lib/python3.7/site-packages/keras/engine/saving.py", line 1235, in load_weights_from_hdf5_group
' elements.')
ValueError: Layer #0 (named "p_conv2d_17" in the current model) was found to correspond to layer p_conv2d_49 in the save file. However the new layer p_conv2d_17 expects 3 weights, but the saved weights have 2 elements.
Found 0 images belonging to 0 classes.
Traceback (most recent call last):
File "/home//PConv-Keras-master/main.py", line 180, in
test_data = next(test_generator)
File "/home//PConv-Keras-master/main.py", line 122, in flow_from_directory
for _ in range(ori.shape[0])], axis=0
File "/home/zxf/anaconda3/envs/tensorflow/lib/python3.6/site-packages/numpy/core/shape_base.py", line 412, in stack
raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack
excuse me . I follow the comand python main.py --name celeba --train ./data/celeba/train --validation ./data/celeba/val --test ./data/celeba/test --vgg_path './data/logs/pytorch_to_keras_vgg16.h5'
but there is no found files about mask and image, why?? i put masks dataset under ./data/masks/train/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.