lanha / dsen2 Goto Github PK

Hi

I've tested the pre-trained model on quite a few regions and has been performing really well, however, I have few questions here

1. What kind of improvements can we expect by increasing training size for a specific area?
2. Can your training code be used push 10m further to 5m? Using the same approach by downscaling 10m to 20m?
3. In what scenarios is this model not good for or may not give expected results?

Thanks

Sorry，I want ask you if you can let us download your test results(RGB images).Because you test on all 15 figures but not all presented on your paper.Sorry to disturb you TT

I read the paper with excitement but I'm having difficulty installing DSen2. Can we please have some installation instructions? I can't find the setup.py file so even a pip install from a local file doesn't work for me. But I'm not that experienced in python. I usually just use pip or conda installs so perhaps I am missing something.

Thanks
Patrice

Hello,

I started playing with the models you provided for Super-Resolution, first trying to reproduce the results from the article Super-resolution of Sentinel-2 images: Learning a globally applicable deep neural
network about the method.
The approach is the following : I took 4 images of size (110 `km)^2 directly from the testing set shared in the git repository (
London : S2A_MSIL1C_20170522T110621_N0205_R137_T30UXC_20170522T110912
El Salvador : S2A_MSIL1C_20170419T155901_N0204_R097_T16PEU_20170419T161354
Japan City : S2A_MSIL1C_20170216T015741_N0204_R060_T52SFB_20170216T015924
NZ mountains : S2A_MSIL1C_20170616T223701_N0205_R072_T59GLL_20170616T223702 )

For the T_{2x} model, I downsampled the 10 and 20m GDS bands (A and B bands) to respectively 20 and 40m GSD, and then I applied the DSen2_20 function in your code to these (which corresponds to the inference with the trained model) to get the B bands super resoluted. Then, I computed the band wise root mean squared error between the super resoluted (20m GSD) B bands and the Ground truth 20m GSD B bands. I did the analog for the S_{6x} model, as described in the paper.

As a result, I obtained this : (in orange : the RMSE for Bicubic and in blue, the RMSE for DSen2)

.The DSen2 model performs slightly better than the bicubic interpolation (at best 1.1 lower RMSE than Bicubic). My problem is that I cannot get the same results as in the paper (3 to 4 times lower RMSE than Bicubic).
Is there something I missed in the pipeline ?

Also, in the Demo code, you compare the super-resoluted B bands with a ground truth called imGT. Is it a 10m GSD ground truth (in that case, is there a specific way to get this resolution ground truth for the B bands ?), or all the input images (im10 and im20) have been downsampled before ?

I run the following line

%run demoDSen2.py but got the following error as below. Please help.

---------------------------------------------------------------------------
UnimplementedError                        Traceback (most recent call last)
File ~\OneDrive - University of Canterbury\DSen2\testing\demoDSen2.py:47
     45 print('Siberia')
     46 im10, im20, imGT = readh5('S2B_MSIL1C_20170725_T43WFQ.mat', imGT=True)
---> 47 SR20 = DSen2_20(im10, im20)
     48 # Evaluation against the ground truth on the 20m resolution bands (simulated)
     49 print('DSen2:')

File ~\OneDrive - University of Canterbury\DSen2\testing\supres.py:27, in DSen2_20(d10, d20, deep)
     25 test = [p10, p20]
     26 input_shape = ((4, None, None), (6, None, None))
---> 27 prediction = _predict(test, input_shape, deep=deep)
     28 images = recompose_images(prediction, border=border, size=d10.shape)
     29 images *= SCALE

File ~\OneDrive - University of Canterbury\DSen2\testing\supres.py:65, in _predict(test, input_shape, deep, run_60)
     63 model.load_weights(predict_file)
     64 print("Predicting using file: {}".format(predict_file))
---> 65 prediction = model.predict(test, verbose=1)
     66 return prediction

File ~\AppData\Local\anaconda3\Lib\site-packages\keras\src\utils\traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File ~\AppData\Roaming\Python\Python311\site-packages\tensorflow\python\eager\execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     51 try:
     52   ctx.ensure_initialized()
---> 53   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     54                                       inputs, attrs, num_outputs)
     55 except core._NotOkStatusException as e:
     56   if name is not None:

UnimplementedError: Graph execution error:

Detected at node 'model_1/conv2d_14/Conv2D' defined at (most recent call last):
    File "<frozen runpy>", line 198, in _run_module_as_main
    File "<frozen runpy>", line 88, in _run_code
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel_launcher.py", line 17, in <module>
      app.launch_new_instance()
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\traitlets\config\application.py", line 992, in launch_instance
      app.start()
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel\kernelapp.py", line 711, in start
      self.io_loop.start()
    File "C:\Users\gji19\AppData\Roaming\Python\Python311\site-packages\tornado\platform\asyncio.py", line 195, in start
      self.asyncio_loop.run_forever()
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\asyncio\base_events.py", line 607, in run_forever
      self._run_once()
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\asyncio\base_events.py", line 1922, in _run_once
      handle._run()
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\asyncio\events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel\kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel\kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel\kernelbase.py", line 406, in dispatch_shell
      await result
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel\kernelbase.py", line 729, in execute_request
      reply_content = await reply_content
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel\ipkernel.py", line 411, in do_execute
      res = shell.run_cell(
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\ipykernel\zmqshell.py", line 531, in run_cell
      return super().run_cell(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py", line 3006, in run_cell
      result = self._run_cell(
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py", line 3061, in _run_cell
      result = runner(coro)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py", line 3266, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py", line 3445, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py", line 3505, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "C:\Users\gji19\AppData\Local\Temp\ipykernel_22004\3374343633.py", line 1, in <module>
      get_ipython().run_line_magic('run', 'demoDSen2.py')
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py", line 2414, in run_line_magic
      result = fn(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\magics\execution.py", line 834, in run
      run()
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\magics\execution.py", line 819, in run
      runner(filename, prog_ns, prog_ns,
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\pylabtools.py", line 213, in mpl_execfile
      safe_execfile(fname,*where,**kw)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py", line 2863, in safe_execfile
      py3compat.execfile(
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\IPython\utils\py3compat.py", line 55, in execfile
      exec(compiler(f.read(), fname, "exec"), glob, loc)
    File "C:\Users\gji19\OneDrive - University of Canterbury\DSen2\testing\demoDSen2.py", line 47, in <module>
      SR20 = DSen2_20(im10, im20)
    File "C:\Users\gji19\OneDrive - University of Canterbury\DSen2\testing\supres.py", line 27, in DSen2_20
      prediction = _predict(test, input_shape, deep=deep)
    File "C:\Users\gji19\OneDrive - University of Canterbury\DSen2\testing\supres.py", line 65, in _predict
      prediction = model.predict(test, verbose=1)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\training.py", line 2554, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\training.py", line 2341, in predict_function
      return step_function(self, iterator)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\training.py", line 2327, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\training.py", line 2315, in run_step
      outputs = model.predict_step(data)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\training.py", line 2283, in predict_step
      return self(x, training=False)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\training.py", line 569, in __call__
      return super().__call__(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\base_layer.py", line 1150, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\utils\traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\functional.py", line 512, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\functional.py", line 669, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\engine\base_layer.py", line 1150, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\utils\traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\layers\convolutional\base_conv.py", line 290, in call
      outputs = self.convolution_op(inputs, self.kernel)
    File "C:\Users\gji19\AppData\Local\anaconda3\Lib\site-packages\keras\src\layers\convolutional\base_conv.py", line 262, in convolution_op
      return tf.nn.convolution(
Node: 'model_1/conv2d_14/Conv2D'
The Conv2D op currently only supports the NHWC tensor format on the CPU. The op was given the format: NCHW
	 [[{{node model_1/conv2d_14/Conv2D}}]] [Op:__inference_predict_function_1337]

Hello sir, i couldn't find the super resolved band 10 from the output of sr60

I can require RGB10 and RGB20 using create_patches.py but how can I get RGB60?

Nice to meet you, I am a master course student from South Korea, and my major is Landscape Architecture. These days, I am trying to use your code and paper to make super resolution Sentinel-2 imagery.

My question is that, if my imagery has :
Super resolution (about 5m?) R,G,B band and
NIR is just 10 m,

Then, could your code work well with the upper configuration?
Because original Sentinel-2 has 10m RGB and NIR. And other bands are low resolution.
But I have only RGB high resolution band and NIR is not that high resolution compared to RGB.

Many thanks for your nice work!
I will wait for your reply :)

I have plenty of Sentinel-2 data need to be loaded , meanwhile, I want to read the whole image of all bands Sentinel-2 data into Keras in a batch file directly. Can you provide the related code or give some suggestions?
Thank you for your attention.

Hi,

How should model training be done for a particular region? Should I only use cloudless or minimal cloud images?

Can sentinel 2 level 2A datasets be used in place of level 1C datasets without changing the code?

Thanks in advance for providing DSen2 source code.

How would you recommend applying the pretrained DSen2 model to images with more than 6 LR bands? I've tried just forcing it through, but get a shape incompatibility for obvious reasons.

The program executes fine, no problems at a glance. However, the image generated (.tif) is simply a black image of size: 165MB. What is happening? How do I get this to correctly work?

`(c:\dsen2) C:\DSen2\DSen2-master\testing>python s2_tiles_supres.py C:\DSen2\DSen2-master\output\S2A_MSIL1C_20170216T015741_N0204_R060_T52SFB_20170216T015924.SAFE\MTD_MSIL1C.xml C:\DSen2\DSen2-master\output\output.tif --roi_x_y "100,100,2000,2000"
2021-06-27 21:54:15.211953: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
Selected UTM Zone: UTM 52N
Selected pixel region: xmin=96, ymin=96, xmax=1997, ymax=1997:
Image size: width=1902 x height=1902
Selected 10m bands: B4 B3 B2 B8
Selected 20m bands: B5 B6 B7 B8A B11 B12
Selected 60m bands:
Loading selected data from: Bands B2, B3, B4, B8 with 10m resolution, UTM 52N
Loading selected data from: Bands B5, B6, B7, B8A, B11, B12 with 20m resolution, UTM 52N
Super-resolving the 20m data into 10m bands
2021-06-27 21:54:21.558217: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-06-27 21:54:21.579561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:03:00.0 name: NVIDIA GeForce GTX 970 computeCapability: 5.2
coreClock: 1.253GHz coreCount: 13 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 208.91GiB/s
2021-06-27 21:54:21.579748: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-27 21:54:21.589145: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-27 21:54:21.590339: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-27 21:54:21.594958: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-06-27 21:54:21.596431: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-06-27 21:54:21.599532: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-06-27 21:54:21.604102: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-06-27 21:54:21.604948: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-27 21:54:21.605083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-27 21:54:21.605896: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-27 21:54:21.606660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:03:00.0 name: NVIDIA GeForce GTX 970 computeCapability: 5.2
coreClock: 1.253GHz coreCount: 13 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 208.91GiB/s
2021-06-27 21:54:21.608467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-27 21:54:22.067331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-27 21:54:22.067438: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-06-27 21:54:22.069050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-06-27 21:54:22.069858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2776 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 970, pci bus id: 0000:03:00.0, compute capability: 5.2)
Symbolic Model Created.
Predicting using file: ../models/s2_032_lr_1e-04.hdf5
2021-06-27 21:54:22.588953: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-27 21:54:28.943358: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-27 21:54:29.218264: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8100
2021-06-27 21:54:29.838396: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-27 21:54:30.161958: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-27 21:54:30.325501: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-27 21:54:30.593855: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-27 21:54:31.153012: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
10/10 [==============================] - 12s 397ms/step
(6, 1902, 1902)
Writing the super-resolved bands in C:\DSen2\DSen2-master\output\output.tif
SRB5 (705 nm)
SRB6 (740 nm)
SRB7 (783 nm)
SRB8A (865 nm)
SRB11 (1610 nm)
SRB12 (2190 nm)

(c:\dsen2) C:\DSen2\DSen2-master\testing>`

There is no information in the repository about license of the code and models. Please add a LICENSE or COPYING file, without clear license it is not possible to (re)use code. If possible it would be really nice to have some free license line GPL or MIT.

Hi Charis,

DSen2 is doing a great job. Thanks a lot.

Could you also provide a version of the code that allows for the processing of Sentinel-2 Level 2A Data?

Currently I run into problems because the sen2cor plugin (SNAP) does not accept the data after they have been processed with DSen2 and vice versa DSen2 does not accept the data which have been processed with sen2cor. But eventually, I need data that have been atmospherically corrected with sen2cor and afterwards sharpened with DSen2.

If you could work on this, I would be very happy.

Best regards!

More of a question; not really an issue.

Hi, I just read your paper and found your approach and results very interesting!

I notice that you say the model weights will be added on March, 31, and I was just wondering if you will be adding workflow code as well? It would be greatly appreciated and experimented with

Hi,there are some problems with matlab 2017b demo codes.
DAGNetwork>iCreateDataDispatcher (line 651)
For an image input layer, the input data for predict must be a single image, a 4D array of images, or an imageDatastore with the correct size.
DAGNetwork>iCreateDataDispatchersForEachInputLayer (line 642)
dispatchers = { iCreateDataDispatcher(X, miniBatchSize, precision) };
DAGNetwork/predict (line 389)
dispatchers = iCreateDataDispatchersForEachInputLayer(X, miniBatchSize, precision);
DSen2 (line 44)
pred = predict(net20, q_image/2000)*2000;
demoDSen2 (line 26)
SR20 = DSen2(im10, im20);
Can you help me ?

Hello, First of all, thanks for the impressive work.
I was trying to use the Colab version for testing and I have issues regarding the super-resolution of 60m bands.
I use the model with run60=True and pass each band with the corresponding order:
For 10m: (B2,B3,B4,B8), 20m:(B5,B6,B7,B8a,B11,B12) and 60m:(B1,B9).

The example below is one crop from the list of the test images. I believe there is something i am missing here to have the performance with SR60. What could I be doing wrong?

Thanks in advance!

Hello,buddy,i just start to learn your DSen2 source code. If you could show me how to generate the imGT? Thank you!

Hi Lanha,

I test your trained model with RMSE,SRE,SAM, UIQ metrics.
But the UIQ and SAM values are not same as the results in your paper.
I use the UIQ and SAM functions that are provied by sewar (one python package).

Hi Lanaras,

Sorry for writing here but I am trying to send you an email at [email protected] but its unreachable, can you provide working email address, please.

Thanks.

It always fails at UnboundLocalError: local variable 'data_path' referenced before assignment

(base) D:\digisat\DSen2\testing>python .\s2_tiles_supres.py D:\digisat\S2B_MSIL1C_20211021T052849_N0301_R105_T43RGM_20211021T072442.zip --clip_to_aoi "100,100,2000,2000"
2021-10-24 15:59:33.631479: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-10-24 15:59:34.981411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2149 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5
Traceback (most recent call last):
  File ".\s2_tiles_supres.py", line 213, in <module>
    args.data_file_path, args.clip_to_aoi, args.copy_original_bands, args.output_dir
  File ".\s2_tiles_supres.py", line 178, in process
    data10, data20, data60, coord, pr_10m = self.start()
  File ".\s2_tiles_supres.py", line 73, in start
    data_list = self.get_data()
  File "..\utils\data_utils.py", line 65, in get_data
    LOGGER.info(f"Data path is {data_path}")
UnboundLocalError: local variable 'data_path' referenced before assignment

I'm trying to pass custom bounds for a complete sentinel-2A tile and also tried without passing the rio param as I want to run on full tile, however it is failing with this error:

2020-10-27 19:50:46.779614: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_110.dll
Selected UTM Zone: UTM 11N
Selected pixel region: xmin=96, ymin=96, xmax=9995, ymax=9995:
Image size: width=9900 x height=9900
Selected 10m bands: B4 B3 B2 B8
Selected 20m bands: B5 B6 B7 B8A B11 B12
Selected 60m bands: B1 B9
Loading selected data from: Bands B2, B3, B4, B8 with 10m resolution, UTM 11N
Loading selected data from: Bands B5, B6, B7, B8A, B11, B12, AOT, CLD, SCL, SNW, WVP with 20m resolution, UTM 11N
ERROR 1: Stream too short

ERROR 1: opj_get_decoded_tile() failed
ERROR 1: Stream too short

ERROR 1: Stream too short

ERROR 1: Stream too short

ERROR 1: Stream too short

ERROR 1: Stream too short

ERROR 1: opj_get_decoded_tile() failed
ERROR 1: opj_get_decoded_tile() failed
ERROR 1: opj_get_decoded_tile() failed
ERROR 1: opj_get_decoded_tile() failed
ERROR 1: opj_get_decoded_tile() failed
Traceback (most recent call last):
  File ".\s2_tiles_supres.py", line 319, in <module>
    data20 = np.rollaxis(
  File "<__array_function__ internals>", line 5, in rollaxis
  File "C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\numeric.py", line 1259, in rollaxis
    n = a.ndim
AttributeError: 'NoneType' object has no attribute 'ndim'

Thanks for making this public and the nice documentation! It works very smoothly. This is a request, rather than an actual issue. I'm processing lots of S2 data - thousands of tiles. I'd really like to integrate this into the processing stream however its taking too long to process given the computing power I have available - I've tested on both cpu and a decent gpu. It would be really nice, and highly useful, if you could have the option to sharpen just a single 20m band, or a more refined selection of bands. Would it be possible to add this functionality?

hey, I am trying to understand your code but stuck on function called get_test_patches60. I am unable to understand the purpose of this function. Can you please guide me on why you used this function.

Thanks alot.

Hi Charis,

I found an error when trying to predict non-square portions of an image (for example --roi_x_y "500,1500,2500,2500").

ignacio@houdini:~/satelites/super_resolution/DSen2/testing$ python s2_tiles_supres.py /media/ignacio/Datos/datasets/satelites/S2_tiles/S2A_MSIL1C_20170608T105651_N0205_R094_T30TWM_20170608T110453.SAFE/MTD_MSIL1C.xml output_file.tif --roi_x_y "500,1500,2500,2500" --copy_original_bands --run_60
Using TensorFlow backend.
Selected UTM Zone: UTM 30N
Selected pixel region: xmin=498, ymin=1500, xmax=2495, ymax=2495:
Image size: width=1998 x height=996
Selected 10m bands: B4 B3 B2 B8
Selected 20m bands: B5 B6 B7 B8A B11 B12
Selected 60m bands: B1 B9
Loading selected data from: Bands B2, B3, B4, B8 with 10m resolution, UTM 30N
Loading selected data from: Bands B5, B6, B7, B8A, B11, B12 with 20m resolution, UTM 30N
Loading selected data from: Bands B1, B9, B10 with 60m resolution, UTM 30N
Super-resolving the 60m data into 10m bands
Symbolic Model Created.
2019-02-11 12:12:07.425593: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-11 12:12:07.509926: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-11 12:12:07.510305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.8225
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.29GiB
2019-02-11 12:12:07.510320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-11 12:12:07.708667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-11 12:12:07.708698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-02-11 12:12:07.708704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-02-11 12:12:07.708898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7040 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Predicting using file: ../models/s2_030_lr_1e-05.hdf5
72/72 [==============================] - 2s     
(2, 1998, 996)
Super-resolving the 20m data into 10m bands
Symbolic Model Created.
Predicting using file: ../models/s2_032_lr_1e-04.hdf5
162/162 [==============================] - 1s     
(6, 1998, 996)
Writing the original 10m bands and the super-resolved bands in output_file.tif
Traceback (most recent call last):
  File "s2_tiles_supres.py", line 407, in <module>
    write_band_data(sr[:, :, bi], "SR" + validated_descriptions[bn], "SR" + bn)
  File "s2_tiles_supres.py", line 380, in write_band_data
    result_dataset.GetRasterBand(bidx).WriteArray(data)
  File "/home/ignacio/anaconda3/lib/python3.6/site-packages/osgeo/gdal.py", line 2623, in WriteArray
    callback_data = callback_data )
  File "/home/ignacio/anaconda3/lib/python3.6/site-packages/osgeo/gdal_array.py", line 378, in BandWriteArray
    raise ValueError("array larger than output file, or offset off edge")
ValueError: array larger than output file, or offset off edge

Looking into the issue it turns out that sr has the coordinates transposed, that is the correct output shape (the one of the data10 array) is (996, 1998, 4) while the shape of sr is (1998, 996, 6).

If I do a sr = np.moveaxis(sr, 0, 1) just after line 405, I can manage to get no errors and save the output image. But then the output image is completely messed out for some reason.

The point is that one shouldn't need to transpose sr because the code works well (the output image makes sense) with square images, but if one does not transpose sr then one cannot save the array because the dimensions are wrong. One solution that will preserve the good behaviour of square crops would be to use sr = sr.reshape(sr.shape[1], sr.shape[0], sr.shape[2]) but that does not seem to work either:

sr = sr.reshape(sr.shape[1], sr.shape[0], sr.shape[2], order='C')

sr = sr.reshape(sr.shape[1], sr.shape[0], sr.shape[2], order='F')

sr = sr.reshape(sr.shape[1], sr.shape[0], sr.shape[2], order='A')

Maybe the error comes from upstairs when the arrays are cropped or from the way the image is segmented and reassembled in patches to feed the DNN.

Have you already encountered this issue?

Hello,

When I run the file "demoDSen2.m" I have the following error:
"> In DSen2 (line 52)
Error using predict (line 84)
Invalid calling syntax for the "predict" command. Type "help
predict" for more information.

Error in DSen2 (line 53)
pred = predict(net60, q_image/2000)*2000;"

I use matlab R2017a and I have all the necessary toolboxes, do you think the matlab version might be the source of the problem? Would it be possible that you please check the version you deposited in Github just to make sure that the problem does not come from somewhere else?

Thank you in advance.
Best,

Hi Charis,

Thank you for the excellent work DSen2!
Yet the ethz and transfernow link for VDSen2 model are both expired. Is there any chance that you could refresh or update the links?

Thanks a lot!
Ming

I tried executing the following command and it generates an output.tif of 8 bytes but I'm unable to read it.

(image-sr-tf) navaneeth@mousebox:~/workspace/image-super-resolution/DSen2/testing$ python s2_tiles_supres.py /home/navaneeth/Documents/acads/sem7/btp/farm-pin-crop-detection-challenge-dataset/2017-01-01/S2A_MSIL1C_20170101T082332_N0204_R121_T34JEP_20170101T084543.SAFE/MTD_MSIL1C.xml output_file.tif --roi_x_y "100,100,2000,2000"
2020-11-13 14:24:36.412933: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Selected UTM Zone: UTM 34S
Selected pixel region: xmin=96, ymin=96, xmax=1997, ymax=1997:
Image size: width=1902 x height=1902
Selected 10m bands: B4 B3 B2 B8
Selected 20m bands: B5 B6 B7 B8A B11 B12
Selected 60m bands:
Loading selected data from: Bands B2, B3, B4, B8 with 10m resolution, UTM 34S
Loading selected data from: Bands B5, B6, B7, B8A, B11, B12 with 20m resolution, UTM 34S
Super-resolving the 20m data into 10m bands
2020-11-13 14:24:52.184602: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-11-13 14:24:52.259091: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:52.260615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 with Max-Q Design computeCapability: 6.1
coreClock: 1.2655GHz coreCount: 16 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2020-11-13 14:24:52.260718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-13 14:24:52.310705: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-13 14:24:52.339329: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-13 14:24:52.345411: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-13 14:24:52.396905: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-13 14:24:52.404364: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-13 14:24:52.485774: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-13 14:24:52.486194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:52.487896: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:52.489278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-13 14:24:52.489955: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-13 14:24:52.535152: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2199995000 Hz
2020-11-13 14:24:52.536521: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555c184b9e20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-13 14:24:52.536593: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-11-13 14:24:52.982174: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:52.983673: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555c18144840 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-11-13 14:24:52.983747: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1070 with Max-Q Design, Compute Capability 6.1
2020-11-13 14:24:52.985240: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:52.986439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 with Max-Q Design computeCapability: 6.1
coreClock: 1.2655GHz coreCount: 16 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2020-11-13 14:24:52.986554: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-13 14:24:52.986660: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-13 14:24:52.986743: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-13 14:24:52.986819: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-13 14:24:52.986894: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-13 14:24:52.986969: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-13 14:24:52.987044: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-13 14:24:52.987260: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:52.988569: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:52.989690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-13 14:24:52.990359: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-13 14:24:54.821179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 14:24:54.821250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-11-13 14:24:54.821271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-11-13 14:24:54.823144: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:54.824157: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 14:24:54.824989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7173 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1)
Symbolic Model Created.
Predicting using file: ../models/s2_032_lr_1e-04.hdf5
WARNING:tensorflow:AutoGraph could not transform <function Model.make_predict_function.<locals>.predict_function at 0x7f4bc1feb378> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: 'arguments' object has no attribute 'posonlyargs'
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2020-11-13 14:24:56.199285: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-13 14:24:58.658132: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
 1/10 [==>...........................] - ETA: 0sWARNING:tensorflow:Callbacks method `on_predict_batch_end` is slow compared to the batch time (batch time: 0.0111s vs `on_predict_batch_end` time: 0.3125s). Check your callbacks.
10/10 [==============================] - 3s 285ms/step
(6, 1902, 1902)
ERROR 4: Unable to open output_file.tif to obtain file list.
Writing the super-resolved bands in output_file.tif
SRB5 (705 nm)
SRB6 (740 nm)
SRB7 (783 nm)
SRB8A (865 nm)
SRB11 (1610 nm)
SRB12 (2190 nm)

Sorry to bother you,but I want to ask you some questions? Do
you train a 40→20 m network or a 20→10 network? From your paper I can see that you train a 40→20 m network and use original 10,20 data to test? Or you train a 20→10 network and use 20,40 data to test?

I;m trying the following line to test on S2A tiles:

python .\s2_tiles_supres.py --clip_to_aoi 100,100,5000,5000 D:\work\explore\convert-sentinel2-to-cogs\extract\S2A_MSIL2A_20200505T181921_N0214_R127_T11SPA_20200505T222608.SAFE\MTD_MSIL2A.xml

Error

Traceback (most recent call last):
  File ".\s2_tiles_supres.py", line 212, in <module>
    Superresolution(
  File ".\s2_tiles_supres.py", line 178, in process
    data10, data20, data60, coord, pr_10m = self.start()
  File ".\s2_tiles_supres.py", line 73, in start
    data_list = self.get_data()
  File "..\utils\data_utils.py", line 65, in get_data
    LOGGER.info(f"Data path is {data_path}")
UnboundLocalError: local variable 'data_path' referenced before assignment

Hi Charis,

I'm having some troubles running the program. I have it running locally where it runs fine. If I use the '--copy_original_bands' flag it runs out of memory, which I somewhat expect as I have 16Gb of RAM. If I exclude the flag it works.

However, when I run it on a Windows server it always runs out of memory, '--copy_original_bands' flag or not. It is in a VM and has 24Gb of RAM with nothing else running and doesn't finish the process.

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 3.59 GiB for an array with shape (9801, 6, 128, 128) and data type float32

I find it odd that it works on my PC with less RAM but when given more it doesn't work.

Hoping you can give me some ideas as to why this is happening or a potential solutions.

Regards,

Peter

Hi Charis,

Thank you for DSen2! I have just started playing with it and is working very well.

I would also like to test the VDSen2 model, but the download links do not appear to be working anymore. Is there any chance that you could refresh or update the links?

Thanks a lot!
Johannes