yym-sia / lingmi-mr Goto Github PK

Hello, thank you for your kind words. Regarding the Endoslam-dataset, there seems to be a misalignment between the provided depth maps and RGB images by the official source. [Unity RGB & Depthmaps misalignment (url)
However, I noticed in your paper that the RGB images and depth maps appear to be aligned. Could you please provide insights into the processing steps you employed to achieve this alignment? I look forward to your response.

Thank you for your sharing. When I was training ,there is some errors . Following is my result.

$ python train_depth.py --config configs/blender_train.json
D:\python\lib\site-packages\torchvision\models_utils.py:135: UserWarning: Using 'weights' as positional parameter(s) is deprecated since 0.13 and may be removed in the future. Please use keyword parameter(s) instead.
warnings.warn(
D:\python\lib\site-packages\torchvision\models_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
D:\python\lib\site-packages\pytorch_lightning\trainer\configuration_validator.py:108: You defined a validation_step but have no val_dataloader. Skipping val loop.

| Name | Type | Params

0 | encoder | ResnetAttentionEncoder | 11.7 M
1 | decoder | DepthDecoder | 3.2 M

14.8 M Trainable params
0 Non-trainable params
14.8 M Total params
59.392 Total estimated model params size (MB)
D:\python\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:224: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 8 which is the number of cpus on this machine) in theDataLoader` init to improve performance.
D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py:1609: The number of training batches (7) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
Epoch 0: 0%| | 0/7 [00:00<?, ?it/s] [ WARN:[email protected]] global loadsave.cpp:248 cv::findDecoder imread_('D:\Datasets\blender_stomach2\output\blender-duodenum-5-211126\images\0190.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 cv::findDecoder imread_('D:\Datasets\blender_stomach2\output\blender-duodenum-5-211126\images\0036.png'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "D:\slam\LINGMI-MR\LINGMI-MR\train_depth.py", line 40, in
trainer.fit(model, train_loader)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "D:\python\lib\site-packages\pytorch_lightning\trainer\call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1112, in _run
results = self._run_stage()
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1191, in _run_stage
self._run_train()
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1214, in _run_train
self.fit_loop.run()
File "D:\python\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
self.advance(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "D:\python\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
self.advance(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 187, in advance
batch = next(data_fetcher)
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 184, in next
return self.fetching_function()
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 265, in fetching_function
self._fetch_next_batch(self.dataloader_iter)
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 280, in _fetch_next_batch
batch = next(iterator)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 569, in next
return self.request_next_batch(self.loader_iters)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 581, in request_next_batch
return apply_to_collection(loader_iters, Iterator, next)
File "D:\python\lib\site-packages\lightning_utilities\core\apply_func.py", line 64, in apply_to_collection
return function(data, *args, **kwargs)
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 628, in next
data = self._next_data()
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 1359, in _process_data
data.reraise()
File "D:\python\lib\site-packages\torch_utils.py", line 543, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "D:\python\lib\site-packages\torch\utils\data_utils\worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "D:\python\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\python\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\slam\LINGMI-MR\LINGMI-MR\datasets\blender_dataset.py", line 83, in getitem
color = self.src.get_color(index)
File "D:\slam\LINGMI-MR\LINGMI-MR\datasets\blender_dataset.py", line 49, in get_color
img = np.transpose(img, (2, 0, 1))
File "<array_function internals>", line 180, in transpose
File "D:\python\lib\site-packages\numpy\core\fromnumeric.py", line 660, in transpose
return _wrapfunc(a, 'transpose', axes)
File "D:\python\lib\site-packages\numpy\core\fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
ValueError: axes don't match array

Epoch 0: 0%| | 0/7 [00:04<?, ?it/s]

Looking forward to your replying.

Thank you for your great job! I follow your instruction here, I download the your scenario dataset and put it under datasets folder, the data_path in blender_train.json is just like this LINGMI-MR/datasets/blender-duodenum-5-211126, without any other changes to this file.
I run the train code, it reports following problem in

Traceback (most recent call last):
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
    predict(args)
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 26, in predict
    raise RuntimeError("len(ckpt) <= 0")
RuntimeError: len(ckpt) <= 0

I thought it may caused by wrong checkpoint file path. So I change the path = Path(args.log_path) in

Traceback (most recent call last):
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
    predict(args)
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 30, in predict
    model = plEndoDepth.load_from_checkpoint(
  File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 139, in load_from_checkpoint
    return _load_from_checkpoint(
  File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 188, in _load_from_checkpoint
    return _load_state(cls, checkpoint, strict=strict, **kwargs)
  File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 234, in _load_state
    obj = cls(**_cls_kwargs)
  File "/root/autodl-tmp/LINGMI-MR/tools/trainers/endodepth.py", line 52, in __init__
    if options.backbone == 'transformer':
AttributeError: 'dict' object has no attribute 'backbone'

I add the following two lines in LINGMI-MR/tools/trainers/endodepth.py and run test code again

import argparse
# transfer the dict object to an ArgumentParser object
options = argparse.Namespace(**options)

The problem above has been successfully solved. But I run across the following problem in

(MIMR) root@autodl-container-4d4f11b352-484d72e2:~/autodl-tmp/LINGMI-MR# python eval_depth.py --config configs/blender_eval.json
/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/torchvision/models/_utils.py:135: UserWarning: Using 'weights' as positional parameter(s) is deprecated since 0.13 and will be removed in 0.15. Please use keyword parameter(s) instead.
  warnings.warn(
/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Traceback (most recent call last):
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
    predict(args)
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 38, in predict
    cam = CameraModel(args.intrinsic_path)
  File "/root/autodl-tmp/LINGMI-MR/geometry/camera.py", line 16, in __init__
    self.K = torch.FloatTensor(param['intrinsics'])
TypeError: string indices must be integers

After debugging the code, from my point of view, the intrinsics.txt in folder blender-duodenum-5-211126/param should in yaml format, not in txt format. Although the function load_yaml can successfully read the content in intrinsics.txt, the content read from intrinsics.txt is in str format, whereas it should be a dict format and contain the keys, such as 'intrinsics' , 'radial_distortion' and 'tangent_distortion'.
The problems I run across are listed above, I'm also curious about the detail of generating the dataset by blender. I would appreciate very much if you could reply me at any time in your convenience. Thank you again for this great work!

ERROR: Ignored the following yanked versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement torchvision>=0.15.0 (from versions: none)
ERROR: No matching distribution found for torchvision>=0.15.0

和python的版本有关吗？

When i use eval_depth.py --config configs/blender_eval.json command, there is an error 'dict' object has no attribute 'backbone'. why?

Hi, very impressive work!
Can you please upload the camera yaml file for the endoslam dataset used at https://github.com/YYM-SIA/LINGMI-MR/blob/main/datasets/endoslam_dataset.py#L21?
Thanks in advance!

I couldn't find instructions in your README regarding training your model on EndoSLAM and ColonDepth. Could I get them?

Hi, really nice work!
I want to know why the sim_dataset.py and blender_dataset.py use the same camera intrinsic K?