yym-sia / lingmi-mr Goto Github PK

ERROR: Ignored the following yanked versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement torchvision>=0.15.0 (from versions: none)
ERROR: No matching distribution found for torchvision>=0.15.0

和python的版本有关吗？

Hi, very impressive work!
Can you please upload the camera yaml file for the endoslam dataset used at https://github.com/YYM-SIA/LINGMI-MR/blob/main/datasets/endoslam_dataset.py#L21?
Thanks in advance!

When i use eval_depth.py --config configs/blender_eval.json command, there is an error 'dict' object has no attribute 'backbone'. why?

I couldn't find instructions in your README regarding training your model on EndoSLAM and ColonDepth. Could I get them?

Thank you for your great job! I follow your instruction here, I download the your scenario dataset and put it under datasets folder, the data_path in blender_train.json is just like this LINGMI-MR/datasets/blender-duodenum-5-211126, without any other changes to this file.
I run the train code, it reports following problem in

Traceback (most recent call last):
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
    predict(args)
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 26, in predict
    raise RuntimeError("len(ckpt) <= 0")
RuntimeError: len(ckpt) <= 0

I thought it may caused by wrong checkpoint file path. So I change the path = Path(args.log_path) in

Traceback (most recent call last):
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
    predict(args)
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 30, in predict
    model = plEndoDepth.load_from_checkpoint(
  File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 139, in load_from_checkpoint
    return _load_from_checkpoint(
  File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 188, in _load_from_checkpoint
    return _load_state(cls, checkpoint, strict=strict, **kwargs)
  File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 234, in _load_state
    obj = cls(**_cls_kwargs)
  File "/root/autodl-tmp/LINGMI-MR/tools/trainers/endodepth.py", line 52, in __init__
    if options.backbone == 'transformer':
AttributeError: 'dict' object has no attribute 'backbone'

I add the following two lines in LINGMI-MR/tools/trainers/endodepth.py and run test code again

import argparse
# transfer the dict object to an ArgumentParser object
options = argparse.Namespace(**options)

The problem above has been successfully solved. But I run across the following problem in

(MIMR) root@autodl-container-4d4f11b352-484d72e2:~/autodl-tmp/LINGMI-MR# python eval_depth.py --config configs/blender_eval.json
/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/torchvision/models/_utils.py:135: UserWarning: Using 'weights' as positional parameter(s) is deprecated since 0.13 and will be removed in 0.15. Please use keyword parameter(s) instead.
  warnings.warn(
/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Traceback (most recent call last):
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
    predict(args)
  File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 38, in predict
    cam = CameraModel(args.intrinsic_path)
  File "/root/autodl-tmp/LINGMI-MR/geometry/camera.py", line 16, in __init__
    self.K = torch.FloatTensor(param['intrinsics'])
TypeError: string indices must be integers

After debugging the code, from my point of view, the intrinsics.txt in folder blender-duodenum-5-211126/param should in yaml format, not in txt format. Although the function load_yaml can successfully read the content in intrinsics.txt, the content read from intrinsics.txt is in str format, whereas it should be a dict format and contain the keys, such as 'intrinsics' , 'radial_distortion' and 'tangent_distortion'.
The problems I run across are listed above, I'm also curious about the detail of generating the dataset by blender. I would appreciate very much if you could reply me at any time in your convenience. Thank you again for this great work!

Hi, really nice work!
I want to know why the sim_dataset.py and blender_dataset.py use the same camera intrinsic K?

Thank you for your sharing. When I was training ,there is some errors . Following is my result.

$ python train_depth.py --config configs/blender_train.json
D:\python\lib\site-packages\torchvision\models_utils.py:135: UserWarning: Using 'weights' as positional parameter(s) is deprecated since 0.13 and may be removed in the future. Please use keyword parameter(s) instead.
warnings.warn(
D:\python\lib\site-packages\torchvision\models_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
D:\python\lib\site-packages\pytorch_lightning\trainer\configuration_validator.py:108: You defined a validation_step but have no val_dataloader. Skipping val loop.

| Name | Type | Params

0 | encoder | ResnetAttentionEncoder | 11.7 M
1 | decoder | DepthDecoder | 3.2 M

14.8 M Trainable params
0 Non-trainable params
14.8 M Total params
59.392 Total estimated model params size (MB)
D:\python\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:224: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 8 which is the number of cpus on this machine) in theDataLoader` init to improve performance.
D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py:1609: The number of training batches (7) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
Epoch 0: 0%| | 0/7 [00:00<?, ?it/s] [ WARN:[email protected]] global loadsave.cpp:248 cv::findDecoder imread_('D:\Datasets\blender_stomach2\output\blender-duodenum-5-211126\images\0190.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 cv::findDecoder imread_('D:\Datasets\blender_stomach2\output\blender-duodenum-5-211126\images\0036.png'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "D:\slam\LINGMI-MR\LINGMI-MR\train_depth.py", line 40, in
trainer.fit(model, train_loader)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "D:\python\lib\site-packages\pytorch_lightning\trainer\call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1112, in _run
results = self._run_stage()
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1191, in _run_stage
self._run_train()
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1214, in _run_train
self.fit_loop.run()
File "D:\python\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
self.advance(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "D:\python\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
self.advance(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 187, in advance
batch = next(data_fetcher)
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 184, in next
return self.fetching_function()
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 265, in fetching_function
self._fetch_next_batch(self.dataloader_iter)
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 280, in _fetch_next_batch
batch = next(iterator)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 569, in next
return self.request_next_batch(self.loader_iters)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 581, in request_next_batch
return apply_to_collection(loader_iters, Iterator, next)
File "D:\python\lib\site-packages\lightning_utilities\core\apply_func.py", line 64, in apply_to_collection
return function(data, *args, **kwargs)
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 628, in next
data = self._next_data()
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 1359, in _process_data
data.reraise()
File "D:\python\lib\site-packages\torch_utils.py", line 543, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "D:\python\lib\site-packages\torch\utils\data_utils\worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "D:\python\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\python\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\slam\LINGMI-MR\LINGMI-MR\datasets\blender_dataset.py", line 83, in getitem
color = self.src.get_color(index)
File "D:\slam\LINGMI-MR\LINGMI-MR\datasets\blender_dataset.py", line 49, in get_color
img = np.transpose(img, (2, 0, 1))
File "<array_function internals>", line 180, in transpose
File "D:\python\lib\site-packages\numpy\core\fromnumeric.py", line 660, in transpose
return _wrapfunc(a, 'transpose', axes)
File "D:\python\lib\site-packages\numpy\core\fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
ValueError: axes don't match array

Epoch 0: 0%| | 0/7 [00:04<?, ?it/s]

Looking forward to your replying.

Hello, thank you for your kind words. Regarding the Endoslam-dataset, there seems to be a misalignment between the provided depth maps and RGB images by the official source. [Unity RGB & Depthmaps misalignment (url)
However, I noticed in your paper that the RGB images and depth maps appear to be aligned. Could you please provide insights into the processing steps you employed to achieve this alignment? I look forward to your response.