yym-sia / lingmi-mr Goto Github PK
View Code? Open in Web Editor NEWA geometry-aware deep network for depth estimation in monocular endoscopy
License: MIT License
A geometry-aware deep network for depth estimation in monocular endoscopy
License: MIT License
Thank you for your great job! I follow your instruction here, I download the your scenario dataset and put it under datasets
folder, the data_path
in blender_train.json
is just like this LINGMI-MR/datasets/blender-duodenum-5-211126
, without any other changes to this file.
I run the train code, it reports following problem in
Line 35 in 1e9e8cb
lightning_logs/version_0/checkpoints/epoch=13-step=2520.ckpt
. After that, the train code runs successfully.Line 26 in 1e9e8cb
Traceback (most recent call last):
File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
predict(args)
File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 26, in predict
raise RuntimeError("len(ckpt) <= 0")
RuntimeError: len(ckpt) <= 0
I thought it may caused by wrong checkpoint file path. So I change the path = Path(args.log_path)
in
Line 23 in 1e9e8cb
path = Path(args.log_path + "/version_0")
. I run this code again and it reports following problem, the problem is the same as #2, the line is in LINGMI-MR/tools/trainers/endodepth.py
Line 52 in 1e9e8cb
Traceback (most recent call last):
File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
predict(args)
File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 30, in predict
model = plEndoDepth.load_from_checkpoint(
File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 139, in load_from_checkpoint
return _load_from_checkpoint(
File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 188, in _load_from_checkpoint
return _load_state(cls, checkpoint, strict=strict, **kwargs)
File "/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 234, in _load_state
obj = cls(**_cls_kwargs)
File "/root/autodl-tmp/LINGMI-MR/tools/trainers/endodepth.py", line 52, in __init__
if options.backbone == 'transformer':
AttributeError: 'dict' object has no attribute 'backbone'
I add the following two lines in LINGMI-MR/tools/trainers/endodepth.py
and run test code again
import argparse
# transfer the dict object to an ArgumentParser object
options = argparse.Namespace(**options)
The problem above has been successfully solved. But I run across the following problem in
Line 16 in 1e9e8cb
(MIMR) root@autodl-container-4d4f11b352-484d72e2:~/autodl-tmp/LINGMI-MR# python eval_depth.py --config configs/blender_eval.json
/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/torchvision/models/_utils.py:135: UserWarning: Using 'weights' as positional parameter(s) is deprecated since 0.13 and will be removed in 0.15. Please use keyword parameter(s) instead.
warnings.warn(
/root/miniconda3/envs/MIMR/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=None`.
warnings.warn(msg)
Traceback (most recent call last):
File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 90, in <module>
predict(args)
File "/root/autodl-tmp/LINGMI-MR/eval_depth.py", line 38, in predict
cam = CameraModel(args.intrinsic_path)
File "/root/autodl-tmp/LINGMI-MR/geometry/camera.py", line 16, in __init__
self.K = torch.FloatTensor(param['intrinsics'])
TypeError: string indices must be integers
After debugging the code, from my point of view, the intrinsics.txt
in folder blender-duodenum-5-211126/param
should in yaml
format, not in txt
format. Although the function load_yaml
can successfully read the content in intrinsics.txt
, the content read from intrinsics.txt
is in str
format, whereas it should be a dict
format and contain the keys, such as 'intrinsics'
, 'radial_distortion'
and 'tangent_distortion'
.
The problems I run across are listed above, I'm also curious about the detail of generating the dataset by blender. I would appreciate very much if you could reply me at any time in your convenience. Thank you again for this great work!
ERROR: Ignored the following yanked versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement torchvision>=0.15.0 (from versions: none)
ERROR: No matching distribution found for torchvision>=0.15.0
和python的版本有关吗?
Thank you for your sharing. When I was training ,there is some errors . Following is my result.
$ python train_depth.py --config configs/blender_train.json
D:\python\lib\site-packages\torchvision\models_utils.py:135: UserWarning: Using 'weights' as positional parameter(s) is deprecated since 0.13 and may be removed in the future. Please use keyword parameter(s) instead.
warnings.warn(
D:\python\lib\site-packages\torchvision\models_utils.py:223: UserWarning: Arguments other than a weight enum or None
for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None
.
warnings.warn(msg)
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
D:\python\lib\site-packages\pytorch_lightning\trainer\configuration_validator.py:108: You defined a validation_step
but have no val_dataloader
. Skipping val loop.
14.8 M Trainable params
0 Non-trainable params
14.8 M Total params
59.392 Total estimated model params size (MB)
D:\python\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:224: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers
argument(try 8 which is the number of cpus on this machine) in the
DataLoader` init to improve performance.
D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py:1609: The number of training batches (7) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
Epoch 0: 0%| | 0/7 [00:00<?, ?it/s] [ WARN:[email protected]] global loadsave.cpp:248 cv::findDecoder imread_('D:\Datasets\blender_stomach2\output\blender-duodenum-5-211126\images\0190.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 cv::findDecoder imread_('D:\Datasets\blender_stomach2\output\blender-duodenum-5-211126\images\0036.png'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "D:\slam\LINGMI-MR\LINGMI-MR\train_depth.py", line 40, in
trainer.fit(model, train_loader)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "D:\python\lib\site-packages\pytorch_lightning\trainer\call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1112, in _run
results = self._run_stage()
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1191, in _run_stage
self._run_train()
File "D:\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1214, in _run_train
self.fit_loop.run()
File "D:\python\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
self.advance(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "D:\python\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
self.advance(*args, **kwargs)
File "D:\python\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 187, in advance
batch = next(data_fetcher)
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 184, in next
return self.fetching_function()
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 265, in fetching_function
self._fetch_next_batch(self.dataloader_iter)
File "D:\python\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 280, in _fetch_next_batch
batch = next(iterator)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 569, in next
return self.request_next_batch(self.loader_iters)
File "D:\python\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 581, in request_next_batch
return apply_to_collection(loader_iters, Iterator, next)
File "D:\python\lib\site-packages\lightning_utilities\core\apply_func.py", line 64, in apply_to_collection
return function(data, *args, **kwargs)
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 628, in next
data = self._next_data()
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "D:\python\lib\site-packages\torch\utils\data\dataloader.py", line 1359, in _process_data
data.reraise()
File "D:\python\lib\site-packages\torch_utils.py", line 543, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "D:\python\lib\site-packages\torch\utils\data_utils\worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "D:\python\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\python\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\slam\LINGMI-MR\LINGMI-MR\datasets\blender_dataset.py", line 83, in getitem
color = self.src.get_color(index)
File "D:\slam\LINGMI-MR\LINGMI-MR\datasets\blender_dataset.py", line 49, in get_color
img = np.transpose(img, (2, 0, 1))
File "<array_function internals>", line 180, in transpose
File "D:\python\lib\site-packages\numpy\core\fromnumeric.py", line 660, in transpose
return _wrapfunc(a, 'transpose', axes)
File "D:\python\lib\site-packages\numpy\core\fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
ValueError: axes don't match array
Epoch 0: 0%| | 0/7 [00:04<?, ?it/s]
Looking forward to your replying.
Hi, really nice work!
I want to know why the sim_dataset.py and blender_dataset.py use the same camera intrinsic K?
When i use eval_depth.py --config configs/blender_eval.json command, there is an error 'dict' object has no attribute 'backbone'. why?
Hello, thank you for your kind words. Regarding the Endoslam-dataset, there seems to be a misalignment between the provided depth maps and RGB images by the official source. [Unity RGB & Depthmaps misalignment (url)
However, I noticed in your paper that the RGB images and depth maps appear to be aligned. Could you please provide insights into the processing steps you employed to achieve this alignment? I look forward to your response.
Hi, very impressive work!
Can you please upload the camera yaml file for the endoslam dataset used at https://github.com/YYM-SIA/LINGMI-MR/blob/main/datasets/endoslam_dataset.py#L21?
Thanks in advance!
I couldn't find instructions in your README regarding training your model on EndoSLAM and ColonDepth. Could I get them?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.