This is amazing work.
Have you tested it on KITTI or CityScapes datasets?

How does it compare to other SLAM pipelines?

When I try to install chamferdist, I get an error similar to krrish94/chamferdist#11 (comment).
CUDA_HOME path is fine and torch.cuda.is_available is also True.

Not sure how to solve it. Any ideas?

OS: Ubuntu 20.04
Pytorch 1.7.1
CUDA 11.2
Python 3.8.5

How can I run these codes?

How do I process the custom data? I have series of rgb images, depth images, camera intrinsic/extrinsic information

❓ Questions and Help

Loop closures are not explicitly discussed in the original paper for GradSLAM nor are present in the project's code nor documentation in an explicit way.
As loop closures play a fundamental role in classical SLAM, I'd like to know if you could explain how GradSLAM deals with them.

As far as I could understand from your paper, you use the "active part of the map" ( 'Ma' ) from features, which they are matched with sensor data (Pa) in a one-to-many fashion. It is not clear to me if the position and orientation of the sensor plays a role in such matching (e.g. if odometry from the integration of velocities plays a role in sensor-to-map matching).

Furthermore, the "Differentiable map fusion" is to me the only thing that resembles loop closure as I guess one can consider that when the same object is seen twice, then they are fused ( "or loop closed" ). Is this the case?
Again, it is not really clear whether the position+orientation resulting from velocity integration plays any role there.
The biggest concern I have is that if odometry is not taken into account, then two very similar observations could be easily "fused".

I would also like to know if "Differentiable map fusion" uses odometry and takes the role of loop closure, if it is reasonable to transform its inputs before entering in the "Differentiable map fusion" module so that a custom trained "loop closure detector" can be learnt from data.

Hi,

I would like to reproduce the performances of KinectFusion from your paper. Do you have reproducible code for it? If not, are you planning to release it?

Best,

📚 Documentation

Thanks for your work!

I think it'd be really helpful to bootstrap a project using gradslam an example of a tiny/simple model and a few lines of code showing how to train such model while, for instance, generating a full map. That can be any type of task.

I didn't find any example of a training for any task, so I was wondering in case none exists, you could provide a few code snippets or in case it does exist then point me out to where that is.

How can I use it on my own dataset? Or do I need to organize my dataset into a format similar to the ICL dataset?

Hi
First, thanks so much for sharing your code and making it public, really appreciate it

I just installed gradslam and trying it on Tum rgbd_dataset_freiburg1_xyz. The final point cloud doesn't look that good. Would really appreciate if you can tell me what I am doing wrong. This is a snippet of the code
odometry = "gradicp" # "gt", "icp", "gradicp"
dataset = TUM(basedir="C:/Users/cv/Documents/gradslam_data/TUM/", sequences="C:/Users/cv/Documents/gradslam_data/TUM/seq", seqlen = 10)
loader = DataLoader(dataset=dataset, batch_size=2)
colors, depths, intrinsics, poses, *_ = next(iter(loader))
rgbdimages = RGBDImages(colors, depths, intrinsics, poses, channels_first=False)

pointclouds, recovered_poses = slam(rgbdimages)
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
slam = PointFusion(odom=odometry, dsratio=4, device=device)
pointclouds, recovered_poses = slam(rgbdimages)
o3d.visualization.draw_geometries([pointclouds.open3d(0)])

Thanks

Thank you for sharing the code! I have managed to install it successfully and run the tutorials from the tutorials branch (although some parts looked odd, but that might just be me). However, there is no reference to live feed processing yet and building custom maps and so on, please correct me if I am wrong. It would make sense for that to be in a separate package, I am just wondering if that's in the works, or should I try to add it in myself for testing purposes?

I tried both
pip install gradslam
and
pip install 'git+https://github.com/gradslam/gradslam.git' -q

Both gives the same error:

ERROR: Could not find a version that satisfies the requirement open3d==0.10.0.0 (from gradslam) (from versions: none)
ERROR: No matching distribution found for open3d==0.10.0.0

It seems the latest open3d is 0.15.0 now, is it possible to upgrade the open3d dependency?

🐛 Bug / Unexpected behaviors

When using distortion in the dataconfigs, the GradSLAMDataset fails to apply the distortion as it tries to use the OpenCV function cv2.undistort on a PyTorch tensor.

See:

In line 222 the colour image is converted from numpy to a torch tensor (color = torch.from_numpy(color)). If self.distortion is given (line 229), cv2.undistort is then applied on the torch tensor instead of the numpy image (line 231).

To Reproduce

Steps to reproduce the behavior:

1. create a dataconfigs that contains a list of distortion coefficients in distortion:

camera_params:
  cx: 318.9576721191406
  cy: 247.3009033203125
  distortion:
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  fx: 607.2198486328125
  fy: 607.144287109375
  image_height: 480
  image_width: 640
  png_depth_scale: 1000
dataset_name: icl

2. run python run_feature_fusion_and_save_map.py --dataset-path [...] --sequence [...] --dataconfig-path [...]/my/dataset/icl.yaml

  File "[...]/gradslam/datasets/basedataset.py", line 233, in __getitem__
    color = cv2.undistort(color, K, self.distortion)
cv2.error: OpenCV(4.8.1) :-1: error: (-5:Bad argument) in function 'undistort'

Expected behavior

The GradSLAMDataset should apply the undistortion with the coefficients.

Just wondering if there is a way to take a voxel array (such as a ndarray of segmented 3d volume) and convert to a point array.

one example the array would be in a format of a 3d array with each value an intensity (if segmented) this would be a low number but worst case an 8 bit file with 255 segments but 0 classed as empty.

The only other way I could think of doing this was to convert this to a 2d format with four columns x, y, z, intensity (or label) but then not sure I could work out how to then get it into gradslam? I may be missing something obvious.

Hi @krrish94!
Thanks for sharing the code. Great work!

I wanted to know how can I reproduce the Analysis of Gradients section of the paper?
Does GradSLAM code-base provide this feature out of the box?

❓ Questions and Help

Hi,

First, amazing paper and repo, its been super interesting to test this SLAM technique.

My question is about the basic Pointfusion and step by step Pointfusion. Is there any advantage of the basic technique or is it always better to use the step method?

Best regards,

Looking at this example: https://github.com/gradslam/gradslam/blob/main/examples/tutorials/pointfusion_tutorial.ipynb, it seems like the poses data is only actually used in the "Advanced visualization" section, and is actually not even used in the SLAM process for the odometry step, given that slam.step seems to return the a pointcloud and a poses array. Given that is the case, if I write my own custom dataloader, and only load in Depth and Color Images, and place all of the reconstruct pose data into an array, and run the plotly_map_update_visualization(advanced visualization function) with that, would it work alright?

How to run gradslam with the TUM dataset?

I want to run gradslam in TUM dataset. I try to run examples/icpslam.py with the command:

python icpslam.py --dataset tum --dataset_path 'fr1's dataset'

But I just got two point clouds for visualization.

Can I run gradslam like orbslam2 or other slam method?

Hi, Thanks for this repository. This is indeed helpful.
I was trying for SLAM error calcuation using the package and ran into this error while using the provide(pcd1, pcd2) function in gradicp.py.

The error is as follows:

Traceback (most recent call last):
File "/hdd/scratch/prashant/code/diff-lidar-slam-dslr*/gans-DSLR.py", line 261, in
slam_error=Slam_error(groundtruth_list, recon_list)
File "/hdd/scratch/prashant/code/diff-lidar-slam-dslr*/SLAM_ERROR.py", line 109, in Slam_error
gnd_graph=find_poseGraph(gradslam.Pointclouds(gnd_scans,gnd_scan_normals)) #list of pytorh Tensor
File "/hdd/scratch/prashant/code/diff-lidar-slam-dslr*/SLAM_ERROR.py", line 92, in find_poseGraph
transform=odomprov.provide(pcd1,pcd2)
File "/hdd/scratch/prashant/code/diff-lidar-slam-dslr*/gradslam/gradslam/odometry/gradicp.py", line 106, in provide
transform, _ = point_to_plane_gradICP(
File "/hdd/scratch/prashant/code/diff-lidar-slam-dslr*/gradslam/gradslam/odometry/icputils.py", line 498, in point_to_plane_gradICP
A, b, chamfer_indices = gauss_newton_solve(
File "/hdd/scratch/prashant/code/diff-lidar-slam-dslr*/gradslam/gradslam/odometry/icputils.py", line 200, in gauss_newton_solve
_KNN = knn_points(src_pc, tgt_pc)
File "/home/prashant/miniconda3/envs/pyt/lib/python3.9/site-packages/chamferdist/chamfer.py", line 266, in knn_points
p1_dists, p1_idx = _knn_points.apply(
File "/home/prashant/miniconda3/envs/pyt/lib/python3.9/site-packages/chamferdist/chamfer.py", line 162, in forward
idx, dists = _C.knn_points_idx(p1, p2, lengths1, lengths2, K, version)
RuntimeError: Not compiled with GPU support.

I am not able to figure out the issue here and why is the error occuring.
I installed gradslam using pip.
I tried on multiple servers, but the same issue.
Request you to help me out.
Eagerly awaiting a response.

Where is the implementation of PyTorch auto differentiation ?

Hi, thanks for sharing the code. I checked the code of gauss_newton_solve in icputils.py but didn't found you using auto differentiation to implement LM solver.

Thank you!

❓ Questions and Help

Hi,

I am wondering how the TUM poses are loaded from the dataloader.

Suppose I have specified to get a sequence length of 3 images, i was expecting to get 2 poses for 3 images such that:
pose_1 = image_1 -> image_2
pose_2 = image_2 -> image_3

But, I noticed that the data loader is returning 3 poses, so what does the third pose correspond with? Am I correct to assume that pose_1 and pose_2 belong to the above mentioned sequence?

I read the original paper on gradslam and I got that it creates differentiable counterparts for different components of slam. The paper mentions that this allows us to backpropagate error from output to input, end-to-end (learning based systems)

My doubt is, what are the learnable parameters here? How does backpropagation help if we dont have any learnable parameters? Does this code has the ability to train or it just presented differentiable counterparts?

hi, i wonder how the jacobian is derived:
[nx, ny, nz, nz * sy - ny * sz, nx * sz - nz * sx, ny * sx - nx * sy] ?
in other words, what is the exact form of your residual function r ? thanks.

🚀 Feature

TPU with JAX ideas for possibility?

❓ Questions and Help

I am only able to get ~8.5 FPS when running gradslam step-by-step on GPU. I am wondering if it's possible to run gradslam at a higher processing speed (closer to 30 FPS). I need to use a batch size of 1, sequence length of 1, and width/height of 300.

Please see the example code below (To Reproduce). I have tried lowering the numiters parameter in SLAM, and that increases FPS to 40, but I notice significant degradation in quality when I visualize the pointcloud.

Furthermore, the following code (unrelated to the example below) seems to be particularly slow. Is there any way to speed this up? The inputs are numpy arrays.

def to_rgbd(frame, depth_frame, intrinsics, poses, device=None):
    frame = torch.tensor([[frame]], dtype=torch.float32)
    depth_frame = torch.tensor([[depth_frame]], dtype=torch.float32).unsqueeze(-1)
    intrinsics = torch.tensor([[intrinsics]], dtype=torch.float32)
    return RGBDImages(frame, depth_frame, intrinsics, poses, device=device)

To Reproduce

from time import time
import torch
from torch.utils.data import DataLoader

from gradslam.datasets.icl import ICL
from gradslam.slam.pointfusion import PointFusion
from gradslam.structures.rgbdimages import RGBDImages
from gradslam.structures.pointclouds import Pointclouds

dataset_path = 'ICL'

if __name__ == "__main__":
    device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
    dataset = ICL(dataset_path, seqlen=1, height=300, width=300)
    loader = DataLoader(dataset=dataset, batch_size=1)

    slam = PointFusion(device=device)
    pointcloud = Pointclouds(device=device)

    num_frames = 100
    time_elapsed = 0.0

    prev_frame = None

    for _ in range(num_frames):
        start = time()
        poses = torch.eye(4).view(1, 1, 4, 4) if prev_frame is None else None
        colors, depths, intrinsics, *_ = next(iter(loader))
        current_frame = RGBDImages(colors, depths, intrinsics, poses, device=device)
        pointcloud, current_frame.poses = slam.step(pointcloud, current_frame, prev_frame)
        prev_frame = current_frame
        time_elapsed += time() - start
    
    print('FPS', num_frames/time_elapsed)

Environment

PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2070 SUPER
Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.0
[conda] Could not collect

Hi, running the example provided here:
https://gradslam.readthedocs.io/en/latest/tutorials/pointfusion_tutorial.html

I've noticed high memory consumption using Pointcloud class. Just an instance creation using this line

slam = PointFusion(device=device)

consumes 900MB of GPU memory.

Also, during each SLAM step, approx size of point clouds (consists RGB, xyz and normals) of 3MB consumes approx 300MB of gpu memory.

I've tried using the inplace flag but this is an issue.
Can you please advise?
Thank you

Is there a way to append an RGBDImages object image-by-image ?

❓ Questions and Help

Its possible to use just point clouds as input?
Without pose matrix or something like that?

I'm trying to register point clouds captured by an industrial laser scanner... but we don't have pose estimation for any point cloud, just the capture order.

Thank you for impressive work and sharing the code. I have run it successfully but I have some questions about the details of gradICP using gradLM.

1. I think the residual |Ax-b|^2 defined in gauss_newton_solve is a linear approximation to point-to-line ICP and form a linear least squares. My question is that the problem can be directly solved by normal equations rather than LM solver. Why add dampling term here?
2. x is updated by a smooth gating function in iterate process. I guess the corresponding code is in icputils.py, line 538:

        sigmoid = 1 / ((1 + torch.exp(-B2 * errdiff)) ** (1 / nu))
        residual_transform = se3_exp(sigmoid * xi)
        src_pc = transform_pointcloud(src_pc[0], residual_transform).unsqueeze(0)

the sigmoid function is different from paper, adding ** (1 / nu) . I try to set nu=1 but get totally wrong result. errors and sigmoids in iterations are:

        Num 0 iteration
        current err is 236.284622, new err is 41.130245, sigmoid = 0.000000
        Num 1 iteration
        current err is 236.284622, new err is 41.130245, sigmoid = 0.000000

Sigmoid funtion seems to reject better lookahead error.

Thank you for your excellent work!

Could you please tell me, How gradslam generate the trajectory file (e.g. for tum-dataset)? Or do I need to write this function myself?

📚 Documentation

The current CONTRIBUTING guide suggests contributors to first run black gradslam and then check for flake8 gradslam errors.

A better way would be to reverse the two steps (i.e., first run flake8 to enlist files with errors, and then run black ONLY on these files).

Once the point cloud is generated, it would be best if we can save it, or better create a mesh and save it.

Assuming a sensor with poor depth quality is used (Eg. a realsense), how does your SLAM algorithm leverage the RGB data?

Hi,
Is there a plan to incorporate color ICP just like OPEN3D implemented based on J.Park, Q.-Y. Zhou, and V. Koltun, Colored Point Cloud Registration Revisited, ICCV, 2017

Thanks

Hi

I am trying to use gradslam on realsense data. I have saved a depth folder, rgb folder and both the timestamps files rgb and depth obtained from realsense
I tried using the TUM loader, but got some errors. So to figure how what I am doing wrong, I took TUM data that I have downloaded(rgbd_dataset_freiburg1_xyz) and which gradslam ran successfully with it and truncated all the data to 10 rgb 10 depth images and cleaned up the timestamps files and groundtruth file to reflect 10 samples only but then got error (shown below) loading the data.
Would appreciate your help infiguring out what i am doing wrong both for the truncated TUM data and the realsense data.
Thanks

load dataset

dataset = TUM(data_path, sequences="C:/Users/cv/Documents/gradslam_data/TUM/seq", seqlen=5)
loader = DataLoader(dataset=dataset, batch_size=1)
colors, depths, intrinsics, poses, *_ = next(iter(loader))

create rgbdimages object

rgbdimages = RGBDImages(colors, depths, intrinsics, poses)
rgbdimages.plotly(0).update_layout(autosize=False, height=600, width=400).show()

----errors

StopIteration Traceback (most recent call last)
in
2 dataset = TUM(data_path, sequences="C:/Users/cv/Documents/gradslam_data/TUM/seq", seqlen=5)
3 loader = DataLoader(dataset=dataset, batch_size=1)
----> 4 colors, depths, intrinsics, poses, *_ = next(iter(loader))
5
6 # create rgbdimages object

~\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py in next(self)
361
362 def next(self):
--> 363 data = self._next_data()
364 self._num_yielded += 1
365 if self._dataset_kind == _DatasetKind.Iterable and \

~\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
400
401 def _next_data(self):
--> 402 index = self._next_index() # may raise StopIteration
403 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
404 if self._pin_memory:

~\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py in _next_index(self)
355
356 def _next_index(self):
--> 357 return next(self._sampler_iter) # may raise StopIteration
358
359 def _next_data(self):

StopIteration:

🚀 Feature

First of all, thanks for the excellent work!
Is it possible to integrate using Intel Realsense? What is the estimated frame rate for HD image?

I am sanity checking our real-world RGB-D capture (collected with RTAB-map) with grad-SLAM for conceptgraphs.

everything works fine with replica, but as soon as I try to load in our dataset, device-side assertions happen. Unfortunately, error is non-deterministic, meaning it's not always thrown on the same frame.

This is the data loader I wrote to load RTAB-map datasets in conceptgraphs:

class RTABmapDataset(GradSLAMDataset):
    """A RGB-D dataset from a RTAB-map export"""

    def __init__(
        self,
        config_dict: dict[str, Any],
        basedir: str,
        sequence: str,
        stride: int | None = None,
        start: int | None = 0,
        end: int | None = -1,
        desired_height: int | None = 480,
        desired_width: int | None = 640,
        load_embeddings: bool | None = False,
        embedding_dir: str | None = "embeddings",
        embedding_dim: int | None = 512,
        **kwargs,
    ):
        self.input_folder = os.path.join(basedir, sequence)

        assert os.path.exists(self.input_folder), "Unknown dataset dir"

        self.pose_path = os.path.join(self.input_folder, "poses.txt")

        super().__init__(
            config_dict,
            stride=stride,
            start=start,
            end=end,
            desired_height=desired_height,
            desired_width=desired_width,
            load_embeddings=load_embeddings,
            embedding_dir=embedding_dir,
            embedding_dim=embedding_dim,
            **kwargs,
        )

    def get_filepaths(self):
        # filter out filepaths without optimized poses
        optim = set(
            np.genfromtxt(fname=self.pose_path, delimiter=" ", skip_header=1)[:, -1].astype(int)
        )

        def isinoptim(x: str) -> bool:
            x = os.path.basename(x).split(".")[0]
            return int(x) in optim
        
        embedding_paths = None

        color_paths = glob.glob(os.path.join(self.input_folder, "rgb", "*.jpg"))
        color_paths = natsorted(filter(isinoptim, color_paths))
        
        depth_paths = glob.glob(os.path.join(self.input_folder, "depth", "*.png"))
        depth_paths = natsorted(filter(isinoptim, depth_paths))

        return color_paths, depth_paths, embedding_paths

    def load_poses(self):
        from scipy.spatial import transform

        #
        trajectory = np.genfromtxt(fname=self.pose_path, delimiter=" ", skip_header=1)
        trajectory = trajectory[:, 1:-1]  # discard timestamp and key

        M = torch.tensor(
            [
                [1, 0, 0, 0],
                [0, 1, 0, 0],
                [0, 0, 1, 0],
                [0, 0, 0, 1]
            ]
        ).float()

        # assuming RTAB-map follows the RH coordinate system (x right, y up, z backward)

        P = []

        for state in trajectory:
            C_to_W = np.zeros((4, 4))
            C_to_W[3, 3] = 1.0
            C_to_W[:3, :3] = transform.Rotation.from_quat(state[3:]).as_matrix()
            C_to_W[:3, 3] = state[:3]
            C_to_W = torch.from_numpy(C_to_W).float()
            C_to_W = M @ C_to_W @ M.T

            P.append(C_to_W)

        return P

with the following config from a multi realsense setup (D435i + T265):

camera_params:
  cx: 318.409
  cy: 244.501
  image_height: 480
  image_width: 640
  fx: 608.794
  fy: 608.776
  png_depth_scale: 1000.0
dataset_name: rtabmap

After processing 50-ish frames, such a warning keeps repeating over and over:

/path/to/venv/lib/python3.10/site-packages/gradslam/slam/fusionutils.py:188: RuntimeWarning: Max of dot product was 1.3151907920837402 > 1. Inputs were not normalized along dim (-1). Was this intentional?
warnings.warn(
2%|███▍ | 67/3170 [00:09<10:56, 4.73it/s]
/path/to/venv/lib/python3.10/site-packages/gradslam/slam/fusionutils.py:188: RuntimeWarning: Max of dot product was 1.4051384925842285 > 1. Inputs were not normalized along dim (-1). Was this intentional?
warnings.warn(
2%|███▌

culminating in the following error:

../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [31679,0,0], thread: [122,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [31679,0,0], thread: [123,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [31679,0,0], thread: [124,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [31679,0,0], thread: [125,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [31679,0,0], thread: [126,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [31679,0,0], thread: [127,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
4%|██████▎ | 123/3170 [00:26<11:01, 4.60it/s]
Traceback (most recent call last):
File "/root/conceptgraphs/experiments/rtabmap-lingotto-600p-floor1-north/../../toolchain/run_slam_rgb.py", line 133, in
main(args)
File "/root/conceptgraphs/experiments/rtabmap-lingotto-600p-floor1-north/../../toolchain/run_slam_rgb.py", line 108, in main
pointclouds, _ = slam.step(pointclouds, frame_cur, frame_prev)
File "/root/miniconda3/envs/conceptgraphs/lib/python3.10/site-packages/gradslam/slam/icpslam.py", line 180, in step
pointclouds = self._map(pointclouds, live_frame, inplace)
File "/root/miniconda3/envs/conceptgraphs/lib/python3.10/site-packages/gradslam/slam/pointfusion.py", line 115, in _map
return update_map_fusion(
File "/root/miniconda3/envs/conceptgraphs/lib/python3.10/site-packages/gradslam/slam/fusionutils.py", line 899, in update_map_fusion
pc2im_bnhw = find_correspondences(pointclouds, rgbdimages, dist_th, dot_th)
File "/root/miniconda3/envs/conceptgraphs/lib/python3.10/site-packages/gradslam/slam/fusionutils.py", line 573, in find_correspondences
pc2im_bnhw, _ = find_similar_map_points(
File "/root/miniconda3/envs/conceptgraphs/lib/python3.10/site-packages/gradslam/slam/fusionutils.py", line 388, in find_similar_map_points
frame_points[pc2im_bnhw[:, 0], pc2im_bnhw[:, 1]] = vertex_maps[
RuntimeError: CUDA error: device-side assert triggered

The RGB-D capture is 3170-frame-long and should cover an area of about 30 x 10 meters.

Any idea what could trigger the error?

• pytorch version (e.g., 1.0): 2.1.2+cu121
• OS: ubuntu 22.04.4 LTS (x86_64)
• how you installed pytorch (conda, pip, source): pip
• python version: 3.10.13
• CUDA/cuDNN version: 11.8.89
• GPU models and configuration: GPU 0: NVIDIA A10G

Thanks for the impressive work and for sharing the code. I am trying to run gradslam using the ScanNet dataset. however, I have not found a way to get sequence associations(e.g.sceneXXXX_XX-seq_Y.txt) for datasets and therefore cannot satisfy the run requirement. Do you have a way to generate sequence associations for the datasets?

❓ Usage of multiple RGBD sensors as input

Thank you for making this great project open to the public!

I have a question regarding the possibility to run the provided SLAM pipelines using observations from different sensors.
For example, could you guide me through (if it is possible) how to use RGBD inputs from different cameras (say, mounted on a robot with known extrinsics and intrinsics)?

Ruslan