This is a repository to run realsense camera devices (RS) using Realsense SDK.

The librealsense SDK can be obtained by either:

1. compiling from source. Specific flags are needed to install the python wrappers and network cpability (refer here).

2. installing the python wrapper/package to run RS can be obtained through pip.

3. installing the pre-build packages.

The code in realsense folder uses the pyrealsense2 python package to stream images/frames from realsense devices.

See here.

It is possible that the request sent by the messenger returns error during start. This is caused by libusb having trouble communicating with the usb port. The error combined with invoke_and_wait in concurrency.h will cause the program to get stuck during pipe.start(). A hack is to keep sending request until no error is returned.

# librealsense/src/uvc/uvc-streamer.cpp:l151

void uvc_streamer::start()
    {
        _action_dispatcher.invoke_and_wait([this](dispatcher::cancellable_timer c)
        {
            if(_running)
                return;
            
            bool __reset_request = true;
            while(__reset_request)
            {
            __reset_request = false;
            
            _context.messenger->reset_endpoint(_read_endpoint, RS2_USB_ENDPOINT_DIRECTION_READ);
            
            {
                std::lock_guard<std::mutex> lock(_running_mutex);
                _running = true;
            }

            for(auto&& r : _requests)
            {
                auto sts = _context.messenger->submit_request(r);
                if(sts != platform::RS2_USB_STATUS_SUCCESS)
                    __reset_request = true;
                    // throw std::runtime_error("failed to submit UVC request while start streaming");
                    
            }
            
            if(__reset_request)
                for(auto&& r : _requests)
                    _context.messenger->cancel_request(r); 
            }
            
            _publish_frame_thread->start();

        }, [this](){ return _running; });
    }

• SENSOR_TIMESTAMP: Device clock / sensor starts taking the image
• FRAME_TIMESTAMP: Device clock / frame starts to transfer to the driver
• BACKEND_TIMESTAMP: PC clock / frame starts to transfer to the driver (HW->Kernel transition)
• TIME_OF_ARRIVAL: PC clock / frame is transfered to the driver (kernel->user space transition)

There is an issue with arm-based chips, i.e. pi4, where the backend_timestamp cannot be obtained.

The clocks for depth and color sensors are different. The frameset tries to map the closest frames together. See here.

The realsense can be syncroized using an external sync cable so that the cameras captures at the exact same timestamp.

The command rs-server needs to be run on the host connected to a RS to stream frames from the RS. The RS library on the host server needs to be compiled with -DBUILD_NETWORK_DEVICE=ON flag in order for the rs-server to work.

Once the rs-server on the host is running realsense-viewer can be used to view the stream on a client that is connected to the host in the same network. Currently there is a bug with SDK-V2.5 where the depth values are not streamed properly and stream settings in the realsense-viewer GUI cannot be changed (keeps crashing). SDK-V2.47 can be used instead.

There is an official guide on open source ethernet networking to stream RS over a network. The following service (taken from the raspberrypi image in the guide) enables rs-server to run constantly.

# /lib/systemd/system/rs-server.service

[Unit]
Description=Intel RealSense Camera Service
After=network-online.target

[Service]
ExecStart=/bin/bash /home/realsense/run-server.sh
WorkingDirectory=/home/realsense/librealsense/build
StandardOutput=inherit
StandardError=inherit
Restart=always
User=realsense

[Install]
WantedBy=multi-user.target

The former is an old flag of the latter one. See here.

The extraction of metadata is supported in windows but requires kernel patching for linux OS. See here and here.

When align is used with rs_net device, the resulting depth images is all zeros.

The rs2::colorizer class performs histogram equalization and thus performs differently compared to the colorizing example using opencv. See here.

See here.

See here

Setting the auto exposure manually may cause drop in FPS. See here.

The RS2_OPTION_AUTO_EXPOSURE_LIMIT options require stream to restart. Unit in mirco sec. See here.

The disparity is given with 1/32 pixel resolution. See here and here.

The rs.align function is slow. Compile it with cuda to make it faster. Uses about 100mb gpu memory for 30fs 848x480 color + depth stream.