SnpServer is modular multi-platform screen sharing server written in C++ with similar features like VNC. Major focus is put on screen capture performance using state of the art hardware accelerated compression algorithms built into modern cpu/gpu/socs. The also experimental client component (SnpClient) is available at https://github.com/martin19/snpclient.

!!! This project is in research and development and no releases have been scheduled. !!!

Do not expect anything to work yet.

As of now the following components are implemented:

• keyboard sharing
• mouse sharing
• cursor sharing
• Raspberry Pi MMAL hardware encoder
• libva hardware encoder
• openh264 software encoder
• x11 grabber
• drm grabber

The SnpSourceGL module implements a screen capture method which uses libdrm and egl. There are some online resources for programming libdrm from a user space perspective, however most documentation seems to go into very much detail from a kernel perspective.

To understand the capture method, one needs to understand a little bit of drm, egl and some dma-buf. I'll explain the interesting parts in the sections below.

DRM/KMS (direct rendering manager, kernel mode setting) is an abstraction for hardware resources of different vendors which provides an api through ioctl calls. As ioctl calls are quite cumbersome to use, a wrapper library called libdrm has been created which should make programming a bit more convenient. With it directly access of GPU resources from user space is possible. What are these resources?

• framebuffers
• planes
• crtcs
• encoders
• connectors

A connector represents an actual mostly physical connector on the GPU (such as the HDMI-1 port, or a Composite out if available..)

Represents some signal transformation module to transform a signal from CRTC for
one or multiple encoders.

Literally a "Cathode ray tube controller" - but of course also supports tft and other displays. It abstracts away display timings.

An image on the framebuffer can be made up of several planes. This abstraction is fairly hard to grasp, and it is not really required to understand it in detail for our use case.

A framebuffer is essentially a chunk of gpu memory which contains the pixel information in a specific format (e.g, 1920x1080x32 - RGBA or some vendor specific pixel format) for example. There is a primary framebuffer which contains the actual pixel information displayed on the screen. There can be other framebuffers, e.g. a smaller one for the cursor is common.

The following illustration shows the relationship between these resource objects.

It turns out, to access a framebuffer it must be discovered first. Taking a look at the diagram suggests the method to do this. Let's say we want to capture the contents of the display connected to hdmi-1 port. To find the corresponding framebuffer we have to trace the path from the bottommost object (e.g. connector HDMI-1) to the (primary) framebuffer at the top. Then using the method drmModeGetFB a handle to the framebuffer object can be acquired.

The framebuffer can be mapped directly to user-space memory which is useful to read back the pixels without additionaly copy steps (zero-copy) or pass it directly to a compression algorithm, such as h.264 encoder. An area can be mapped to CPU adressable space by using a function called mmap.

mmap(nullptr, width * height * bytesPerPixel, PROT_READ, MAP_SHARED, dmaBufFd, 0);

The following call returns a pointer to the actual framebuffer. According to the access mode it can be directly read from and/or written to. The argument dmaBufFd is a file descriptor referencing the actual framebuffer we want to access.

On many gpus the framebuffer does not necessarily need to be in linear format in gpu memory. There are vendor specific tiled representations that exist because the gpu can access/cache memory in a more efficient way. Interpreting and displaying an image available in tiled memory format as RGBA image produces an image related to the original image (because the number of pixels in the image and the order of the individual color channels might be the same as for an linear RGBA format). Below is an example of a linux desktop in VC4 tiled format interpreted as linear RGB.

As we can see, directly accessing and displaying a tiled representation in GPU memory poses a problem. But what can we do to access the framebuffer in linear format? The idea is to either tell the application creating the framebuffer to create it in linear format or to let the GPU essentially transform the image back into a linear read-out format.

The application in charge for creating framebuffers is usually the windowing system -- which is either X11 or Wayland on Linux. More specifically it is the component inside the windowing system responsible for talking to the GPU. If we find this component an tell it to create the framebuffer in linear format instead of tiled format we are lucky. It turns out for X11 said component is called xf86-video-modesetting. We can patch the software in such a way that framebuffers are actually created as linear ones (see xf86-video-modesetting fork . This approach comes with a major drawback however: falling back to linear format has a performance impact on the performance of the desktop. Especially 3D applications seem to perform much worse than in tiled format. Another drawback from an application viewpoint is the necessity of a patched driver.

Another way to access a framebuffer in linear format is to convert it back to linear format for every captured frame. This sounds like an expensive operation to do in software as every pixel has to be rearranged, however the GPU should be able to do this quickly as it has the necessary "circuits built in". So how can we tell the gpu to do this conversion for us?

The idea is to do the following:

1. Create another framebuffer in linear format - the capture-framebuffer
2. For each frame, let the gpu copy the primary framebuffer in tiled format inside the capture-framebuffer
3. read out the capture-framebuffer

It turns out egl provides the tools to do this conversion for us: there are extensions in egl that enable interfacing with a drm framebuffer.

glEGLImageTargetTexture2DOES

The extension glEGLImageTargetTexture2DOES enables us to render anything to an GLImage texture. We use this to

We create two images. The first image imagePrimary references our primary framebuffer. We need to pass the appropriate parameters such as image format which is the tiled format DRM_FORMAT_MOD_BROADCOM_VC4_T_TILED in this case, the file descriptor of the dmaBuf, width and height of the framebuffer. The second image imageCapture references our readout framebuffer. We have created this framebuffer beforehand with the appropriate functions in libdrm (drmModeAddFB2WithModifiers) in linear read out format DRM_FORMAT_XRGB8888.

EGLAttrib const attribute_list_primary[] = {
    EGL_WIDTH, (int)fbPrimary->fbPtr->width,
    EGL_HEIGHT, (int)fbPrimary->fbPtr->height,
    EGL_LINUX_DRM_FOURCC_EXT, DRM_FORMAT_XRGB8888,
    EGL_DMA_BUF_PLANE0_FD_EXT, fbPrimary->dmaBufFd,
    EGL_DMA_BUF_PLANE0_OFFSET_EXT, 0,
    EGL_DMA_BUF_PLANE0_PITCH_EXT, (int)fbPrimary->fbPtr->pitch,
    EGL_DMA_BUF_PLANE0_MODIFIER_LO_EXT, DRM_FORMAT_MOD_BROADCOM_VC4_T_TILED & 0xFFFFFFFF,
    EGL_DMA_BUF_PLANE0_MODIFIER_HI_EXT, DRM_FORMAT_MOD_BROADCOM_VC4_T_TILED >> 32,
    EGL_DMA_BUF_PLANE0_MODIFIER_LO_EXT, I915_FORMAT_MOD_X_TILED & 0xFFFFFFFF,
    EGL_DMA_BUF_PLANE0_MODIFIER_HI_EXT, I915_FORMAT_MOD_X_TILED >> 32,
    EGL_NONE};
    
EGLAttrib const attribute_list_capture[] = {
    EGL_WIDTH, (int)fbCapture->fb2Ptr->width,
    EGL_HEIGHT, (int)fbCapture->fb2Ptr->height,
    EGL_LINUX_DRM_FOURCC_EXT, DRM_FORMAT_XRGB8888,
    EGL_DMA_BUF_PLANE0_FD_EXT, fbCapture->dmaBufFd,
    EGL_DMA_BUF_PLANE0_OFFSET_EXT, 0,
    EGL_DMA_BUF_PLANE0_PITCH_EXT, (int)fbCapture->fb2Ptr->pitches[0],
    EGL_DMA_BUF_PLANE0_MODIFIER_LO_EXT, DRM_FORMAT_MOD_LINEAR & 0xFFFFFFFF,
    EGL_DMA_BUF_PLANE0_MODIFIER_HI_EXT, DRM_FORMAT_MOD_LINEAR >> 32,
    EGL_NONE};    
    
imagePrimary = eglCreateImage(eglDisplay,
                              EGL_NO_CONTEXT,
                              EGL_LINUX_DMA_BUF_EXT,
                              (EGLClientBuffer)nullptr,
                              attribute_list_primary);

//create capture image - in linear mode
imageCapture = eglCreateImage(eglDisplay,
                              nullptr,
                              EGL_LINUX_DMA_BUF_EXT,
                              (EGLClientBuffer)nullptr,
                              attribute_list_capture);    

In the main capture loop we let egl read the input image into our tiled input texture and write the output into our linear capture texture. We can then memory map the output image for direct read-out.

glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, imagePrimary);
glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, imageCapture);                              

Copyright (c) 2021 @martin19

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.