zero copy support:master about libnop HOT 2 CLOSED

Hi Mark,

Thanks for your patience getting back to you.

Unfortunately, the serializer isn't designed to handle encoding from sources other than memory. It would be very easy to break the protocol otherwise, since the serializer wouldn't be sure about what happened between calls.

Are you on a platform that supports mmap? If that's the case, then there might be something we can add to support your use case.

The library supports a concept called fungibility, which means you can define several versions of a data type and the library can use any of them interchangeably, as long as they would produce the same format data. We can use this mechanism to write an alternate Sprite that can point to an mmap'ed file.

To expand on your example, it might look something like this:

// Define the primary protocol for a Sprite.
struct Sprite {
  // Other fields ...
  std::vector<uint8_t> img;

  NOP_STRUCTURE(Sprite, /* other fields */, img);
};

// Define an alternate data type that conforms to the Sprite protocol and can be used
// with mmap.
struct SpriteMmap {
  // Other fields ...
  size_t img_size;
  uint8_t* img_data;

  NOP_STRUCTURE(SpriteMmap, /* other fields */, (img_size, img_data));
};

// Make sure SpriteMmap is fungible with Sprite.
static_assert(nop::IsFungible<Sprite, SpriteMmap>::value);

int fd = // Open file for reading.
const size_t file_size = // Size of image file in bytes.
uint8_t* img_data = mmap(nullptr, file_size, PROT_READ, MAP_SHARED, fd, /*offset*/ 0);

SpriteMmap sprite_file{/*other fields*/, file_size, img_data};

std::vector<uint8_t> buffer(nop::Encoding<SpriteMap>::Size(sprite_file));
nop::Serializer<nop::BufferEncoder> serializer{buffer.data(), buffer.size()};

serializer.Write(sprite_file);

Right now logical buffers only support grouping a size member with an array member, but we could add support for grouping a size member with a pointer member.

Does this sound like it would solve your problem?

from libnop.

Thanks, it would not work, but supporting zero copy is quite fundamental in my use case and I'm surprised that libraries don't support it. I'm using yas now where it is possible do to.

from libnop.