is a cross-platform C++ library that provides functions to import and export numeric I/Q data to or from various file formats.

Two groups of file format are considered:

• special binary formats that contain I/Q data, as used by various instruments, for instance iq-tar.
• generic file formats that can be used to exchange raw data with standard PC tools, such as Microsoft Office or Matlab.

The library provides standardized read and write functions that encapsulate all file operations. The following file formats are supported:

Whenever supported by the file format, the following meta data will be stored when writing a file using IDataImportExport.

The following operating systems are supported

This library uses the following modules.

This library uses a lot of open source packages. We would like to thank the authors for their valuable work. A list of all used packages is located in lib/doc/Inventory.xml.

Install git and cmake.

Clone this repository to your local hard disc.

• make sure that zlib and libarchive and uuid-dev are available. If not, install via "sudo apt-get install zlib1g-dev libarchive-dev uuid-dev". maybe you also need something like "sudo cp /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libzlib.so"
• make sure doxygen is installed (sudo apt-get install doxygen)
• download https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.0-patch1/src/hdf5-1.10.0-patch1.tar and unpack
• Build hdf5:
  • create build folder "build" at the level of the "hdf5-1.10.0-patch1"-folder
  • cd "hdf5-1.10.0-patch1/build"
  • set install path and configure project using cmake. E.g., install to folder "usr/local/hdf5": "cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr/local/hdf5 -DHDF5_ENABLE_Z_LIB_SUPPORT=1 ../".
  • build target: "make"
  • install: "make install" -> hdf5 should now be available in "/usr/local/hdf5"
• build libdai:
  • create build folder e.g. "build"
  • cd "build"
  • set cmake prefix path: "export CMAKE_PREFIX_PATH=/usr/local/hdf5/share/cmake"
  • configure project: "cmake -DCMAKE_BUILD_TYPE={Debug|Release} .."
  • build project: make

• make sure that you have Visual Studio 2017 64 installed.
• download and install nunit 2.x and doxygen from the www.
• and also download the html help workshop, which generates a chm file.
• download and build zlib:
  • unpack zlib from 3rdparty directory to your hard disc and change into the unpacked directory
  • mkdir build
  • cd build
  • cmake -G"Visual Studio 15 2017 Win64" .. or
  • cmake -G "Visual Studio 16 2019" -A x64 ..
  • cmake --build . --config Release
• build daiex:
  • Change to the source directory of your local copy of the daiex repository.
  • Create a folder build.
  • Change to this folder
  • cmake -G"Visual Studio 15 2017 Win64" -DCMAKE_BUILD_TYPE=Release .. or
  • cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_BUILD_TYPE=Release ..
  • cmake --build . --config Release

Create a package, which includes all files you need to use daiex

• cmake --build . --config Release --target package If you have not downloaded and installed the HTML help workshop from Microsoft, you will get a strange error (setlocale).

• download and install homebrew
• install all needed packages with brew. On my machine there were the following packets (brew list): cmake, gmp, libarchive, ossp-uuid, xz, gcc, hdf5, libmatio, mpfr, gdbm, isl, libmpc, openssl, readline, szip.
• build libdai:
  • create build folder e.g. "build"
  • cd "build"
  • configure project: "cmake -DCMAKE_BUILD_TYPE={Debug|Release}"
  • build project: make

The version of libmatio downloaded by brew uses szip instead of zlib. You should have a look at the szip license or try to compile libmatio with zlib yourself. Using macOS there are some problems with .mat files so some tests are disabled at the moment.

Tests are executed via the daitest application. Usage: 'daitest {path-to-testfile} {path-to-temp-dir}'. {path-to-testfile} has to point to the location of the tests files, which is './test/res' and {path-to-temp-dir} is a location to temporarily write the test file.

By default no "big-file-tests" are executed. To include tests that write a large amount of data run 'daitest {path-to-testfile} {path-to-temp-dir} --gtest_also_run_disabled_tests'.

Further documentation generated with doxygen can be found in build/lib/doc/gen/. Open index.html with your browser.

The main interface to perform read and write operations is IDataImportExport, which is implemented by various classes like IqTar to support the different file formats.

When you work with the daiex library, you need the following include

#include "dataimportexport.h"

and namespace

using namespace rohdeschwarz::mosaik::dataimportexport;

To open a .iqx file, declare a instance of Iqx class and provide the path and name of the file. Use UTF-8 encoding for all strings passed to classes of LibDataImportExport.

Iqx inIqx("/tmp/example.iqx");

To read data from the file, the file must be opened to scan the contained data. During this process only meta data will be read, but no actual I/Q values. Most methods will return an error code. Evaluate this code to be ErrorCode::Success for successful operation. Otherwise an error message can be obtained using getErrorText for the specified error code.

The readOpen method opens the file in read only mode. The parameter arrayNames returns string identifiers of the data arrays found in this file. Use these identifiers to read the corresponding I/Q data.

vector<string> arrayNames;
int ret = inIqx.readOpen(arrayNames);
if (ret != ErrorCodes::Success)
{
  printf("file open failed\n");
  exit(1);
}

getMetadata gives additional information about the data

vector<ChannelInfo> channels;
map<string, string> metadata;
ret = inIqx.getMetadata(channels, metadata);
if (ret != ErrorCodes::Success)
{
  printf("get metadata failed\n");
  exit(1);
}

cout << "Number of channels: " << channels.size() << endl;
for (const auto& c : channels)
{
  cout << "Information for channel named " '" << c.getChannelName()  << "' ": clock rate: " << c.getClockRate() << ", center frequency: " << c.getFrequency() << endl;
}

// print all meta data found in file
for (const auto& kvp : metadata)
{
  cout << "Key: " << kvp.first << " has value: " << kvp.second << endl;
}

If you want to change parameters that affect the read or write behaviour of the data format, changes have to be applied before calling readOpen() or writeOpen(). For instance, the data order of an IQW file to be read can be changed using

Iqw readFile(filename);
returnCode = readFile.setDataOrder(IqDataOrder::IQIQIQ);
returnCode = readFile.readOpen(arrayNames);

In a similar fashion, the preview of I/Q data for the IqTar file format can be disabled using

IqTar writeFile(filename);
returnCode = writeFile.setPreviewEnabled(false);
returnCode = writeFile.writeOpen(...);

The actual I/Q data arrays can be accessed using the corresponding array names returned by IDataImportExport.readOpen. Each data array can be read with single or double precision regardless of the precision used to save the data to file. The read methods will perform the conversion internally.

for (size_t arrayIdx = 0; arrayIdx < arrayNames.size(); ++arrayIdx)
{
  // get the number of values stored in the array
  int64_t size = readFile.getArraySize(arrayNames[arrayIdx]);

  // Read data into vector.
  // For better read performance, resize the vector to the number of elements you want to read.
  // Otherwise, memory allocation will be performed by the read operation.
  vector<float> floatVector(size);

  // read data
  returnCode = readFile.readArray(arrayNames[arrayIdx], floatVector, size);

  // If you would like to read only a certain selection of the data array, define an additional offset.
  // The offset defines the number of values that are skipped before the read operation starts.
  size_t offset = 10;
  size_t nofValuesToRead = 100;
  returnCode = readFile.readArray(arrayNames[arrayIdx], floatVector, nofValuesToRead, offset);

  // If you prefer to read data to an array instead to a vector, you can do so.
  // Memory has to be preallocated.
  double doubleArray[100];
  returnCode = readFile.readArray(arrayNames[arrayIdx], &doubleArray, 100);
}

Data can not only be read array-wise using the stored array names (e.g. "Channel1_I", "Channel1_Q"), but also channel-wise (e.g. "Channel1"). Therefor use

returnCode = readFile.readChannel("Channel1", floatVector, size);

and

returnCode = readFile.readChannel("Channel1", &doubleArray, size);

respectively.

The read-offset specified in readChannel refers to the number of I/Q pairs, e.g. an offset of 4 skips the first 2 I/Q pairs.

returnCode = readFile.readChannel("Channel1", &doubleArray, size, offset);

When you are done with your read operations, close the file by calling

returnCode = readFile.close();

If a file format does not match the specification, an error code will be returned when readOpen() is called. In case of the CSV and Matlab file format, data fields can still be read using index-based access, cf. ICsvSelector and IArraySelector.

Attention: When reading large amounts of data incrementally reuse the allocated memory in order to avoid heap fragmentation. Array length is returned as int64_t as data arrays in a file can be very large, but the number of values that can be read with one function call is restricted to size_t.

Saving data to file is realized in a similar fashion than reading data. Therefore use methods writeOpen, writeArray or writeChannel and close.

vector<vector<float>> writeVector;
// ... fill data ...
// writeVector contains all data vectors, i.g. vector<vector containing Channel1_I values, vector containing Channel1_Q values>
// for a 1 Channel complex data array. For two complex channels, writeVector must contain 4 data vectors, etc.

// create channel information
double centerFreq = 15.5;
double clock = 1000.2;
vector<ChannelInfo> channelInfos;
channelInfos.push_back(ChannelInfo("Channel1", clock, centerFreq));

// create object of desired file format
IqTar writeFile("c:/myfile.iq.tar");

// open file for write operations: define the I/Q data format used and the number of arrays to be written.
size_t nofArraysToWrite = writeVector.size();
returnCode = writeFile.writeOpen(IqDataFormat::Complex, nofArraysToWrite, "My R/S test application", "My comment for test application", channelInfo);

// you can add optional meta data as key-value pairs to the file, if supported by the file format (currently all formats but IQW)
map<string, string> metadata;
metadata.insert(make_pair("Key", "Value"));
metadata.insert(make_pair("Key2", "Value2"));
returnCode = writeFile.writeOpen(IqDataFormat::Complex, nofArraysToWrite, "My R/S test application", "My comment for test application", channelInfo, &metadata);


for (size_t i = 0; i < 4; ++i)
{
  // fill writeVector with new data...
  // and write data to file
  returnCode = writeFile.appendArrays(writeVector);
}

// When all data has been added, call close
returnCode = writeFile.close();

Analogous to readArray, data can also be added using arrays instead of vectors. Therefore, provide a vector containing pointers to the actual data arrays and an additional vector that contains the lengths of the data arrays.

vector<float*> writeArray;
// ... add pointers to data arrays to writeArray ...

vector<size_t> arrayLengths;
arrayLength.push_back("length of array1");
arrayLength.push_back("length of array2");

// write data
writeFile.appendArrays(writeArray, arrayLengths);

Channels are written in a similar fashion:

// note that nofArraysToWrite is 1 instead of 2, as only 1 array containing interleaved I/Q data of the channel
// is passed instead of two arrays of separate I and Q values.
size_t nofArraysToWrite = 1;
returnCode = writeFile.writeOpen(IqDataForm::Complex, nofArraysToWrite, "My R/S test application", "My comment for test application", channelInfo);

vector<vector<float>> writeChannelVector;
// ...fill data ...
// writeChannelVector contains the data data of each channel per vector element, e.g. vector<Channel1 values, Channel2 values, etc.>.
// Complex-valued channels must contain I/Q data in interleaved format per channel, i.e. IQIQIQ.

vector<size_t> channelLengths;
channelLengths.push_back("nof values of channel");

returnCode = writeFile.appendChannels(writeChannelVector, channelLengths);

Performance: Many file formats make use of temporary files while adding new I/Q data. The final file is written when the close() operation is called. The default location used to write temporary files is the system's temp-directory, which is usually located on the primary hard drive. To obtain the best write performance, make sure to store temporary files on the same hard-drive than your final file is going to be stored. To change the temporary file location, call setTempDir on the respective file object.

The directory app contains a more detailed example. If you use the library with Windows, you have to include windows.h, otherwise you get compile errors.

The LibDataImportExport library provides a .NET Wrapper for the native c++ dll, in order to make the functionallity of LibDataImportExport usable in managed projects. An installation of the .NET Framework is required on the target machine. For details on how to use the wrapper classes, please refer to the Doxygen documentation.

The daiex library is licensed under the Apache 2.0 license. For more information see license.txt