Tracking issue for:

• https://github.com/apache/tsfile/security/code-scanning/5

There are two Code scanning alerts need to be fixed.

https://github.com/apache/tsfile/security/code-scanning/3
https://github.com/apache/tsfile/security/code-scanning/4

Recently I analyzed the memory usage when using CPP to insert data with a Tablet. Based on the current implementation, there is still some work to be done on the memory management of CPP. For specific loads and memory requirements, we need to provide a suitable and accurate calculation method.

The tsfile Python code is already available in the repository (https://github.com/apache/tsfile/tree/develop/python). Publishing this package to PyPI would significantly enhance the user experience for Python developers. By making the tsfile package easily accessible through PyPI, it would simplify the installation process, reduce setup time, and increase the overall adoption of tsfile among the Python community. This step would undoubtedly contribute to broader usage and community engagement with tsfile.

When the amount of data written is minimal, multiple flush operations may lead to exceptions during the writing process to the file index.

pass:

TEST_F(TsFileWriterTest, MultiFlush) {
    std::string device_path = "device1";
    std::string measurement_name = "temperature";
    common::TSDataType data_type = common::TSDataType::INT32;
    common::TSEncoding encoding = common::TSEncoding::PLAIN;
    common::CompressionType compression_type =
        common::CompressionType::UNCOMPRESSED;
    ASSERT_EQ(tsfile_writer_->register_timeseries(device_path, measurement_name,
                                                  data_type, encoding,
                                                  compression_type),
              E_OK);
    for (int i = 1; i < 2; i++) {
        TsRecord record(i, device_path);
        DataPoint point(measurement_name, i);
        record.append_data_point(point);
        ASSERT_EQ(tsfile_writer_->write_record(record), E_OK);
    }
    ASSERT_EQ(tsfile_writer_->flush(), E_OK);

    for (int i = 2; i < 10; i++) {
        TsRecord record(i, device_path);
        DataPoint point(measurement_name, i);
        record.append_data_point(point);
        ASSERT_EQ(tsfile_writer_->write_record(record), E_OK);
    }
    ASSERT_EQ(tsfile_writer_->flush(), E_OK);
    ASSERT_EQ(tsfile_writer_->close(), E_OK);
}

FAILED

TEST_F(TsFileWriterTest, MultiFlush) {
    std::string device_path = "device1";
    std::string measurement_name = "temperature";
    common::TSDataType data_type = common::TSDataType::INT32;
    common::TSEncoding encoding = common::TSEncoding::PLAIN;
    common::CompressionType compression_type =
        common::CompressionType::UNCOMPRESSED;
    ASSERT_EQ(tsfile_writer_->register_timeseries(device_path, measurement_name,
                                                  data_type, encoding,
                                                  compression_type),
              E_OK);
    for (int i = 1; i < 2; i++) {
        TsRecord record(i, device_path);
        DataPoint point(measurement_name, i);
        record.append_data_point(point);
        ASSERT_EQ(tsfile_writer_->write_record(record), E_OK);
    }
    ASSERT_EQ(tsfile_writer_->flush(), E_OK);

    for (int i = 2; i < 3; i++) {
        TsRecord record(i, device_path);
        DataPoint point(measurement_name, i);
        record.append_data_point(point);
        ASSERT_EQ(tsfile_writer_->write_record(record), E_OK);
    }
    ASSERT_EQ(tsfile_writer_->flush(), E_OK);
    ASSERT_EQ(tsfile_writer_->close(), E_OK);
}

It is reported that the current TsFile-CPP can only flush once before closing the file.
Multiple flushes before closing should be supported for continuous writing.

The TsFile-CPP can only be comiled on unix currently.
Compilation on Windows should be supported.

The format of TsFile is going to evolve to V4 and TsFile-CPP should also be updated.

Many compressors are still unimplemented in the current version, and we should supplement them.

We have introduced new datatypes such as DATE, TIMESTAMP, BLOB, and STRING in TsFile-Java, which should also be supported in TsFile-CPP.
Also it seems TEXT is not yet supported, which is an important type.

Does the TsFile_CPP project require unit tests? Can we use third-party libraries such as Gtest or CppUnit ?

Test code as below:

        CTsFileWriter writer = ts_writer_open(TSFILE_NAME, &code);
        ASSERT_EQ(code, 0);
        ASSERT_NE(writer, nullptr);
        code = ts_writer_close(writer);
        ASSERT_EQ(access(TSFILE_NAME, F_OK), 0);
        ASSERT_EQ(code, E_OK); // assert failed

Add aligned timeseries suport in TsFile-CPP.

It is reported that the implementation of the TS_2Diff decoder in TsFile-CPP will mistakenly skip one byte, resulting in a wrong decoding result.

There is an UT failed. We need to take a look.

https://github.com/apache/tsfile/actions/runs/9804364778/job/27072016556

The time compression and encoding can be specified via config. However, this information is not included in a TsFile, and one cannot know the time compression and encoding if he only has the TsFile.

For TsFile V4, there is a property map at the end of the file, so we can and WE SHOULD record the time compression and encoding in the map.

For TsFile V3 and older, unfortunately, there seems to be no good solution, but maybe we can provide some protection mechanism: if the decoding or uncompressing of a time page fails, the reader should try another decoding/compression automatically until success or all methods have failed.

Is concurrent access to a TsFile possible using the API directly or should such accesses be handled using locks?

TEST_F(BitMapTest, Clear) {
common::BitMap bitmap;
bitmap.init(100);
bitmap.set(10);
bitmap.set(20);
bitmap.clear(10);
EXPECT_FALSE(bitmap.test(10)); // This line cannot execute correctly
EXPECT_TRUE(bitmap.test(20));
}

To correctly clear a bit, the line *start_addr = (*start_addr) ^ (~bit_mask);
should be corrected to *start_addr = (*start_addr) & ~bit_mask;

Bugfix: #114

I noticed that the DictionaryEncoder uses a BitPackingEncoder as the inner encoder in the CPP edition. However, in the Jave edition, the inner encoder is a RleEncoder (with bit-packing), which may cause incompatibility.

All encodings and compression in the CPP edition should be checked to ensure compatibility.

The "remove" method misses updating the value of "size_", and incorrectly returns "common::E_NOT_EXIST" after finding the element.

The serialization and deserialization logic of TsFileMeta do not match, the handling of the bloom filter is missing during serialization.

    int serialize_to(common::ByteStream &out) {
        int ret = common::E_OK;
        if (RET_FAIL(index_node_->serialize_to(out))) {
        } else if (RET_FAIL(common::SerializationUtil::write_i64(meta_offset_,
                                                                 out))) {
        }
        return ret;
    }

    int deserialize_from(common::ByteStream &in) {
        int ret = common::E_OK;
        void *index_node_buf = page_arena_->alloc(sizeof(MetaIndexNode));
        void *bloom_filter_buf = page_arena_->alloc(sizeof(BloomFilter));
        if (IS_NULL(index_node_buf) || IS_NULL(bloom_filter_buf)) {
            return common::E_OOM;
        }
        index_node_ = new (index_node_buf) MetaIndexNode(page_arena_);
        bloom_filter_ = new (bloom_filter_buf) BloomFilter();

        if (RET_FAIL(index_node_->deserialize_from(in))) {
        } else if (RET_FAIL(
                       common::SerializationUtil::read_i64(meta_offset_, in))) {
        } else if (RET_FAIL(bloom_filter_->deserialize_from(in))) {
        }
        return ret;
    }

Additional checks need to be added at delete and assert to prevent the program from crashing unexpectedly.
Here is the MR: #126

We may meet the same issue with apache/iotdb#3739.

We can fix it by adding a same file like
https://github.com/apache/iotdb/blob/master/.gitattributes

BitPackDecoder calls free for the variables current_buffer_, packer_, and tmp_buf in the destructor, but there is no corresponding malloc in the constructor. This will cause the program to crash.

~BitPackDecoder() { destroy(); }

void destroy() { /* do nothing for BitpackEncoder */
        delete (packer_);
        delete[] current_buffer_;
        common::mem_free(tmp_buf);
}

I found several files under the reader/filter directory with syntax errors that caused compilation failures. Additionally, there are logical errors in some functions, such as OrFilter::get_time_ranges.
I have fixed these issues.The relevant unit tests are attached in the PR. #135

The contents of the generated files of the programs in example and benchmark are inconsistent between unix and win. In Windows, if we open a file without O_BINARY flags, the system will replace '\n' with '\r\n' when the file is written

There is a bug that we cannot read tsfile by python wrapper on windows. Therefore currently we skipped the test on windows.

Here is the error we met on windows.
https://github.com/apache/tsfile/actions/runs/10202054823/job/28225338682