powturbo / turbo-base64 Goto Github PK

ClickHouse/ClickHouse#41957

Hello,
tb64declen() and tb64dec() return 0 when the input base64 has length of 785
it's repeatable with the following b64

UVNUSwEGBgACQAAF//4A//0C+fb//QcF/fgJAP8GCvgEB/0JAP77/fwJBQH2APz/AAIA/vUDAAID+wIC/vwA/gIH/fr//f0CBwH8/QMBAwgAAvf9AgAMEf0A/wX3BgAC+QD/Bv0LBQEB/QT9Bg3+A/7+Bf70/PwB+PsI9vkJAwv+AP4EAwD4//r8/fUBA/wD+P8FAAQCBQEBAP/9BgMH//4F//wAAPn4/AAH/P8JAgAAAfsGAAEJ/wAC/v75BwIHA/sF+v4ABfr8AP79A/37BQgBAAD6AAYCBAcD/QL3CP72/wAAAQAABP7+AP4AAvgB/PkAAQD+DgUABAAFBfAA///7/QoEBAEEAv8DC/8BAPoB+v7/AP//AAr+AO0S+/oFAgYF/wb7BQP99QcCAQT5CPkAAAj9AAT/8/n1AQMD9/UL/wAAAgL//fgIBv/8Aw4A+/4GCPYAAQED9AkDAvkG/gP8AggC/fr/Bwb+AgD/9f3/BgIF+BH9BP3+AgMEBAkGA/4AAwv6AQUDAPz+BwQC+/L2+f75/wX/APj/AwkAAQX+AAD6//oDAgwAAAH9+AP/AP4A/QIA/An2+/n9AAAFAAMCBAYD+wH+/QD7BP//9gABA/T3CAAJAP0A/PgB/wICAAX7BAL6BQIE//z+/wcB9/z6AvUEAAIE/wEECf748P0DAv0E/AEFAAD7+Qz/9wDeCQvq+gT09/P8/AAdERf+8df57AsC+wzv+fzY+Qrt8hgbAecG7B31AgL1+Pj6+OUP8wX5Bub9+AD6CgE=

is there any limitation on the input?

Thanks

ClickHouse/ClickHouse#8397 (comment)

The library behaves worse than https://github.com/aklomp/base64
on strings of average length 77 bytes:

:) SELECT avg(length(URL)) FROM test.hits

SELECT avg(length(URL))
FROM test.hits

┌──avg(length(URL))─┐
│ 77.54074297450794 │
└───────────────────┘

You can download the test data here:
https://clickhouse.yandex/docs/en/getting_started/example_datasets/metrica/

Nice work
but makefile let it include lots of bug and uneasy to use on different platform
I really can't config it on my computer
May you provide a CMake version to beat this? https://github.com/aklomp/base64

ClickHouse/ClickHouse#9491

Some files (conf.h and time_.h) have GPL v2 License. And it looks like a mistake (since all other files including LICENSE have another license).

If it's a mistake, please, update license text in these files and remove GPL v2 License. It will allow the usage of this library in projects where GPLv2 is incompatible.

Thanks.

Given enough buffer space, would in-situ encoding be feasible you think ?

Keep up the great work

Which shows MB/s of the input data, rather than output.

And you doesn't clarify that anywhere! But clarify that Kb means 1000.

Are you also calculating MB/s for decompressors in terms of input size? So if you compress gigabytes of zeros, then the performance of the decompressor will be around zero (because the input will be extremely small).

If we imagine a base64 modification in which encoding and decoding take the same amount of time, then your benchmark will show that decoding is 1.333 times faster.

So I think it's wrong.

FYI, Turbo-Base64 was submitted to vcpkg port. You are welcome to review and comment here

Hello,

Compilation fails on Apple Clang as a couple of files need to include stdlib.h.

Please add includes to turbob64c.c and turbob64v128.c.

diff --git a/turbob64c.c b/turbob64c.c
index f52be2d..8598f1d 100644
--- a/turbob64c.c
+++ b/turbob64c.c
@@ -24,6 +24,7 @@
 // Turbo-Base64: Scalar encode
 #include "turbob64_.h"
 #include "turbob64.h"
+#include <stdlib.h>
 
 size_t tb64enclen(size_t n) { return TB64ENCLEN(n); }
  
diff --git a/turbob64v128.c b/turbob64v128.c
index b1f2ba0..d0cdc4d 100644
--- a/turbob64v128.c
+++ b/turbob64v128.c
@@ -24,6 +24,7 @@
 //  Turbo-Base64: ssse3 + arm neon functions (see also turbob64v256)
 
 #include <string.h>
+#include <stdlib.h>
 
   #if defined(__AVX__)
 #include <immintrin.h>

Hi @powturbo
Thanks for your great work!
Recently, I discovered that when using this library to encode image files, there are some strange characters appearing at the end. Printing them out shows 'NULL' or just some patterns. ( Like ”7mlbMjdKxLobZAOx6jFekoqMbHg==�#��+Z��8Z�s)��k_H���pd�?���Ծ ” "Px/wA7sn4uWWf/AAj/AA3/ALQooor0Yg==NULLNULLNULL"

My code：

std::ifstream ifs(file_path, std::ios::binary);
if (!ifs.is_open()) {
    std::cerr << "Unable to open file: " << file_path << std::endl;
}

ifs.seekg(0, std::ios::end);
auto size = ifs.tellg();
ifs.seekg(0, std::ios::beg);

// Read the file content into a char buffer
auto buf = new unsigned char[size];
ifs.read((char *) buf, size);

//use turbobase64
auto outsize = tb64enclen(size);
auto out = new uint8_t[outsize];

size_t num_enc = tb64enc(buf, size, out); //error handle

out[num_enc] = 0;

std::string str_encode(out, out + num_enc);

std::cout << str_encode << std::endl;

I'm confused. Shouldn't the size of a string converted to Base64 be fixed? Why are there unknown characters appearing

Hey, Seems like a great library can you please provide a simple example of how to use this library? I having issues figuring a few things out like do I have to allocate the memory for the output buffer too after decoding, I think a sample simple example will help a lot of people.

Thank you

Hello,

It is often not feasible to prepare the entire input buffer or the entire output buffer prior to a single call of a transformation algo.

For example, look at zlib API. User provides input chunks in succession and output chunks in succession. The entire data to be transformed does not have to be in memory at once.

https://zlib.net/zlib_how.html

Please provide this kind of multi-call API!

I'm not sure the output is always correct wrt the padding at the end of the generated encoding (though it is still reversible).

char kBase64Chars[] =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";

char* out = malloc(turbob64len(64) + 1024);
memset(out, 0, turbob64len(64) + 1024);
turbob64enc(kBase64Chars, 64, out);

The output:
QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVphYmNkZWZnaGlqa2xtbm9wcXJzdHV2d3h5ejAxMjM0NTY3ODkrLwAAAA==
which is 92 char.

The expected output is:
QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVphYmNkZWZnaGlqa2xtbm9wcXJzdHV2d3h5ejAxMjM0NTY3ODkrLw==
which is 88 char.

Note also

turbob64len(64) == 88

but will cause an overrun. This seem to be a known issue, since the benchmark over-allocates by 1024, but I'm not seeing any documentation.

Maybe a URL Safe variant version? Mapping "+" to "-" And "/" to "_" ?

hello!!! I finished base64 encode/decode previously but run a little slow ,i want to use you code in vs2015 , what can i do

Hi @powturbo ,

I'm using this great lib of yours for decoding BASE64 encoded image data exported from Microsoft Outlook. While things work smoothly with all my test data, I have one image failing to be decoded properly. I have attached the file base64.zip containing a file base64.dat (10880 bytes) for your reference.

When I pass the entire encoded data (of size 10880 bytes) to tb64dec(), it returns 0 (== error) from within tb64v128dec() at line 136: if(!(rc=tb64xdec(ip, inlen&(64-1), op)) || vaddvq_u8(vshrq_n_u8(xv,7))) return 0; //decode all

Reason being: tb64xdec() returns 0 immediately because the argument (inlen&(64-1)) equals 0 (as inlen == 10880) which is checked within that function. I don't think this is an intended behaviour.

If I decode the exact same data in 4 byte BASE64 chunks by repeatedly calling tb64dec() with proper offsets into the data, everything works just fine, so I conclude that a) the input data is correctly BASE64 encoded (I also checked with several BASE64 validitors successfully) and b) the above return code 0 is not correct ( the comment "// decode all" behind "return 0;" also suggests that there may be some different behaviour intended.

I would even argue that whenever you pass data of a size which is a multiple of 64 bytes to the tb64dec() function, it will return 0 instead of the size of the decoded data.

I therefore suggest to change the code as follows:

File turbob64v128.c:
[...]
//BG DEL line 136: if(!(rc=tb64xdec(ip, inlen&(64-1), op)) || vaddvq_u8(vshrq_n_u8(xv,7))) return 0; //decode all
//BG ADD line 136:
size_t rc = 0;
if (inlen&(64-1)) { // if inlen was not a multiple of 64 bytes, there's exactly inlen-(ip-in) == inlen&(64-1) bytes left to decode, but don't call if there's nothing left!
if(!(rc=tb64xdec(ip, inlen&(64-1), op)) || vaddvq_u8(vshrq_n_u8(xv,7)))
return 0; // failure!
}
//BG ADD end
return (op-out)+rc;

Without having looked into it deeper, I think tb64v256dec() probably suffers from the same issue.

Or maybe its all me. Any feedback or comments?

Kind regards,
Björn

base64.zip

It's great that you can convert gigabytes of data per second, but that's if all those gigabytes are in one long line!

This is a rare scenario. Whereas the usual scenario is processing a large set of small strings (1kb isn't very small). And some people might think that processing a large set of strings will take the same amount of time as if it were one long string, but that would make a huge difference. Try to process random sizes from say 1 to 64 bytes until the N megabytes is reached.

Although prefetching speeds up on big data, for small sizes it fills the cache with data that the application might not need. And worst of all, if the prefetch is done on the unallocated page (which could be next), then an exception will be thrown, and although the OS will handle and ignore this exception, it takes time.

So there also should be a test, where the input addresses are not consecutive, but randomized. (So miss-prefetching will only make worse.)

And the best thing would be a graph showing the cost of each size, starting at 1 until the graph turns into a straight line. How people are doing to show the performance of memcpy. Like this.