teaengineering / libchronicle Goto Github PK
View Code? Open in Web Editor NEWShared-memory interprocess communication from C using OpenHFT's chronicle-queue protocol
License: Apache License 2.0
Shared-memory interprocess communication from C using OpenHFT's chronicle-queue protocol
License: Apache License 2.0
libchronicle can append and tail an existing queue and roll to a new queuefile, in either v4 or v5 format however it cannot create a new queue from scratch.
Appender code does not insert index
pages nor maintain index2index
.
Tailer codes does not use either to find correct replay point, does a linear scan.
Needs analysis of the upstream Java code to determine how to interpret the powers of two that control indexing.
Our append function does not write EOF to the current queuefile when rolling to the next. So our own tailer will get stuck on the last entry of the old queuefile, even though the appender has continued ahead.
We consult the clock before polling the appender, and will switch queuefiles in preparation for the next write. This is not correct. instead:
I would like to work on Scala Native bindings of your C library.
Could you please tell me which part of the API should be useful to be exposed for external use?
Thanks.
In the spice code I found the following TODO:
current iteration requires a Java Chronicle-Queues appender to be writing to the same
queue on a periodic timer to roll over the log files correctly and maintain the index structures.
Could you please provide more details on that?
couldn't find the timer.q file referred to.
exists: should be implemented in https://github.com/TeaEngineering/libchronicle/blob/master/bindings/kdb/shm-ipc-server.q
as
exists:{1h$count key x}
See docs and code at:
https://www.javadoc.io/doc/net.openhft/chronicle-queue/4.5.12/net/openhft/chronicle/queue/RollCycle.html
https://www.javadoc.io/doc/net.openhft/chronicle-queue/4.5.12/net/openhft/chronicle/queue/RollCycles.html
DAILY
HOURLY
HUGE_DAILY
LARGE_DAILY
MINUTELY
SMALL_DAILY
TEST_DAILY
TEST_HOURLY
TEST_SECONDLY
TEST2_DAILY
TEST4_DAILY
XLARGE_DAILY
It looks like the k.h
definition has changed in KxSystems/kdb
and the code base no longer compiles:
❱ make
wget https://raw.githubusercontent.com/KxSystems/kdb/master/c/c/k.h
--2020-09-12 14:57:10-- https://raw.githubusercontent.com/KxSystems/kdb/master/c/c/k.h
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.40.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.40.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4126 (4.0K) [text/plain]
Saving to: ‘k.h’
k.h 100%[===============================================================================>] 4.03K --.-KB/s in 0s
2020-09-12 14:57:10 (91.9 MB/s) - ‘k.h’ saved [4126/4126]
gcc -o obj/hpet.so hpet.c -DKXVER=3 -fPIC -I. -Wall -std=gnu99 -shared
gcc -o obj/shmipc.so shmipc.c -DKXVER=3 -fPIC -I. -Wall -std=gnu99 -shared
gcc -o obj/shmmain shmmain.c -DKXVER=3 -fPIC -I. -Wall -std=gnu99 -g -O0
In file included from shmmain.c:18:0:
mock_k.h:62:3: error: conflicting types for ‘dl’
K dl(void* fnptr, int n) {
^~
In file included from ./shmipc.c:32:0,
from shmmain.c:15:
./k.h:74:118: note: previous declaration of ‘dl’ was here
extern K ee(K),ktj(I,J),ka(I),kb(I),kg(I),kh(I),ki(I),kj(J),ke(F),kf(F),kc(I),ks(S),kd(I),kz(F),kt(I),sd1(I,K(*)(I)),dl(V*f,J),
^~
shmmain.c: In function ‘main’:
shmmain.c:199:21: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
fprintf(tmp, "%lld %lld %lld\n", clock, bytes, r->j);
~~~^
%ld
shmmain.c:214:24: warning: format ‘%lld’ expects argument of type ‘long long int *’, but argument 3 has type ‘uint64_t * {aka long unsigned int *}’ [-Wformat=]
r = fscanf(tmp, "%lld %lld %lld\n", &clock, &bytes, &index);
~~~^ ~~~~~~
%ld
shmmain.c:214:34: warning: format ‘%lld’ expects argument of type ‘long long int *’, but argument 5 has type ‘uint64_t * {aka long unsigned int *}’ [-Wformat=]
r = fscanf(tmp, "%lld %lld %lld\n", &clock, &bytes, &index);
~~~^ ~~~~~~
%ld
shmmain.c:217:21: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 2 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
printf(" did %lld %lld %lld\n", clock, bytes, index);
~~~^
%ld
shmmain.c:217:31: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 4 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
printf(" did %lld %lld %lld\n", clock, bytes, index);
~~~^
%ld
Makefile:31: recipe for target 'obj/shmmain' failed
make: *** [obj/shmmain] Error 1
Mock stubs are provided but it would be better to replace K with void* and have this plugable, will make future language bindings easier.
There are references to directory-listing
and not metadata
:
https://github.com/TeaEngineering/libchronicle/blob/master/native/shmipc.c#L140
https://github.com/TeaEngineering/libchronicle/blob/master/bindings/kdb/shm.q#L23
This file is not used in Chronicle Queue 5:
They are almost the same, except files directory-listing.cq4t was in earlier versions of Chronicle Queue, while metadata.cq4t is applicable for Chronicle Queue 5.0 onwards.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.