Comments (12)
Hi @tafia, fstlib
is verified to compile correctly with the VS
, Clang
, GCC
and Intel
compilers. In the project I've currently only included a build setup using Visual Studio 2017
, but it would be nice to include also a build setup on linux, what are your tools of choice on linux?
greetings
from fstlib.
Thanks for answering!
GCC or CLang would be nice!
from fstlib.
Great, the compiler options that are used by R
on Linux are:
-O3 -fopenmp -Wall -pipe -Wno-unused -pedantic -mtune=generic -DNDEBUG
The library uses OpenMP
for multi-threading, so that one is very important. O3
helps making the compressor- and bit-shifter code faster...
from fstlib.
I'm still struggling building it on linux. Do you have per chance the whole commands used with gcc?
from fstlib.
Hi @tafia, thanks for spending time on that! The Makevars
file used by R
can be found here. This file uses macro's that are defined by the R
framework (such as SHLIB_OPENMP_CXXFLAGS
), but the general structure should be comparable to a normal makefile. The actual build output on Linux can be seen on the Travis build machine that compiles the package for Linux. The actual build log is located under the tab R CMD check out logs. The complete compiler options from that log are (here for the code file compression.o):
g++ -std=gnu++11 -I/home/travis/R-bin/lib/R/include -DNDEBUG -fopenmp -I. -Ifstcore
-Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress
-Ifstcore/ZSTD/compress -I"/home/travis/R/Library/Rcpp/include" -I/home/travis/R-bin/include
-fpic -g -O2 -O0 --coverage -c fstcore/compression/compression.cpp
-o fstcore/compression/compression.o
You don't need the R
libraries obviously :-). Also, Travis uses -O2 which is not ideal.
I apologize for not being able to spend more time on creating a decent build setup for Linux at the moment (I'm finalizing the release of the fst
package for R
).
But I will get back to doing that as soon as possible!
from fstlib.
Here is a working script to create libfst.so in Linux.
It is generated and modified based on the output of command 'Rscript -e "devtools::document();' executed in fstpackage/fst.
BTW, suggest we re-organize file structure of fstpackage/fstlib to have something similar to fstpackage/fst, e.g. rename lib to core and move files in programs/fstcpp to the parent directory of core.
from fstlib.
Hi @damondd , thanks for supplying the script!
Yes, the current structure is not very obvious. The code in the programs/fstcpp directory is a simple C++
implementation of the fstlib
library which is currently only used by the googletest
unit tests defined in tests\fstlibtests. The rational for putting it in a separate programs directory is to allow it to grow into a command line tool for manipulation of fst
files (that would be one of several example programs that use the fstlib
library).
But you're right, that could be more clearly structured and the tests should be able to run without a dependency on the programs directory.
A better approach might be to structure the repository in a way comparable to the ZSTD
library and with the googletests
run automatically on Travis like in this repository (and with a proper makefile supplied to the user as well)
Thanks for sharing your ideas!
from fstlib.
@MarcusKlik, I have created a branch at https://github.com/damondd/fstlib/tree/cmake, which shows how I build fstlib on local Linux environment, just FYI:
cd fstpackage/fstlib
mkdir build
cd build
# build release version (by default)
cmake -DCMAKE_INSTALL_PREFIX=/var/tmp/fstlib ..
# build debug version
# cmake -DCMAKE_INSTALL_PREFIX=/var/tmp/fstlib -DCMAKE_BUILD_TYPE=Debug ..
make
make install
And here are the primary changes in the initial commit:
Copy 'programs/fstcpp/' to new directory 'fstlib' and 'lib/' to 'fstlib/fstcore'.
Unify the way include internal headers like <fstlib/XX.h> <fstlib/fstcore/XX.h>.
Internal gtest is broken for Linux, so copied the latest googletest/include and googletest/src from github.
from fstlib.
Hi @damondd, thanks a lot, that's great!
I will study your setup with much interest when I can. Currently I am travelling and don't really have the means to do so, but I will get back to you in a few weeks (hope you don't mind).
Thanks for sharing your code!
from fstlib.
Thanks both of you! It is definitely not as straightforward as I imagined, I'll have a try asap.
from fstlib.
The cmake branch worked fine (I mean, I haven't tested the lib yet but it seems that the whole compilation went fine). THANK YOU!
On the other hand, the script didn't work on @MarcusKlik master branch. I am on ubuntu 18.04. I have run the script from the root directory.
$ sh build.sh
In file included from programs/fstcpp/fsttable.cpp:6:0:
programs/fstcpp/fsttable.h: In member function ‘virtual void BlockWriter::SetBuffersFromVec(long long unsigned int, long long unsigned int)’:
programs/fstcpp/fsttable.h:514:3: error: ‘memset’ was not declared in this scope
memset(naInts, 0, nrOfNAInts * 4); // clear NA bit metadata block (neccessary?)
^~~~~~
programs/fstcpp/fsttable.h:514:3: note: suggested alternative: ‘wmemset’
memset(naInts, 0, nrOfNAInts * 4); // clear NA bit metadata block (neccessary?)
^~~~~~
wmemset
programs/fstcpp/fsttable.h:565:4: error: ‘strncpy’ was not declared in this scope
strncpy(activeBuf + lastPos, str, pos - lastPos);
^~~~~~~
programs/fstcpp/fsttable.h:565:4: note: suggested alternative: ‘wcpncpy’
strncpy(activeBuf + lastPos, str, pos - lastPos);
^~~~~~~
wcpncpy
programs/fstcpp/fsttable.h: In member function ‘FstTable* FstTable::SubSet(std::vector<std::__cxx11::basic_string<char> >&, long long unsigned int, long long unsigned int) const’:
programs/fstcpp/fsttable.h:618:19: error: ‘find’ is not a member of ‘std’
auto it = std::find(colNames->begin(), colNames->end(), *colIt);
^~~~
programs/fstcpp/fsttable.h:618:19: note: suggested alternative: ‘end’
auto it = std::find(colNames->begin(), colNames->end(), *colIt);
^~~~
end
g++: error: ./fst_table.o: No such file or directory
from fstlib.
@tafia please pull the branch again, I've uploaded FstRowReader and FstColReader, as well as the sample c++ usage code in readme.
from fstlib.
Related Issues (10)
- Large datasets HOT 3
- Looking forward to full description of fst format HOT 11
- Can the small factors levels limits be increased from <128? HOT 1
- Documentation on setting up a `C++` project using the `fstlib` library
- Scan multi-threaded code for false sharing
- Project readme file
- Fix coveralls
- Format specifications and C++ API docs HOT 1
- Zero-row tables are correctly serialized
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fstlib.