Comments (3)
pmyteh
commented on July 30, 2024
OK, I've done some digging and narrowed down what the problem seems to be. For background, Debian provides a sizeable number of CRAN packages through its system package manager apt. This includes RcppParallel, but not (yet?) quanteda. As the system packages are built binaries they save an enormous amount of manual re-installation (and re-compilation) on R version upgrades. My system has the r-cran-rcppparallel package installed, along with the tidyverse and a bunch of other stuff.
The underlying issue appears to be that tbb::atomic was deprecated in TBB 2020 and withdrawn in recent versions of TBB/oneAPI. The documentation suggests that std::atomic should be the replacement for c++17 and above (which we're using).
Unfortunately RcppParallel can be built with a wide range of different versions of TBB/oneAPI, including the internal bundled version, a locally installed copy, or the system TBB/oneAPI library. The internal version of TBB distributed with RcppParallel seems to be TBB 2019. Quanteda relies on this; with a later version of the libraries tbb::atomic doesn't exist and quanteda won't build. The Debian version of RcppParallel is linked against the system version, which is TBB 2021.
I have tried simply replacing the typedefs in lib.h with std::atomic, adding #include <atomic> and recompiling, but this generates a different set of build failures:
In file included from fcm.cpp:1:
../inst/include/lib.h: In function ‘std::vector<long unsigned int> quanteda::register_ngrams(Rcpp::List, Rcpp::IntegerVector, MapNgrams&)’:
../inst/include/lib.h:270:23: error: no matching function for call to ‘tbb::detail::d1::concurrent_unordered_map<std::vector<unsigned int>, std::atomic<unsigned int>, quanteda::hash_ngram, quanteda::equal_ngram>::insert(std::pair<std::vector<unsigned int>, std::atomic<unsigned int> >)’
270 | map.insert(std::pair<Ngram, IdNgram>(patterns[g], ids[g]));
| ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/lib/R/site-library/RcppParallel/include/oneapi/tbb/concurrent_unordered_map.h:21,
from /usr/lib/R/site-library/RcppParallel/include/tbb/../oneapi/tbb.h:43,
from /usr/lib/R/site-library/RcppParallel/include/tbb/tbb.h:17,
from /usr/lib/R/site-library/RcppParallel/include/RcppParallel/TBB.h:10,
from /usr/lib/R/site-library/RcppParallel/include/RcppParallel.h:21,
from ../inst/include/lib.h:5:
/usr/lib/R/site-library/RcppParallel/include/oneapi/tbb/detail/_concurrent_unordered_base.h:432:10: note: candidate: ‘template<class InputIterator> void tbb::detail::d1::concurrent_unordered_base<Traits>::insert(InputIterator, InputIterator) [with Traits = tbb::detail::d1::concurrent_unordered_map_traits<std::vector<unsigned int>, std::atomic<unsigned int>, quanteda::hash_ngram, quanteda::equal_ngram, tbb::detail::d1::tbb_allocator<std::pair<const std::vector<unsigned int>, std::atomic<unsigned int> > >, false>]’
432 | void insert( InputIterator first, InputIterator last ) {
| ^~~~~~
/usr/lib/R/site-library/RcppParallel/include/oneapi/tbb/detail/_concurrent_unordered_base.h:432:10: note: template argument deduction/substitution failed:
../inst/include/lib.h:270:23: note: candidate expects 2 arguments, 1 provided
270 | map.insert(std::pair<Ngram, IdNgram>(patterns[g], ids[g]));
| ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
I think this is the compiler not being able to find a suitable function dispatch target given the new typedefs, but I'm afraid my experience of C++ templates is zero and I'm not confident in digging any deeper.
I have solved this problem for me by uninstalling the system version of RcppParallel and reinstalling it through CRAN (which uses the bundled TBB library). This allows at least CRAN's quanteda release to build fine.
I'm leaving this issue open: it would obviously be nice for quanteda to handle this situation if easily possible, but in any case this will hopefully provide breadcrumbs to the workaround for anybody unlucky enough to follow in my footsteps! And I'm sure RcppParallel will update its bundled library at some point so a fix may be necessary at some future point.
from quanteda.
koheiw
commented on July 30, 2024
Thanks for reporting the error. I investigated why replacing tbb::atomic with std::atomic does not work. It seems that std::pair takes only tbb::atomic...
#include <RcppParallel.h>
#include <atomic>
std::vector<unsigned int> v1 = {1U, 2U};
std::atomic_uint v2(100U);
tbb::atomic<unsigned int> v3(100U);
std::pair< std::vector<unsigned int>, std::atomic_uint> p1(v1, v2); # errors
std::pair< std::vector<unsigned int>, tbb::atomic<unsigned int> > p2(v1, v3);
/usr/include/c++/11/bits/stl_pair.h:452:9: note: template argument deduction/substitution failed:
test.cpp:7:66: note: ‘std::vector<unsigned int>’ is not derived from ‘std::tuple<_Tps ...>’
7 | std::pair< std::vector<unsigned int>, std::atomic_uint> p1(v1, v2);
from quanteda.
koheiw
commented on July 30, 2024
These work!
#include <RcppParallel.h>
#include <atomic>
std::vector<unsigned int> v1 = {1U, 2U};
std::atomic_uint v2(100U);
std::pair< std::vector<unsigned int>, std::atomic_uint> p2(v1, v2.load());
std::pair< std::vector<unsigned int>, unsigned int> p3(v1, v2.load());from quanteda.
Related Issues (20)
- tbb::parallel_for crash
- Changes to generics and UseMethod in R development version breaking tokens generic
- NOTE created by clean script not removing Makevars.win
- Create safe pattern converter
- Install failure Fedora 39 g++ 13.2.1 quanteda 4.0.0; configure.ac error HOT 11
- TBB requirements need updating
- Homebrew installation of TBB still not found on macOS when compiling HOT 1
- quanteda cannot be updated HOT 1
- configure script could use tidying up HOT 1
- "randomly" failing test-fcm HOT 2
- Remove links to quanteda.textstats
- rolling stylometry HOT 2
- Quanteda build_tokens error when using rsplit from collapse package HOT 2
- Inconsistent results of corpus_reshape(to="sentences") HOT 1
- Consider passing ... to print
- Make tokens_substitute() to replace characters in tokens?
- Add more explicit information on enabling parallelization in quanteda >v4.0.0 HOT 1
- Experiencing problem with textmodel_mlp
- Add apply_if to tokens_ngrams()
- Error in parallel computing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quanteda.