Comments (7)
Operating System: Linux 3.10.0-1160.62.1.el7.x86_64 (this is on HMS O2 with conda-forge python installed via mamba)
ohhhh, you're running on centos. Yeah, support for centos has been very hard, and I suspect that's the actual issue here.
I was able to get it to build and ship for CentOS on conda forge, so you could their docker setup if you want. but I've never got it to compile correctly locally.
I can try to take a crack at it on O2... but in general, haven't been developing there.
from nd2.
Gotcha.
I can believe that the undefined symbols (problem 2) is probably entirely a CentOS issue. I was hoping that installing all the relevant libraries (gcc, libstdcxx-ng, etc.) into a virtualenv from conda-forge would offer a way around the issue, if I can find the right versions of everything. This does appear to have been possible, at least a couple years ago. Some googling turned up:
- https://stackoverflow.com/questions/48453497/anaconda-libstdc-so-6-version-glibcxx-3-4-20-not-found
- ContinuumIO/anaconda-issues#483
- constantinpape/z5#43
It would be nice to have it build on O2, since that's where all my big ND2 files are (and where I can run dask distributed). I'm assuming the way to go about doing this is:
- play around and see if we can find versions of libstdcxx, libtiff, etc. that make things work
- if that fails, just use the conda-forge docker setup via singularity on O2 (I've never used singularity before)
It's less obvious to me why CentOS would cause the SDK .so files to be not installed at all (problem 1). I'm very confused by this and don't know quite how to attack it.
from nd2.
If you (or anyone) wants to dig into this, I think digging through the logs for the successful centos builds using the conda-forge CI would be a good start. The last one is here: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=474173&view=results
note the build & host requirements in the recipe ...
all the env vars during build here
After building, I think there is still an important (re)localization step, and I think it starts around here.
Note that most of the difficulty here comes from the fact that we're using precompiled sdk shared library files provided by LIM ... so library linking and dependencies is more complicated as a result
from nd2.
Finally got around to playing around with this more. I can run pytest locally on O2 with nd2 from conda-forge with all tests passing, and I can build nd2 on O2 and get all tests to pass except those in tests/test_dask_dispatch.py
and tests/test_segfaults.py
. (See MWEs below.) There the memmaps don't seem to be closed when they should be, and I cannot fathom why it works for the conda-forge package but doesn't when I build myself on O2. Unless you recognize the BufferErrors I'm seeing, I think I'm going to give up (being able to run most of the test suite is good enough).
Building locally
mamba env create -n nd2test
mamba activate nd2test
mamba install -c conda-forge cython black flake8 flake8-docstrings imagecodecs aicsimageio ipython isort mypy pre-commit psutil pydocstyle pytest pytest-cov wurlitzer xarray resource_backed_dask_array python=3.10.4 requests
git clone [email protected]:tlambert03/nd2.git
cd nd2
python scripts/download_samples.py
make
pip install --no-deps -e .
LD_LIBRARY_PATH=$CONDA_PREFIX/lib pytest
This gives 512 errors, all of them either BufferError
or AssertionError
. I rerun with pytest tests/test_segfaults.py
to show the error for a single test (it's the same as all the other failed tests):
============================= test session starts ==============================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/jqs1/projects/nd2, configfile: setup.cfg
plugins: cov-3.0.0
collected 525 items / 524 deselected / 1 selected
tests/test_segfaults.py FE [100%]
==================================== ERRORS ====================================
________________________ ERROR at teardown of test_seg _________________________
@pytest.fixture(autouse=True)
def no_files_left_open():
files_before = {p for p in psutil.Process().open_files() if p.path.endswith("nd2")}
yield
files_after = {p for p in psutil.Process().open_files() if p.path.endswith("nd2")}
> assert files_before == files_after == set()
E AssertionError: assert set() == {popenfile(pa...flags=557056)}
E Extra items in the right set:
E popenfile(path='/home/jqs1/projects/nd2/tests/data/jonas_header_test2.nd2', fd=16, position=0, mode='r', flags=557056)
E Use -v to get more diff
tests/conftest.py:46: AssertionError
=================================== FAILURES ===================================
___________________________________ test_seg ___________________________________
def test_seg():
with nd2.ND2File(str(DATA / "jonas_header_test2.nd2")) as f:
img = f.to_xarray(delayed=True, squeeze=False, position=0)
> a = img.compute()
tests/test_segfaults.py:13:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../mambaforge/envs/nd2/lib/python3.10/site-packages/xarray/core/dataarray.py:947: in compute
return new.load(**kwargs)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/xarray/core/dataarray.py:921: in load
ds = self._to_temp_dataset().load(**kwargs)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/xarray/core/dataset.py:861: in load
evaluated_data = da.compute(*lazy_data.values(), **kwargs)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/base.py:600: in compute
results = schedule(dsk, keys, **kwargs)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/threaded.py:81: in get
results = get_async(
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/local.py:508: in get_async
raise_exception(exc, tb)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/local.py:316: in reraise
raise exc
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/local.py:221: in execute_task
result = _execute_task(task, data)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/core.py:119: in _execute_task
return func(*(_execute_task(a, cache) for a in args))
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/core.py:119: in <genexpr>
return func(*(_execute_task(a, cache) for a in args))
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/core.py:119: in _execute_task
return func(*(_execute_task(a, cache) for a in args))
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/optimization.py:990: in __call__
return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/core.py:149: in get
result = _execute_task(task, cache)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/core.py:119: in _execute_task
return func(*(_execute_task(a, cache) for a in args))
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/utils.py:41: in apply
return func(*args, **kwargs)
../../mambaforge/envs/nd2/lib/python3.10/site-packages/dask/array/core.py:514: in _pass_extra_kwargs
return func(*args[len(keys) :], **kwargs)
src/nd2/nd2file.py:351: in _dask_block
self.close()
src/nd2/nd2file.py:82: in close
self._rdr.close()
src/nd2/_sdk/latest.pyx:57: in nd2._sdk.latest.ND2Reader.close
cpdef close(self):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> self._mmap.close()
E BufferError: cannot close exported pointers exist
src/nd2/_sdk/latest.pyx:60: BufferError
=========================== short test summary info ============================
FAILED tests/test_segfaults.py::test_seg - BufferError: cannot close exported...
ERROR tests/test_segfaults.py::test_seg - AssertionError: assert set() == {po...
================== 1 failed, 524 deselected, 1 error in 3.45s =================
Using the conda-forge build
To determine whether this is a compile-time or run-time problem, I want to verify that the conda-forge version passes all tests when running on CentOS (it does):
mamba env create -n nd2test
mamba activate nd2test
mamba install -c conda-forge python=3.10.4 nd2 aicsimageio xarray resource_backed_dask_array pytest requests
git clone [email protected]:tlambert03/nd2.git
cd nd2
python scripts/download_samples.py
pytest
gives
============================= test session starts ==============================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/jqs1/projects/nd2, configfile: setup.cfg
collected 525 items
tests/test_aicsimage.py .......... [ 1%]
tests/test_dask_dispatch.py ........ [ 3%]
tests/test_reader.py ................................................... [ 13%]
........................................................................ [ 26%]
...................................................sss.................. [ 40%]
........................................................................ [ 54%]
...................................ssssss............................... [ 68%]
............................................xxxx.x....x...xx............ [ 81%]
....................x............. [ 88%]
tests/test_readme.py . [ 88%]
tests/test_rescue.py . [ 88%]
tests/test_sdk.py ...................................................... [ 98%]
..... [ 99%]
tests/test_segfaults.py . [100%]
============= 507 passed, 9 skipped, 9 xfailed in 63.28s (0:01:03) =============
from nd2.
E BufferError: cannot close exported pointers exist
this is what I just fixed in #55
are you by chance using a main
that is more than 6 days old?
from nd2.
Oh, I'm an idiot. All tests pass. Thanks Talley!
from nd2.
Nice! That was easy 😂
from nd2.
Related Issues (20)
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- Getting ND2 series images in the order they were captured HOT 3
- order of axes HOT 8
- feature: implement lazy loading for binary data
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- Coordinates for RoiShapeType.Polygon not showing HOT 3
- todo: turn color integers into tuples (or something meaningful)
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- Missing axis HOT 10
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- [test-bot] pip install --pre is failing
- bioio-nd2 failing to load from large file HOT 1
- todo: improve handling of files with subloops
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nd2.