Comments (23)
coretl
commented on July 25, 2024
3
Ok, looks like epicscorelibs.path.pyepics won't set the environment variable if it is already sets. Looks like you are using the conda python in hyperion, so it is already set (possibly by https://github.com/conda-forge/pyepics-feedstock/blob/7674e88710d481bc231627d37070db3dd6380b43/recipe/build.sh#L5).
Quick fix: unset PYEPICS_LIBCA before running hyperion
Longer term: Maybe we should do the unsetting here: https://github.com/bluesky/ophyd/blob/master/ophyd/__init__.py#L39
@tacaswell @danielballan @mrakitin any thoughts?
from ophyd-async.
coretl
commented on July 25, 2024
Can you do a pip freeze to see versions of dependencies? Is this on process exit, or does it crash when you are in the middle of something?
from ophyd-async.
olliesilvester
commented on July 25, 2024
I'm struggling to find the point at which this occurs, it seems to not be in the same place each time. It occurs sometime between hyperion launching and connecting to regular ophyd devices.
Pip freeze is:
accessible-pygments==0.0.4
aioca==1.7
aiohttp==3.9.1
aiosignal==1.3.1
alabaster==0.7.13
aniso8601==9.0.1
anyio==4.0.0
appdirs==1.4.4
asciitree==0.3.3
asttokens==2.4.0
async-timeout==4.0.3
attrs==23.1.0
Babel==2.13.0
backcall==0.2.0
beautifulsoup4==4.12.2
bidict==0.22.1
black==23.10.1
blinker==1.6.3
blueapi==0.3.15
bluesky==1.12.0.post1+g9b0d729
bluesky-kafka==0.10.0
bluesky-live==0.0.8
boltons==23.0.0
build==1.0.3
cachetools==5.3.1
caproto==1.1.0
certifi==2023.7.22
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.3.1
click==8.1.3
cloudpickle==3.0.0
colorama==0.4.6
comm==0.1.4
confluent-kafka==2.2.0
contourpy==1.1.1
coverage==7.3.2
cycler==0.12.1
dask==2023.10.0
databroker==1.2.5
dataclasses-json==0.6.1
decorator==5.1.1
Deprecated==1.2.14
distlib==0.3.7
dls-bluesky-core==0.0.1
-e git+ssh://[email protected]/DiamondLightSource/dodal.git@af16a0451d05640375686687651843e822dc9de8#egg=dls_dodal
dnspython==2.4.2
docopt==0.6.2
doct==1.1.0
docutils==0.20.1
email-validator==2.1.0.post1
entrypoints==0.4
epicscorelibs==7.0.7.99.0.2
event-model==1.19.8
exceptiongroup==1.1.3
executing==2.0.0
fastapi==0.98.0
fasteners==0.19
filelock==3.12.4
Flask==3.0.0
Flask-RESTful==0.3.10
fonttools==4.43.1
freephil==0.2.1
frozenlist==1.4.0
fsspec==2023.10.0
gitdb==4.0.11
GitPython==3.1.40
googleapis-common-protos==1.56.1
graypy==2.1.0
greenlet==3.0.0
grpcio==1.59.0
h11==0.14.0
h5py==3.10.0
hdf5plugin==4.2.0
HeapDict==1.0.1
historydict==1.2.6
httpcore==0.18.0
httptools==0.6.1
httpx==0.25.0
humanize==4.8.0
-e git+ssh://[email protected]/DiamondLightSource/hyperion.git@d15b3ace89d698caa322a0ce8498402ab5b2eae2#egg=hyperion
identify==2.5.30
idna==3.4
imageio==2.31.6
imagesize==1.4.1
importlib-metadata==6.8.0
importlib-resources==6.1.0
iniconfig==2.0.0
intake==0.6.4
ipython==8.16.1
ipywidgets==8.1.1
ispyb==8.0.1
itsdangerous==2.1.2
jedi==0.19.1
Jinja2==3.1.2
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
jupyterlab-widgets==3.0.9
kiwisolver==1.4.5
livereload==2.6.3
locket==1.0.0
MarkupSafe==2.1.3
marshmallow==3.20.1
matplotlib==3.8.0
matplotlib-inline==0.1.6
mockito==1.4.0
mongoquery==1.4.2
msgpack==1.0.7
msgpack-numpy==0.4.8
multidict==6.0.4
mypy==1.6.1
mypy-extensions==1.0.0
mysql-connector-python==8.1.0
networkx==3.2
nexgen @ git+https://github.com/dials/nexgen.git@db4858f6d91a3d07c6c0f815ef752849c0bf79d4
nodeenv==1.8.0
nose2==0.14.0
nslsii==0.9.1
numcodecs==0.12.1
numpy==1.26.1
opencv-python-headless==4.8.1.78
opentelemetry-api==1.20.0
opentelemetry-distro==0.41b0
opentelemetry-exporter-jaeger==1.20.0
opentelemetry-exporter-jaeger-proto-grpc==1.20.0
opentelemetry-exporter-jaeger-thrift==1.20.0
opentelemetry-instrumentation==0.41b0
opentelemetry-sdk==1.20.0
opentelemetry-semantic-conventions==0.41b0
ophyd==1.9.0
ophyd-async==0.2.0
orjson==3.9.9
p4p==4.1.10
packaging==23.2
pandas==2.1.1
parso==0.8.3
partd==1.4.1
pathlib2==2.3.7.post1
pathspec==0.11.2
pexpect==4.8.0
pickleshare==0.7.5
pika==1.3.2
Pillow==10.0.1
PIMS==0.6.1
Pint==0.22
pipdeptree==2.13.0
platformdirs==3.11.0
pluggy==1.3.0
ply==3.11
pre-commit==3.5.0
prettytable==3.9.0
prompt-toolkit==3.0.39
protobuf==4.21.12
psutil==5.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
pvxslibs==1.2.3
py==1.11.0
pydantic==1.10.13
pydata-sphinx-theme==0.14.1
pyepics==3.5.2
Pygments==2.16.1
pymongo==4.5.0
pyOlog==4.5.0
pyparsing==3.1.1
pyproject-api==1.6.1
pyproject_hooks==1.0.0
pytest==7.4.2
pytest-asyncio==0.21.1
pytest-cov==4.1.0
pytest-random-order==1.1.0
python-dateutil==2.8.2
python-dotenv==1.0.0
python-multipart==0.0.6
pytz==2023.3.post1
PyYAML==6.0.1
redis==5.0.1
referencing==0.30.2
requests==2.31.0
rpds-py==0.10.6
ruff==0.1.1
scanspec==0.6.3
scipy==1.11.3
semver==3.0.2
setuptools-dso==2.10a1
six==1.16.0
slicerator==1.1.0
smmap==5.0.1
sniffio==1.3.0
snowballstemmer==2.2.0
soupsieve==2.5
Sphinx==7.2.6
sphinx-autobuild==2021.3.14
sphinx-copybutton==0.5.2
sphinx_design==0.5.0
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-serializinghtml==1.1.9
SQLAlchemy==1.4.49
stack-data==0.6.3
starlette==0.27.0
stomp.py==8.1.0
suitcase-mongo==0.4.0
suitcase-msgpack==0.3.0
suitcase-utils==0.5.4
super-state-machine==2.0.2
tabulate==0.9.0
thrift==0.16.0
tifffile==2023.9.26
tomli==2.0.1
toolz==0.12.0
tornado==6.3.3
tox==3.28.0
tox-direct==0.4
tqdm==4.66.1
traitlets==5.11.2
types-mock==5.1.0.2
types-PyYAML==6.0.12.12
types-requests==2.31.0.10
typing-inspect==0.9.0
typing_extensions==4.5.0
tzdata==2023.3
tzlocal==5.2
ujson==5.8.0
urllib3==2.0.7
uvicorn==0.23.2
uvloop==0.19.0
virtualenv==20.24.6
watchfiles==0.21.0
wcwidth==0.2.8
websocket-client==1.6.4
websockets==12.0
Werkzeug==3.0.0
widgetsnbextension==4.0.9
workflows==2.26
wrapt==1.15.0
xarray==2023.10.1
yarl==1.9.3
zarr==2.16.1
zict==2.2.0
zipp==3.17.0
zocalo==0.30.2
from ophyd-async.
coretl
commented on July 25, 2024
Please could you import aioca._catools._Context and print these variables sometime after you've connected a traditional ophyd device?
from ophyd-async.
coretl
commented on July 25, 2024
It would also be good to know if this line is ever run:
It shouldn't be, as you say you are never using ophyd-async devices, so you will never call the connect function on an ophyd-async epics signal
from ophyd-async.
olliesilvester
commented on July 25, 2024
I'm not sure how helpful this will be, but it seems like we get this error whenever there's some other Exception caused in Hyperion. Eg if we have
if __name__ == "__main__":
raise Exception("blah")
Then the logs show the same error. If I switch back to ophyd-async 0.1.0 then I don't see this. This is without importing ophyd-async, but it's in the virtual environment.
Also:
ca available: False
_ca_context: None
_channel_caches: {}
from ophyd-async.
coretl
commented on July 25, 2024
What changed between 0.1 and 0.2 is that a temporary import of epicscorelibs was put in when importing ophyd_async.core. My guess is that somehow pyepics is using a different libca to that import.
Can you try running this line and see what it prints?
https://github.com/pyepics/pyepics/blob/c770f4cb9647c023e6e5eaede87c76f0524ab0ac/epics/ca.py#L332
from ophyd-async.
olliesilvester
commented on July 25, 2024
The return value was for find_libca: /dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.10/epics/lib/linux-x86_64/libca.so
from ophyd-async.
coretl
commented on July 25, 2024
Ok, that is the problem, two versions of libca loaded. I will see if I can work out how that happened...
from ophyd-async.
coretl
commented on July 25, 2024
What is os.environ['PYEPICS_LIBCA']?
from ophyd-async.
olliesilvester
commented on July 25, 2024
There doesn't seem to be anything in os.environ with this key
from ophyd-async.
coretl
commented on July 25, 2024
I suspect that these lines are not being run:
https://github.com/bluesky/ophyd/blob/9a0bf3f5fd8c98ee23e5d51c35db695c86a6a921/ophyd/__init__.py#L38-L42
But I can't see how that might be the case. Can you insert print statements in to see when these lines are being run, and whether it gives an import error?
from ophyd-async.
olliesilvester
commented on July 25, 2024
Yes, it looks like the import epicscorelibs.path.pyepics line is never being ran. Print statements around this line work, but the import itself doesn't happen, nor does it raise an import error
Edit: Actually it does import it - after running this line, os.environ['PYEPICS_LIBCA'] is /dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.10/epics/lib/linux-x86_64/libca.so
from ophyd-async.
olliesilvester
commented on July 25, 2024
Pylance also says the module can't be accessed and the line is greyed out
from ophyd-async.
coretl
commented on July 25, 2024
What happens if you import epicscorelibs.path.pyepics in your code?
from ophyd-async.
olliesilvester
commented on July 25, 2024
It imports it, and the environment is the same as the above
from ophyd-async.
coretl
commented on July 25, 2024
Alternatively we could get epicscorelibs to be more forceful and override the environment variable even if it is already set...
https://github.com/mdavidsaver/epicscorelibs/blob/cfdc449eb0e4e4efd6beac3eb323051c9a2128aa/src/python/epicscorelibs/path/pyepics.py#L21C38-L21C38
from ophyd-async.
coretl
commented on July 25, 2024
Ok, looks like
epicscorelibs.path.pyepicswon't set the environment variable if it is already sets. Looks like you are using the conda python in hyperion, so it is already set (possibly by https://github.com/conda-forge/pyepics-feedstock/blob/7674e88710d481bc231627d37070db3dd6380b43/recipe/build.sh#L5).Quick fix:
unset PYEPICS_LIBCAbefore running hyperion Longer term: Maybe we should do the unsetting here: https://github.com/bluesky/ophyd/blob/master/ophyd/__init__.py#L39@tacaswell @danielballan @mrakitin any thoughts?
bump
from ophyd-async.
tacaswell
commented on July 25, 2024
I think we should go the other way and rely on the conda-packaged epics libs when in conda-land.
from ophyd-async.
coretl
commented on July 25, 2024
Ok how about this as a strategy:
- In ophyd we:
- Use
PYEPICS_LIBCAset to a conda provided path - Or use
epicscorelibsif it is there from pip - Or fallback to the default
pyepicsbundledlibcafor legacy pip
- Use
- In aioca we:
- Use
AIOCA_LIBCAset to a conda provided path - Or use
epicscorelibsif it is there from pip - Or fail if not
- Use
- In p4p it is either compile time dependant on
epicscorelibsor a conda provided path
The only thing I'm not sure about is whether to make epicscorelibs a dependency of aioca in pip, or whether to make people type pip install aioca[epicscorelibs] to get the bundled libs...
from ophyd-async.
mrakitin
commented on July 25, 2024
Sounds good to me. I think pip install aioca[epicscorelibs] is a bit more flexible to support libs injection from conda and override from pip if needed.
from ophyd-async.
coretl
commented on July 25, 2024
Can you make pip install aioca install epicscorelibs and conda install aioca not?
from ophyd-async.
mrakitin
commented on July 25, 2024
Yes, conda dependencies are managed independently, but we want epicscorelibs there from conda, right? We can have different variants of the package too, such as we did for bluesky: https://github.com/conda-forge/bluesky-feedstock/blob/main/recipe/meta.yaml#L26 (and similar to tiled).
from ophyd-async.
Related Issues (20)
- `AsyncStatus` takes an `Awaitable` according to type signature but rejects anything that isn't a coroutine HOT 3
- Pilatus arms too early
- Clearer documentation on how to use the reading context manager in a `StandardReadable`
- `set_mock_values` does not work with `observe_value` HOT 5
- epics_signal_rw fail to connect with typeerror when they are not set at epics HOT 1
- Pilatus arming logic does not fully wait for armed PV HOT 1
- Make `PilatusController` support Pilatus 3 deadtime HOT 1
- Allow SoftSignals metadata (units, limits etc.)
- Rename `DetectorControl.arm` to make the function of the returned status clearer
- Convert signal readings and `get_values` to pure python types
- Remove type conversion in `device_save_loader.py`
- Require units for signals and motors HOT 4
- `StandardReadable` breaks with nested hinted signals HOT 1
- Reading/configuration assert error message is the wrong way around
- Only support the formalized pvi structure from PandA HOT 1
- Drop old version of event model
- Make a PandA flyer that is implemented with a PCOMP block
- Make a motor record flyer
- Use the run and stage decorators for the plans in the tests
- Device names should be reported within all errors HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ophyd-async.