areaDetector driver for GenICam cameras using the Aravis library on Linux.
Home Page: https://areadetector.github.io/master/ADAravis/ADAravis.html
An EPICS areaDetector driver for GenICam cameras using the Aravis library.
Additional information:
Hello,
When using the ADAravis module with a Prosilica GC655C camera, all parameters work as expectedm with the exception of the gain. Changing the PV value has no effect on the image. No errors appear on the IOC console, and the camera continues to work without issue, but the gain value is ignored.
When using the aravisGigE module with the same camera, gain is controllable.
Thanks.
James
Hi, @MarkRivers !
I have found that when we power cycle a GigE camera, the reset process seems to work and it can acquire frames, but most of the parameters do not work. They get reset to 0 / No / N.A., so we can't see what the values on the camera are, and setting things has no effect, for example AcquireTime, although the readbacks update in the IOC.
Is this something you have seen before?
We are running a version based on R2-1
On a couple of our servers (about 10-15 camera instances each), we noticed memory usage climbing, eventually swapping out and causing high CPU usage.
These servers are currently running the legacy areadetector-aravis driver 3.0, with libaravis 0.6. Valgrind shows the following leak which grows with time:
==3187580== 13,176 bytes in 549 blocks are definitely lost in loss record 12,970 of 13,239
==3187580== at 0x483877F: malloc (vg_replace_malloc.c:307)
==3187580== by 0x924AD48: g_malloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6600.8)
==3187580== by 0x5F81A45: arv_gc_enumeration_get_available_int_values (in /usr/lib/x86_64-linux-gnu/libaravis-0.6.so.0.0.0)
==3187580== by 0x5F8C87B: arv_device_get_available_enumeration_feature_values (in /usr/lib/x86_64-linux-gnu/libaravis-0.6.so.0.0.0)
==3187580== by 0x48A8320: aravisCamera::runScanner() (aravisCamera.cpp:961)
==3187580== by 0x48AE981: aravisCamera::FeatureScanner::run() (aravisCamera.cpp:254)
==3187580== by 0x4DBCBA9: epicsThreadCallEntryPoint (epicsThread.cpp:83)
==3187580== by 0x4DC5679: start_routine (osdThread.c:403)
==3187580== by 0x6031EA6: start_thread (pthread_create.c:477)
==3187580== by 0x5261DEE: clone (clone.S:95)
I just pulled in the latest master versions of this and ADGenICam, and upgraded libaravis to 0.8.6. Now, I get a similar leak which might be growing with WriteInt32 calls (in my brief test, 5200 bytes lost with just starting an IOC, and 6,240 lost if I change some int parameters):
==2232427== 6,240 bytes in 336 blocks are definitely lost in loss record 5,447 of 5,625
==2232427== at 0x483877F: malloc (vg_replace_malloc.c:307)
==2232427== by 0x60A0D48: g_malloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6600.8)
==2232427== by 0x5F9E01D: arv_gc_enumeration_dup_available_int_values (in /usr/lib/x86_64-linux-gnu/libaravis-0.8.so.0.8.6)
==2232427== by 0x4890068: arvFeature::readEnumChoices(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<int, std::allocator<int> >&) (arvFeature.cpp:137)
==2232427== by 0x5F56EA4: GenICamFeature::read(void*, bool) (GenICamFeature.cpp:418)
==2232427== by 0x5F57CD6: GenICamFeatureSet::readAll() (GenICamFeature.cpp:616)
==2232427== by 0x5F5E6F4: ADGenICam::writeInt32(asynUser*, int) (ADGenICam.cpp:137)
==2232427== by 0x4893B84: ADAravis::writeInt32(asynUser*, int) (ADAravis.cpp:584)
==2232427== by 0x4B8342C: writeInt32 (asynPortDriver.cpp:1996)
==2232427== by 0x4B98744: processCallbackOutput (devAsynInt32.c:528)
==2232427== by 0x4B63F6C: portThread (asynManager.c:913)
==2232427== by 0x4D96679: start_routine (osdThread.c:403)
I am not sure if this is actually the primary or only cause of dwindling memory. But in any case, I see in arvFeature.cpp:137, we create array gint64 *values, with arv_gc_enumeration_dup_available_string_values. values are then pushed into a vector. Then, another similar array strings is g_free()d, but values is not.
Now I get lost, because for reasons not entirely clear to me, if I do g_free(values) and try to acquire images from the camera,
the driver just passes along empty buffers:
2023/03/21 14:04:38.743 ADAravis:processBuffer: w: 1920, h: 1200, size: 0, expected_size: 4608000
I'm getting these messages in IOC shell when trying to talk to FLIR BFS-PGE-70S7M camera:
2020/09/15 12:24:22.169 Param[ACQ_PERIOD] GenICamFeature::write: node AcquisitionFrameRate is not writable
2020/09/15 12:24:35.885 Param[GC_FRAMERATE] GenICamFeature::write: node AcquisitionFrameRate is not writable
2020/09/15 12:25:27.845 ADAravis:writeInt32 error, status=3 function=95 GC_FRAMERATE_ENABLE, value=0
2020/09/15 12:25:30.945 Param[GC_FRAMERATE] GenICamFeature::write: node AcquisitionFrameRate is not writable
2020/09/15 12:26:23.230 Param[GC_D_ExposureTime] GenICamFeature::write: node ExposureTime is not writable
2020/09/15 12:26:30.109 ADAravis:writeInt32 error, status=3 function=165 GC_E_ExposureMode, value=2
2020/09/15 12:26:30.109 LAB:FLIR1:GC_ExposureMode devAsynInt32::processCallbackOutput process write error
2020/09/15 12:26:32.195 Param[GC_D_ExposureTime] GenICamFeature::write: node ExposureTime is not writable
2020/09/15 12:26:49.272 ADAravis:writeInt32 error, status=3 function=424 GC_C_TriggerSoftware, value=0
2020/09/15 12:26:49.272 LAB:FLIR1:GC_TriggerSoftware devAsynInt32::processCallbackOutput process write error
2020/09/15 12:27:00.478 Param[GC_D_AcquisitionFrameRate] GenICamFeature::write: node AcquisitionFrameRate is not writable
2020/09/15 12:27:07.245 Param[GC_D_AcquisitionFrameRate] GenICamFeature::write: node AcquisitionFrameRate is not writable
2020/09/15 12:27:09.614 Param[GC_D_AcquisitionFrameRate] GenICamFeature::write: node AcquisitionFrameRate is not writable
Not sure why these parameters would not be writable. Aravis lib is tag ARAVIS_0_7_2, if it matters.
Attached below.
I used epics base 7.0.4
ADAravis R2-2.
The error is pointing to missing decompressMono12p and decompressMono12Packed function call.
I suspect I missed certain library reference in configuration setup.
thanks for help.
==============================================================================
make -C ./configure install
make[1]: Entering directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/configure'
make -C O.linux-x86_64 -f ../Makefile TOP=../.. \
T_A=linux-x86_64 install
make[2]: Entering directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/configure/O.linux-x86_64'
perl -CSD /opt/epics/base-7.0.4/bin/linux-x86_64/convertRelease.pl checkRelease
make[2]: Leaving directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/configure/O.linux-x86_64'
make -C O.linux-x86_64-debug -f ../Makefile TOP=../.. \
T_A=linux-x86_64-debug install
make[2]: Entering directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/configure/O.linux-x86_64-debug'
perl -CSD /opt/epics/base-7.0.4/bin/linux-x86_64/convertRelease.pl checkRelease
make[2]: Leaving directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/configure/O.linux-x86_64-debug'
make[1]: Leaving directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/configure'
make -C ./aravisApp install
make[1]: Entering directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/aravisApp'
make -C ./src install
make[2]: Entering directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/aravisApp/src'
make -C O.linux-x86_64 -f ../Makefile TOP=../../.. \
T_A=linux-x86_64 install
make[3]: Entering directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/aravisApp/src/O.linux-x86_64'
/usr/bin/g++ -D_GNU_SOURCE -D_DEFAULT_SOURCE -D_X86_64_ -DUNIX -Dlinux -O3 -Wall -mtune=generic -m64 -fPIC -I. -I../O.Common -I. -I. -I.. -I../../../include/compiler/gcc -I../../../include/os/Linux -I../../../include -I/opt/epics/modules/synApps_6_1_epics7/support/asyn-R4-38/include -I/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADSupport/include/os/Linux -I/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADSupport/include -I/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADCore/include -I/opt/epics/base-7.0.4/include/compiler/gcc -I/opt/epics/base-7.0.4/include/os/Linux -I/opt/epics/base-7.0.4/include -I/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADGenICam/include -I/usr/include/glib-2.0/glib/ -I/usr/include/glib-2.0/ -I/usr/lib/x86_64-linux-gnu/glib-2.0/include/ -I/usr/local/include/aravis-0.8 -c ../ADAravis.cpp
../ADAravis.cpp: In member function ‘asynStatus ADAravis::processBuffer(ArvBuffer*)’:
../ADAravis.cpp:761:13: error: ‘decompressMono12p’ was not declared in this scope
decompressMono12p(width*height, leftShift, (epicsUInt8 *)pIn->pData, (epicsUInt16 *)pRaw->pData);
^~~~~~~~~~~~~~~~~
../ADAravis.cpp:763:13: error: ‘decompressMono12Packed’ was not declared in this scope
decompressMono12Packed(width*height, leftShift, (epicsUInt8 *)pIn->pData, (epicsUInt16 *)pRaw->pData);
^~~~~~~~~~~~~~~~~~~~~~
/opt/epics/base-7.0.4/configure/RULES_BUILD:248: recipe for target 'ADAravis.o' failed
make[3]: *** [ADAravis.o] Error 1
make[3]: Leaving directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/aravisApp/src/O.linux-x86_64'
/opt/epics/base-7.0.4/configure/RULES_ARCHS:58: recipe for target 'install.linux-x86_64' failed
make[2]: *** [install.linux-x86_64] Error 2
make[2]: Leaving directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/aravisApp/src'
/opt/epics/base-7.0.4/configure/RULES_DIRS:85: recipe for target 'src.install' failed
make[1]: *** [src.install] Error 2
make[1]: Leaving directory '/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADAravis/aravisApp'
/opt/epics/base-7.0.4/configure/RULES_DIRS:85: recipe for target 'aravisApp.install' failed
make: *** [aravisApp.install] Error 2
Hi,
I'm sure this isn't an issue more just my lack of understanding so feel free to close. I noticed you've started supporting aravis-0.8, from what I can tell this version hasn't been released yet. Am i mistaken or are you using a beta release or something?
On our Imaging Source cameras, hitting ARResetCamera when the camera is already initialized and connected results in a degraded state where the camera cannot be controlled. The only remedy I have found at this point is to restart the camera.
to reproduce:
aravisConfig("ARV1", "The Imaging Source Europe GmbH-36510395")
ADAravis: Looking for camera 'The Imaging Source Europe GmbH-36510395'...
ADAravis: Your tick frequency is 1000000
So your timestamp resolution is 1000.000000 ns
epics> dbpf test-cam:ARResetCamera 1
ADAravis: Looking for camera 'The Imaging Source Europe GmbH-36510395'...
I am a GigEVision device.
2023/04/28 14:11:25.791 ADAravis:makeCameraObject: Another client has control of this camera.
2023/04/28 14:11:25.791 ADAravis:writeInt32 error, status=3 function=115 ARAVIS_RESET, value=1
2023/04/28 14:11:25.791 FE_SCS1:VD_D0739:cam1:ARResetCamera devAsynInt32::processCallbackOutput process write error
2023/04/28 14:11:25.792 FE_SCS1:VD_D0739:cam1:ARConnectCamera devAsynInt32::processCallbackOutput process write error
2023/04/28 14:11:26.322 ADAravis:writeInt32 error, status=3 function=90 READ_STATUS, value=1
2023/04/28 14:11:26.322 FE_SCS1:VD_D0739:cam1:ReadStatus devAsynInt32::processCallbackOutput process write error
2023/04/28 14:11:27.322 ADAravis:writeInt32 error, status=3 function=90 READ_STATUS, value=1
...
The cause (I think): the GigE channel control privilege register is not reset when ARResetCamera is toggled and makeCameraObject() is called. Therefore, the new device instance comes back without control of the camera.
Proposed solution: relinquish control of the channel if we have an existing camera, that camera is a GigEVision device, and we have client control of the device.
From d84d90db2ead73c53ad6e157c2e5f15dc8daa43c Mon Sep 17 00:00:00 2001
From: Evan Daykin <[email protected]>
Date: Fri, 28 Apr 2023 14:39:56 -0400
Subject: [PATCH 1/1] Fix: Relinquish channel control on reset
---
aravisApp/src/ADAravis.cpp | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/aravisApp/src/ADAravis.cpp b/aravisApp/src/ADAravis.cpp
index 72fcdef..feb2068 100644
--- a/aravisApp/src/ADAravis.cpp
+++ b/aravisApp/src/ADAravis.cpp
@@ -398,7 +398,20 @@ asynStatus ADAravis::makeCameraObject() {
GErrorHelper err;
/* remove old camera if it exists */
+ /* relinquish CCP before unreferencing it */
if (this->camera != NULL) {
+ if(arv_camera_is_gv_device(this->camera) && arv_gv_device_is_controller(ARV_GV_DEVICE(this->device))){
+ if (!(arv_gv_device_leave_control(ARV_GV_DEVICE(this->device),err.get()))){
+ asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,
+ "%s:%s: Current ArvGvDevice has control, but control couldn't be relinquished. err=%s\n",
+ driverName, functionName, err->message);
+ }
+ else {
+ asynPrint(this->pasynUserSelf, ASYN_TRACE_FLOW,
+ "%s:%s: Relinquished control of camera.\n",
+ driverName, functionName);
+ }
+ }
g_object_unref(this->camera);
this->camera = NULL;
}
--
2.30.2
In ntndArrayConverter::fromDataTimeStamp() we find
// pvAccess uses Posix time, NDArray uses EPICS time, need to convert
seconds += POSIX_TIME_AT_EPICS_EPOCH;
however, ADAravis passes the timestamp from the camera unmodified. Since there are probably not many cameras out there using timestamps based on the EPICS epoch this is guaranteed to be wrong. I would therefore suggest to let ADAravis convert the camera timestamp into an EPICS timestamp (by subtracting POSIX_TIME_AT_EPICS_EPOCH).
A bigger problem, unfortunately, is that GigE-Vision 2.0 cameras actually support PTP timestamps which use TAI; therefore, ADAravis should (at least optionally, if the camera supports PTP) convert TAI -> EPICS Time. The problem is that I know of no portable, efficient and automatic way to do that (while using a hack such as a global parameter that holds the 'current' offset and ignoring the fact that during a leap-second there my be inconsistent time stamps is easy).
Note that ADSpinnaker and ADVimba are also affected (I have seen that both vendors advertise GigE cameras with PTP support).
I have seen a problem with the Oryx camera. I set the ImageMode to Multiple. But then sometimes when I change some other features it changes to "Continuous" mode, and I need to select "Multiple" again before starting acquisition.
This is a bug and needs to be tracked down.
I was browsing the code and noticed that there are static (i.e., global) variables nConsecutiveBadFrames and nBadFramesPrior. Probably not a huge problem in practice but I thought I bring it to your attention as it may cause confusion if multiple devices are in use and experiencing problems.
It would be helpful if aravis library errors could be propagated and printed instead of a unhelpfully generic message like eg. "Making stream failed".
ADAravis/aravisApp/src/ADAravis.cpp
Lines 435 to 439 in 45ec346
This would entail passing a GError** as the last argument of arv_camera_create_stream(), and then printing GError::message.
GError* err = NULL;
this->stream = arv_camera_create_stream (this->camera, NULL, NULL, &err);
if (this->stream == NULL) {
asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,
"%s:%s: Making stream failed, retrying in 5s... : %s\n",
driverName, functionName, err->message);
g_error_free(err); GError* need to be freed with g_error_free(). cf. GErrorHelper is a c++ish way to ensure that this is done: https://github.com/mdavidsaver/aravisGigE/blob/d90a9995886b3d7590a17f5ebf4e28886f7ed2dd/aravisGigEApp/src/ghelper.h#L50
GErrorHelper err;
this->stream = arv_camera_create_stream (this->camera, NULL, NULL, err.get());
if (this->stream == NULL) {
asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,
"%s:%s: Making stream failed, retrying in 5s... : %s\n",
driverName, functionName, err->message);Discovered by testing fix for #18 (not related at all, to be clear):
Environment:
Arch: linux-x86_64-debug
EPICS Base 7.0.7
ADGenICam R1-8, master commit 639aa4f
ADAravis R2-3, master commit ebca7d0
Camera: The Imaging Source Europe DMK 33GX174, via gigabit ethernet
Additional Plugins: Only NDStdArrays Image plugin for viewing output array in CS-Studio
Steps to reproduce:
arv-tool-0.8 genicam per the documentationaravisConfig(portName, aravisCameraName)
Result:
ArraySize_RBV Remains zero. In IOC, the following output occurs:
2023/03/21 15:30:00.404 ADAravis:processBuffer: w: 1920, h: 1200, size: 0, expected_size: 4608000
If ImageMode is set to 'Continuous' the error goes away.
I'm trying to figure out what is happening to the image data from the FLIR blackfly S BFS PGE 70S7M detector. Every now and then there seems to appear a variable amount of rows of corrupt pixels that can be observed in the captured images.
Detector is producing 3208x2200 image in Mono16 format. For these tests I've used the built-in test pattern generator of the detector. Similar results were observed without using test pattern, with the actual images taken.
Corrupted:
Fine:
These were produced by a BOB OPI that shows the StdArray ArrayData in the image widget. I can reproduce bad captures quite easily ; takes a couple of acquisitions to get a bad one. That is with single image mode. If running in continuous mode, 1 sec acquire period the I can acquire 100+ images and all would be OK (the bad ones are more rare, but they do appear).
The color pattern and image ticks are kind of screwed up ; that is probably phoebus issue.
In order to eliminate the widget / BOB OPI / phoebus, I created a file saving plugin (FileRaw) that dumps the raw pixel data into a file (top right OPI on the above screenshots). The result of these dumps was then loaded into gimp2.10 that can handle raw image data with 16bit pixels.
Here is the result:
This makes me think the problem is not in the phoebus/OPI.
Next, I used arv-camera-test test CLI tool from aravis-0.8.1 to see if I can capture the same faulty rows. I added the pixels data dump to that utility as well, to create raw binary files (same as with FileRaw plugin).
I ran 10 acquisitions, 1 second apart, 5x times (50 images in total). Here is the CLI output of one acquisition for reference:
Looking for the first available camera
vendor name = FLIR
model name = Blackfly S BFS-PGE-70S7M
device id = 20177339
image width = 3208
image height = 2200
horizontal binning = 1
vertical binning = 1
payload = 14115200 bytes
exposure = 2003 µs
gain = 0 dB
gv n_stream channels = 1
gv current channel = 0
gv packet delay = 1000 ns
gv packet size = 8985 bytes
image 14115200 bytes, pixel format 01100007
pixel format ARV_PIXEL_FORMAT_MONO_16
wrote 14115200 bytes to file 1600947290_90019.data
1 frame/s - 14.1 MiB/s
Completed buffers = 1
Failures = 0
Underruns = 0
I loaded the files from these acquisitions into gimp as well. None of them showed any faulty rows.
I'm not sure what to conclude on where the issue is. I've never had EPICS / AD behave like this with other detectors. What is different for me is this particular detector itself and the latest ADGenICam based ADAravis, aravis 0.8 software stack I'm using these days. Sadly, I have not used this software stack with any other detector so far hence I'm not sure if this is at all an AD issue.
It seems that the aravis transport is fine from what I've tested so far. That being said, I'm not sure how EPICS / AD could have messed up the image pixel data since the raw data plugin is not even moving data out of IOC, no PVs involved, not network.
Any pointers on how to narrow the origin of the faulty rows is greatly appreciated.
PixelFormat Mono12Packed or Mono12p allow using the full resolution of 12-bit or 10-bit cameras using only 75% of the bandwidth of Mono16. For many cameras these formats lets them run faster than with PixelFormat=Mono16. For example the FLIR Oryx ORX-10G-51S5M can do 130 frames/s with AdcBitWidth=10 and PixelFormat=Mono12Packed, but only 89 frames/s with PixelFormat=Mono16.
We have a couple cameras in high-radiation areas which fairly regularly need to be restarted due to radiation corrupting the RAM. In #12 , it's correctly noted that after power-cycling the 'reset camera' button must be pressed. Is there existing functionality for this to happen automatically? If not, on disconnection, is there a reliable connection state indicator?
I would like to create a polling thread to monitor this indicator, and on a transition from connected -> disconnected, begin trying to re-initialize every 5 seconds. Or would this be more straight forward to implement in database logic?
One of the cameras we are working with uses 14 bit data type to deliver pixel data after converting raw readout to temperature. With that said we did not achieve something that works with 8/16 it data types. Uncommenting the following line gives success:
ADAravis/aravisApp/src/ADAravis.cpp
Line 93 in 1455cc4
Is the a way to get away without maintaining this it in a separate branch/fork?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.