Comments (7)
The DAG needs "file sensors" to check whether the mask and geometry files already exist (and trigger fetch_mask
and/or fetch_geom
if they do not).
from btx.
From @valmar:
Ok, so basically, what we need to do is pull the following files from the website:
https://www.ondamonitor.com/html/files/lcls/{hutch}/run_om.sh
https://www.ondamonitor.com/html/files/lcls/{hutch}/monitor.yaml
Where hutch is mfx or cxi.
We must then pull the latest geometry and mask from btx
from btx.
From @valmar:
We can work on these files, because they currently contain "default" nodes. For example, run_om.sh for cxi has the following nodes: daq-mfx-mon02,daq-mfx-mon03,daq-mfx-mon04,daq-mfx-mon05
These are usually OK! I can make an empty template and we could fill them in with the correct nodes. Or we could just use these if they are usually OK!
The main problem I see is this: in order to get the available nodes, we must run wherepsana but we must run it on the DAQ machine (say cxi-daq), which might be a problem
One thing we could do is have this information "somewhere" on the network where AirFlow can get it, like the queue as we discussed yesterday
from btx.
After discussing with @valmar , Murali, Wilko and Thorsten this morning, it became apparent that this use case was not adapted to Airflow DAG runs triggered from the eLog, as it requires web access, and targets the beamline operator for localized tasks rather than a general user task that could be run across multiple HPC.
Instead, what needs to be implemented is:
- cron jobs on pslogin to regularly git pull
btx
,mrxv
andomdevteam.github.io
to/cds/sw/package/autosfx/
- a simple script that can be run from
${hutch}-daq
by${hutch}opr
to:- create
/cds/home/opr/${hutch}opr/OM-GUI/${experiment}/om-workspace
- copy
monitor.yaml
andrun_om.sh
from/cds/sw/package/autosfx/omdevteam.github.io/html/files/lcls/${hutch}
- get the latest mask and geometry from
/cds/sw/package/autosfx/mrxv/
for the relevant detector. - (bonus) run
wherepsana
and updaterun_om.sh
with the corresponding monitoring nodes.
- create
from btx.
cron job now setup:
[fpoitevi@pslogin01 btx]$ ./scripts/pull_repos_cron_pslogin.sh
# Date: Fri Apr 22 18:15:32 PDT 2022 | User: fpoitevi | Location: /cds/sw/package/autosfx/btx
Pulling to /cds/sw/package/autosfx/btx
Already up-to-date.
Pulling to /cds/sw/package/autosfx/mrxv
Already up-to-date.
Pulling to /cds/sw/package/autosfx/omdevteam.github.io
Already up-to-date.
[fpoitevi@pslogin01 btx]$ crontab -l
@daily /cds/sw/package/autosfx/btx/scripts/pull_repos_cron_pslogin.sh >> /cds/sw/package/autosfx/btx/cronjob.log 2>&1
from btx.
An initial attempt at automation for the second task above was drafted in #55
However, this will probably not be used in a fully automated fashion in production, due to unpredictable issues like the one we just encountered where the monitoring nodes were weirdly defined in the DAQ config file, resulting in MPI connection weirdness that could have impacted OM's performances.
from btx.
Cron job setup on S3DF as well:
[fpoitevi@sdflogin003 scripts]$ crontab -l
@daily /sdf/group/lcls/ds/tools/btx/scripts/pull_repos_cron_pslogin.sh >> /sdf/group/lcls/ds/tools/btx/cronjob.log 2>&1
from btx.
Related Issues (20)
- `PiPCA` documentation
- PiPCA: Adding binning/downsampling option in `get_images()`
- PiPCA: timing bottlenecks
- Absolutely. There are conventions that should be consistent within the class regarding private/exposed methods and parameters, which can be added in future updates.
- Add maxevents to timetool analysis
- Centralized PV library? HOT 1
- Detector geometry: flag or update when camera length changes
- Detector geometry: refinement strategies HOT 1
- Crystal indexing: theory and implementations
- Script to generate a default yaml HOT 4
- Simple interface for the config to generate small data files
- Job summary tasks
- `run_analysis` fails if mask is not provided
- Airflow integration on s3df HOT 1
- `TypeError` when an event does not have an ebeamPhotonEnergy value HOT 1
- rename `timetool_diagnostics`
- enhance and test `ih5` everywhere
- Fix airflow for psffb after confirmation of succesful s3df integration
- `retrieve_pixel_index_map` not working for pnCCD detector HOT 1
- Change logging level as a command line argument
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from btx.