reanahub / reana-demo-atlas-recast Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 33.0 164 KB

REANA example - RECAST analysis based on ATLAS software stack

License: MIT License

CMake 10.36% C++ 48.69% C 1.19% Python 17.88% Dockerfile 2.54% Gherkin 14.10% Shell 5.24%

reana-demo-atlas-recast's People

Contributors

Stargazers

Watchers

reana-demo-atlas-recast's Issues

workflow: add kubernetes_memory_limit

Enrich example with kubernetes_memory_limit directive in order to be able to run the workflow with as little memory consumption as possible. This means:

Run the workflow as is now, and observe real memory usage via kubectl top pods or via logging into the node and using top for each job. For example, 2GiB can be consumed for one step.
Amend workflow definition to introduce kubernetes_memory_limit for each step and use value observed above.
Run example successfully on REANA and lower the value in order to have them as low as possible for each step.

This will allow ultimately to run many ATLAS RECAST demo workflows in parallel.

Offtopic help please

Im trying to make “Grid Runnig” step from ATLAS SOFTWARE TUTORIAL and I receave following error: No member named ‘scanRucio’ in namespace ‘SH’.
My steps:
setupATLAS
lsetup panda
cd …/build
asetup --restore
source x86_64-*/setup.sh
cd …/build/
cmake …/source/
make
cd …/run
export ALRB_TutorialData=/cvmfs/atlas.cern.ch/repo/tutorials/asg/cern-feb2021
-voms-proxy-init voms-atlas
Username and password, authorized
lsetup rucio
The output of above command is here:

Requested: rucio …
Setting up emi 4.0.2-1_200423.fix3 …
Skipping: grid middleware already setup (from UI)
Setting up rucio 1.25.3 …
Info: Setting compatibility to centos7
Setting up xrootd 5.1.1-x86_64-centos7 …
Skipping: Skipping xrootd for x86_64-centos7-gcc8-opt since Athena has it.
---------------------- Information for user ----------------------------
emi:
Your proxy has 11h:56m:29s remaining
Read this on when and how to setup your proxy …
http://cern.ch/go/66jM
Your RFC proxy has a nonRepudiation key.
Please report this to DAST with any site access issues
A temporary workaround is: voms-proxy-init -voms atlas -old
rucio:
Warning: An ATLAS release was setup and rucio may not work.
If you have rucio issues, retry with the wrapper version.
eg: lsetup “rucio -w”

After this I tried to start macros

root -l -b -q $ROOTCOREDIR/scripts/load_packages.C …/source/MyAnalysis/share/ATestSubmit.cxx(“myGridJob”)

Here the output from terminal:

Processing /cvmfs/atlas.cern.ch/repo/sw/software/21.2/AnalysisBase/21.2.139/InstallArea/x86_64-centos7-gcc8-opt/scripts/load_packages.C...
xAOD::Init INFO Environment initialised for data access
Processing ../source/MyAnalysis/share/ATestSubmit.cxx("myGridJob")...
In file included from input_line_16:1:
/afs/cern.ch/user/o/ogavva/ROOTAnalysisTutorial/source/MyAnalysis/share/ATestSubmit.cxx:22:7: error: no member named 'scanRucio' in namespace 'SH'
SH::scanRucio(sh, "data16_13TeV.periodAllYear.physics_Main.PhysCont.DAOD_ZMUMU.repro21_v01/");

TInterpreter::TCling::... WARNING Problems in declaring string 'namespace DataVector_detail { template <typename B1, typename B2, typename B3> class VirtBases; }' were encountered.
TInterpreter::TCling::... WARNING Problems in declaring string 'template <typename T, typename BASE> class DataVector;' were encountered.
TInterpreter::TCling::... WARNING Problems in compiling forward declarations for module libEventLoop: '
#line 1 "libEventLoop dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_Autoloading_Map;
namespace std{template <typename _T1, typename _T2> struct __attribute__((annotate("$clingAutoload$bits/stl_pair.h")))  __attribute__((annotate("$clingAutoload$string")))  pair;
}
namespace std{template <typename _Tp> class __attribute__((annotate("$clingAutoload$bits/allocator.h")))  __attribute__((annotate("$clingAutoload$string")))  allocator;
}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/Algorithm.h")))  Algorithm;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/AnaAlgorithmWrapper.h")))  AnaAlgorithmWrapper;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/Driver.h")))  __attribute__((annotate("$clingAutoload$EventLoop/BatchDriver.h")))  Driver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/BatchDriver.h")))  BatchDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/JobConfig.h")))  __attribute__((annotate("$clingAutoload$EventLoop/BatchJob.h")))  JobConfig;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/Job.h")))  __attribute__((annotate("$clingAutoload$EventLoop/BatchJob.h")))  Job;}
namespace EL{struct __attribute__((annotate("$clingAutoload$EventLoop/BatchJob.h")))  BatchJob;}
namespace EL{struct __attribute__((annotate("$clingAutoload$EventLoop/BatchSegment.h")))  BatchSegment;}
namespace EL{struct __attribute__((annotate("$clingAutoload$EventLoop/BatchSample.h")))  BatchSample;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/CondorDriver.h")))  CondorDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/DirectDriver.h")))  DirectDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/GEDriver.h")))  GEDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/KubernetesDriver.h")))  KubernetesDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/LLDriver.h")))  LLDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/LSFDriver.h")))  LSFDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/LocalDriver.h")))  LocalDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/OutputStream.h")))  OutputStream;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/MetricsSvc.h")))  MetricsSvc;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/SoGEDriver.h")))  SoGEDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/TorqueDriver.h")))  TorqueDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/VomsProxySvc.h")))  VomsProxySvc;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoop/SlurmDriver.h")))  SlurmDriver;}
class __attribute__((annotate("$clingAutoload$AsgMessaging/StatusCode.h")))  __attribute__((annotate("$clingAutoload$EventLoop/Algorithm.h")))  StatusCode;
namespace EL{typedef ::StatusCode StatusCode __attribute__((annotate("$clingAutoload$EventLoop/StatusCode.h")))  __attribute__((annotate("$clingAutoload$EventLoop/Algorithm.h"))) ;}
'
TInterpreter::AutoParse   ERROR   Error parsing payload code for class EL::Job with content:

#line 1 "libEventLoop dictionary payload"

#ifndef HAVE_PRETTY_FUNCTION
#define HAVE_PRETTY_FUNCTION 1
#endif
#ifndef HAVE_64_BITS
#define HAVE_64_BITS 1
#endif
#ifndef __IDENTIFIER_64BIT__
#define __IDENTIFIER_64BIT__ 1
#endif
#ifndef ATLAS
#define ATLAS 1
#endif
#ifndef ROOTCORE
#define ROOTCORE 1
#endif
#ifndef XAOD_STANDALONE
#define XAOD_STANDALONE 1
#endif
#ifndef XAOD_ANALYSIS
#define XAOD_ANALYSIS 1
#endif
#ifndef ROOTCORE_RELEASE_SERIES
#define ROOTCORE_RELEASE_SERIES 25
#endif
#ifndef PACKAGE_VERSION
#define PACKAGE_VERSION "EventLoop-00-00-00"
#endif
#ifndef PACKAGE_VERSION_UQ
#define PACKAGE_VERSION_UQ EventLoop-00-00-00
#endif
#ifndef USE_CMAKE
#define USE_CMAKE 1
#endif

#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#include "EventLoop/Algorithm.h"
#include "EventLoop/AnaAlgorithmWrapper.h"
#include "EventLoop/BatchDriver.h"
#include "EventLoop/BatchJob.h"
#include "EventLoop/BatchSegment.h"
#include "EventLoop/BatchSample.h"
#include "EventLoop/CondorDriver.h"
#include "EventLoop/DirectDriver.h"
#include "EventLoop/GEDriver.h"
#include "EventLoop/Job.h"
#include "EventLoop/KubernetesDriver.h"
#include "EventLoop/LLDriver.h"
#include "EventLoop/LSFDriver.h"
#include "EventLoop/LocalDriver.h"
#include "EventLoop/OutputStream.h"
#include "EventLoop/MetricsSvc.h"
#include "EventLoop/SoGEDriver.h"
#include "EventLoop/StatusCode.h"
#include "EventLoop/TorqueDriver.h"
#include "EventLoop/VomsProxySvc.h"
#include "EventLoop/SlurmDriver.h"

#undef  _BACKWARD_BACKWARD_WARNING_H

/afs/cern.ch/user/o/ogavva/ROOTAnalysisTutorial/source/MyAnalysis/share/ATestSubmit.cxx:34:7: error: no type named 'Job' in namespace 'EL'
EL::Job job;
~~~~^
/afs/cern.ch/user/o/ogavva/ROOTAnalysisTutorial/source/MyAnalysis/share/ATestSubmit.cxx:37:33: error: no member named 'Job' in namespace 'EL'
job.options()->setString( EL::Job::optSubmitDirMode, "unique-link");
                          ~~~~^
TInterpreter::TCling::... WARNING Problems in declaring string 'namespace DataVector_detail { template <typename B1, typename B2, typename B3> class VirtBases; }' were encountered.
TInterpreter::TCling::... WARNING Problems in declaring string 'template <typename T, typename BASE> class DataVector;' were encountered.
TInterpreter::TCling::... WARNING Problems in compiling forward declarations for module libAnaAlgorithmDict: '
#line 1 "libAnaAlgorithmDict dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_Autoloading_Map;
namespace EL{class __attribute__((annotate("$clingAutoload$AnaAlgorithm/AnaAlgorithm.h")))  AnaAlgorithm;}
namespace EL{class __attribute__((annotate("$clingAutoload$AnaAlgorithm/AnaAlgorithmConfig.h")))  AnaAlgorithmConfig;}
'
TInterpreter::AutoParse   ERROR   Error parsing payload code for class EL::AnaAlgorithmConfig with content:

#line 1 "libAnaAlgorithmDict dictionary payload"

#ifndef HAVE_PRETTY_FUNCTION
#define HAVE_PRETTY_FUNCTION 1
#endif
#ifndef HAVE_64_BITS
#define HAVE_64_BITS 1
#endif
#ifndef __IDENTIFIER_64BIT__
#define __IDENTIFIER_64BIT__ 1
#endif
#ifndef ATLAS
#define ATLAS 1
#endif
#ifndef ROOTCORE
#define ROOTCORE 1
#endif
#ifndef XAOD_STANDALONE
#define XAOD_STANDALONE 1
#endif
#ifndef XAOD_ANALYSIS
#define XAOD_ANALYSIS 1
#endif
#ifndef ROOTCORE_RELEASE_SERIES
#define ROOTCORE_RELEASE_SERIES 25
#endif
#ifndef PACKAGE_VERSION
#define PACKAGE_VERSION "AnaAlgorithm-00-00-00"
#endif
#ifndef PACKAGE_VERSION_UQ
#define PACKAGE_VERSION_UQ AnaAlgorithm-00-00-00
#endif
#ifndef EIGEN_DONT_VECTORIZE
#define EIGEN_DONT_VECTORIZE 1
#endif

#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#ifndef ANA_ALGORITHM__ANA_ALGORITHM_DICT_H
#define ANA_ALGORITHM__ANA_ALGORITHM_DICT_H

#ifdef __GNUC__
# pragma GCC diagnostic ignored "-Wdeprecated-declarations"
#endif

#include "AnaAlgorithm/AnaAlgorithm.h"
#ifdef ROOTCORE
#include "AnaAlgorithm/AnaAlgorithmConfig.h"
#endif

#endif

#undef  _BACKWARD_BACKWARD_WARNING_H

/afs/cern.ch/user/o/ogavva/ROOTAnalysisTutorial/source/MyAnalysis/share/ATestSubmit.cxx:40:7: error: no type named 'AnaAlgorithmConfig' in namespace 'EL'
EL::AnaAlgorithmConfig alg;
~~~~^
TInterpreter::TCling::... WARNING Problems in compiling forward declarations for module libEventLoopGrid: '
#line 1 "libEventLoopGrid dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_Autoloading_Map;
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoopGrid/GridDriver.h")))  GridDriver;}
namespace EL{class __attribute__((annotate("$clingAutoload$EventLoopGrid/PrunDriver.h")))  PrunDriver;}
'
TInterpreter::AutoParse   ERROR   Error parsing payload code for class EL::PrunDriver with content:

#line 1 "libEventLoopGrid dictionary payload"

#ifndef HAVE_PRETTY_FUNCTION
#define HAVE_PRETTY_FUNCTION 1
#endif
#ifndef HAVE_64_BITS
#define HAVE_64_BITS 1
#endif
#ifndef __IDENTIFIER_64BIT__
#define __IDENTIFIER_64BIT__ 1
#endif
#ifndef ATLAS
#define ATLAS 1
#endif
#ifndef ROOTCORE
#define ROOTCORE 1
#endif
#ifndef XAOD_STANDALONE
#define XAOD_STANDALONE 1
#endif
#ifndef XAOD_ANALYSIS
#define XAOD_ANALYSIS 1
#endif
#ifndef ROOTCORE_RELEASE_SERIES
#define ROOTCORE_RELEASE_SERIES 25
#endif
#ifndef PACKAGE_VERSION
#define PACKAGE_VERSION "EventLoopGrid-00-00-00"
#endif
#ifndef PACKAGE_VERSION_UQ
#define PACKAGE_VERSION_UQ EventLoopGrid-00-00-00
#endif
#ifndef USE_CMAKE
#define USE_CMAKE 1
#endif
#ifndef USE_CMAKE
#define USE_CMAKE 1
#endif

#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#include "EventLoopGrid/GridDriver.h"
#include "EventLoopGrid/PrunDriver.h"

#undef  _BACKWARD_BACKWARD_WARNING_H

/afs/cern.ch/user/o/ogavva/ROOTAnalysisTutorial/source/MyAnalysis/share/ATestSubmit.cxx:54:5: error: no type named 'PrunDriver' in namespace 'EL'
EL::PrunDriver driver;
~~~~^


This is my macros ATestSubmit.cxx : 

#include <SampleHandler/ToolsDiscovery.h>

void ATestSubmit (const std::string& submitDir)
{
// Set up the job for xAOD access:
xAOD::Init().ignore();

// create a new sample handler to describe the data files we use
SH::SampleHandler sh;

// scan for datasets in the given directory
// this works if you are on lxplus, otherwise you'd want to copy over files
// to your local machine and use a local path.  if you do so, make sure
// that you copy all subdirectories and point this to the directory
// containing all the files, not the subdirectories.

// use SampleHandler to scan all of the subdirectories of a directory for particular MC single file:
//const char* inputFilePath = gSystem->ExpandPathName ("$ALRB_TutorialData/mc16_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_PHYS.e6337_s3126_r10201_p4172/");
//SH::ScanDir().filePattern("DAOD_PHYS.21569875._001323.pool.root.1").scan(sh,inputFilePath);

SH::scanRucio (sh, "data16_13TeV.periodAllYear.physics_Main.PhysCont.DAOD_ZMUMU.repro21_v01/");

// set the name of the tree in our files
// in the xAOD the TTree containing the EDM containers is "CollectionTree"
sh.setMetaString ("nc_tree", "CollectionTree");

// further sample handler configuration may go here

// print out the samples we found
sh.print ();

// this is the basic description of our job
EL::Job job;
job.sampleHandler (sh); // use SampleHandler in this job

job.options()->setString( EL::Job::optSubmitDirMode, "unique-link");

// add our algorithm to the job
EL::AnaAlgorithmConfig alg;
alg.setType ("MyxAODAnalysis");

// set the name of the algorithm (this is the name use with
// messages)
alg.setName ("AnalysisAlg");

// later on we'll add some configuration options for our algorithm that go here

job.algsAdd (alg);

// make the driver we want to use:
// this one works by running the algorithm directly:
// EL::DirectDriver driver;
EL::PrunDriver driver;
// we can use other drivers to run things on the Grid, with PROOF, etc.
driver.options()->setString("nc_outputSampleName", "user.ogavva.test.%in:name[2]%.%in:name[6]%");

// process the job using the driver
driver.submit (job, submitDir);
}

no output plots

As mentioned IRL, the example stopped producing plots if one rebuilds images and reruns the example from scratch using vanilla yadage. (I.e. also outside of REANA.)

How to reproduce:

$ docker rmi atlas/analysisbase
$ cd eventselection
$ docker build -t reanahub/reana-demo-atlas-recast-eventselection . --no-cache
$ cd ../statanalysis
$ docker build -t reanahub/reana-demo-atlas-recast-statanalysis .
$ cd ..
$ mkvirtualenv yadage27 -p /usr/bin/python2.7
$ yadage-run _run workflow/workflow.yml ./workflow/test_inputs/inp1.yml

The ROOT files were produced:

$ find _run | grep root
_run/eventselection/submitDir/input/sample.root
_run/eventselection/submitDir/hist-sample.root
_run/eventselection/submitDir/hist/sample.root
_run/eventselection/submitDir/driver.root

as well as the JSON fit results:

$ find _run | grep limit
_run/statanalysis/fitresults/limit_data.json
_run/statanalysis/fitresults/limit_data_nomsignal.json

but no plots:

$ find _run | grep png

The statanalysis logs show that:

$ tail -10 _run/statanalysis/_packtivity/statanalysis.run.log
2019-11-28 11:38:51,883 | pack.statanalysis.ru |   INFO | NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE
2019-11-28 11:38:51,883 | pack.statanalysis.ru |   INFO | 1  alpha_syst1  -7.70828e-01   9.50708e-01   4.66492e-03  -4.71670e-03
2019-11-28 11:38:51,883 | pack.statanalysis.ru |   INFO | ERR DEF= 0.5
2019-11-28 11:38:51,884 | pack.statanalysis.ru |   INFO | AsymptoticCalculator::EvaluateNLL -  value = -1210.79 for poi fixed at = 2    fit time : Real time 0:00:00, CP time 0.000
2019-11-28 11:38:51,884 | pack.statanalysis.ru |   INFO | [#0] PROGRESS:Eval --          ASIMOV data qmu_A = 16.9355 condNLL = -1210.79 uncond -1219.26
2019-11-28 11:38:51,884 | pack.statanalysis.ru |   INFO | [#0] PROGRESS:Eval -- poi = 2 qmu = 9.06996 qmu_A = 16.9355 sigma = 0.485994  CLsplusb = 0.00129921 CLb = 0.865125 CLs = 665.885
2019-11-28 11:38:51,918 | pack.statanalysis.ru |   INFO | cling::DynamicLibraryManager::loadLibrary(): libtiff.so.5: cannot open shared object file: No such file or directory
2019-11-28 11:38:51,918 | pack.statanalysis.ru |   INFO | Error in <TInterpreter::TCling::AutoLoad>: failure loading library libASImage.so for TASImage
2019-11-28 11:38:51,919 | pack.statanalysis.ru |   INFO | cling::DynamicLibraryManager::loadLibrary(): libtiff.so.5: cannot open shared object file: No such file or directory
2019-11-28 11:38:52,092 | pack.statanalysis.ru |   INFO | 0.5 0.574709475331

Note the libtiff troubles.

@lukasheinrich :

(1) Is the code of the example recent and how you would like it to look? Perhaps there were some changes on the ATLAS base image side that were not propagated to the REANA repo?

(2) Can we pin the base image and all free dependencies to some particular versions? The example uses currently the latest base image:

eventselection/Dockerfile:1:FROM atlas/analysisbase:latest
statanalysis/Dockerfile:1:FROM atlas/analysisbase

as well as ian unpinned htfools library:

statanalysis/Dockerfile:3:RUN sudo sh -c "source /home/atlas/release_setup.sh && pip install hftools"

This is prone to failing sooner or later... For a nicely complete analysis preservation example, it would be good to use pinned versions of all dependencies.

Can you please have a look and advise which versions to use?

Make a 'main' branch

@tiborsimko or @lukasheinrich Can I get you to do me a favor and just do the following to get a main branch to exist (not even be the default)?

git checkout origin/master -b main && git push -u origin HEAD

The reason I want this is because in https://github.com/yadage/yadage-schemas/pull/44/files the value of the default target branch in generic_github_url got changed from master to main. This went into yadage-schemas v0.11.0 which is now getting picked up in recast-atlas causing things like https://gitlab.cern.ch/recast-atlas/examples/public-examples-tests/-/merge_requests/4 to fail because examples/rome is picking up

spec:
  toplevel: github:reanahub/reana-demo-atlas-recast
  workflow: workflow/workflow.yml

and https://github.com/reanahub/reana-demo-atlas-recast/ doens't have a main branch for it to get targeted in the github:<username/repo[@branch]>[:subpath] toplevel format.

I'm going to fix this in https://github.com/recast-hep/recast-atlas but it would be good to have this get fixed everywhere that has something like this copied as well.

serial: move auto generated files to analysis_build folder for better consistency

Related #36 (comment)

docs: update for simplified v0.3.0 reana-client commands

inputs: dxaod at CERN?

@lukasheinrich What is the path to the dxaod input file hosted at CERN? The NYU certificate seem to have changed, so davix cannot open it:

Running sample: sample
Processing File: http://physics.nyu.edu/~lh1132/capdemo/mc15_13TeV.123456.cap_recast_demo_signal_one.root
DavixOpen                 ERROR   can not open file with davix: Failure (Neon): Server certificate verification failed: issuer is not trusted after 3 attempts (6)
/build1/atnight/localbuilds/nightlies/21.2/athena/PhysicsAnalysis/D3PDTools/SampleHandler/Root/ToolsOther.cxx:68:exception: failed to open file: http://physics.nyu.edu/~lh1132/capdemo/mc15_13TeV.123456.cap_recast_demo_signal_one.root
terminate called after throwing an instance of 'RCU::ExceptionMsg'

statanalysis data files

The statistical analysis data files are located in:

$ ls -l statanalysis/data/
total 8
-rw-r--r-- 1 simko simko 3662 Jun 17 11:20 background.root
-rw-r--r-- 1 simko simko 3598 Jun 17 11:20 data.root

i.e. they are hard-coded inside the container.

@lukasheinrich Is this wanted? What about bringing them out? That might constitute a nicer example...

docs: add Launch-on-REANA badge

Note: similar to and depends on reanahub/reana-demo-helloworld#60

Add "Launch-on-REANA" badge to the README file.

workflow: scalability tests

Develop an internal tiny Python script (or a Juypyter notebook) that would accept N as an argument and would launch N-times the demo workflow and would collect and print histogram of running times.

For example, running 30 workflows locally on my laptop gives:

$ for i in {1..30}; reana-client run -w test64mi
...
$ reana-client list --filter name=test64mi
NAME       RUN_NUMBER   CREATED               STARTED               ENDED                 STATUS  
test64mi   30           2021-09-23T08:42:49   2021-09-23T08:43:27   2021-09-23T08:45:53   finished
test64mi   29           2021-09-23T08:42:48   2021-09-23T08:43:25   2021-09-23T08:45:55   finished
test64mi   28           2021-09-23T08:42:46   2021-09-23T08:43:23   2021-09-23T08:45:51   finished
test64mi   27           2021-09-23T08:42:45   2021-09-23T08:43:21   2021-09-23T08:45:51   finished
test64mi   26           2021-09-23T08:42:44   2021-09-23T08:43:18   2021-09-23T08:46:00   finished
test64mi   25           2021-09-23T08:42:42   2021-09-23T08:43:16   2021-09-23T08:45:55   finished
test64mi   24           2021-09-23T08:42:41   2021-09-23T08:43:14   2021-09-23T08:45:48   finished
test64mi   23           2021-09-23T08:42:40   2021-09-23T08:43:12   2021-09-23T08:45:49   finished
test64mi   22           2021-09-23T08:42:38   2021-09-23T08:43:10   2021-09-23T08:45:50   finished
test64mi   21           2021-09-23T08:42:37   2021-09-23T08:43:08   2021-09-23T08:45:49   finished
test64mi   20           2021-09-23T08:42:36   2021-09-23T08:43:06   2021-09-23T08:45:39   finished
test64mi   19           2021-09-23T08:42:35   2021-09-23T08:43:04   2021-09-23T08:45:36   finished
test64mi   18           2021-09-23T08:42:33   2021-09-23T08:43:01   2021-09-23T08:45:30   finished
test64mi   17           2021-09-23T08:42:32   2021-09-23T08:42:59   2021-09-23T08:45:58   finished
test64mi   16           2021-09-23T08:42:31   2021-09-23T08:42:57   2021-09-23T08:45:50   finished
test64mi   15           2021-09-23T08:42:30   2021-09-23T08:42:55   2021-09-23T08:45:38   finished
test64mi   14           2021-09-23T08:42:29   2021-09-23T08:42:53   2021-09-23T08:45:48   finished
test64mi   13           2021-09-23T08:42:28   2021-09-23T08:42:51   2021-09-23T08:45:45   finished
test64mi   12           2021-09-23T08:42:27   2021-09-23T08:42:49   2021-09-23T08:45:40   finished
test64mi   11           2021-09-23T08:42:25   2021-09-23T08:42:47   2021-09-23T08:45:47   finished
test64mi   10           2021-09-23T08:42:24   2021-09-23T08:42:45   2021-09-23T08:45:04   finished
test64mi   9            2021-09-23T08:42:23   2021-09-23T08:42:43   2021-09-23T08:45:09   finished
test64mi   8            2021-09-23T08:42:22   2021-09-23T08:42:41   2021-09-23T08:44:55   finished
test64mi   7            2021-09-23T08:42:21   2021-09-23T08:42:39   2021-09-23T08:45:28   finished
test64mi   6            2021-09-23T08:42:20   2021-09-23T08:42:37   2021-09-23T08:44:24   finished
test64mi   5            2021-09-23T08:42:19   2021-09-23T08:42:34   2021-09-23T08:43:55   finished
test64mi   4            2021-09-23T08:42:18   2021-09-23T08:42:33   2021-09-23T08:43:43   finished
test64mi   3            2021-09-23T08:42:17   2021-09-23T08:42:30   2021-09-23T08:43:36   finished
test64mi   2            2021-09-23T08:42:16   2021-09-23T08:42:28   2021-09-23T08:43:44   finished
test64mi   1            2021-09-23T08:42:14   2021-09-23T08:42:26   2021-09-23T08:45:47   finished

It would be useful to collect and output the number of successfully finished workflows, the number of failed workflows, teh lowest runtime, the mean runtime, the median runtime, the longest runtime, basically a nice statistic graph with the runtime distribution, for example bar graph when given run was submitted and how long it took whilst the x-axis is the run number, and a overall histogram of runtimes where x-axis would be run time.

This script could then be used to launch 10, 100, 500, 1000 workflows and collect the information about the scalability of the system.

Note: the script does not have to do anything with reana-demo-atlas-recast per se, so we could use it for other demo repositories too. It could therefore be committed to the reana repository as a reana-dev run-example ... option or a new command there...

workflow: develop serial workflow for debugging

Similarly as for reana-demo-cms-h4l, let's develop reana-debug.yaml that would be the Serial representation of Yadage workflow. Basically, it would launch commands from workflow/steps.yml in sequential order.

This will be useful for debugging purposes e.g. to test any scalability issues associated with r-w-e-yadage engine vs those with the platform itself.

docker: non-root user

The example stopped working with latest REANA (that is not compatible with environments forcing root users) because of:

> git grep USER
eventselection/Dockerfile:USER root
statanalysis/Dockerfile:USER root

We can now switch back to using uid=501 or uid=1000 with gid=0 and/or not forcing any user.

docs: simplification

Simplify main README.rst, propagate local CWL/Yadage usage instructions to corresponding yaml files, amend inputs/workflow/outputs order in reana.yaml files, remove author emails, see reanahub/reana-demo-worldpopulation#25 for example of things to do.

local docker run showing error 139

FYI I was seeing troubles running ATLAS analysis base image on my system:

$ docker pull atlas/analysisbase
$ docker run -i -t --rm atlas/analysisbase /bin/sh
139

This is basically the same problem as described in docker/for-linux#58 related to CentOS6 images.

I had to boot with vsyscall=emulate to make it work.

Posting just in case somebody encounters the same problem.

statanalysis example results

The output examples are located in the statistical analysis directory:

$ ls -l statanalysis/example_results/
total 48
-rw-r--r-- 1 simko simko   174 Jun 17 11:20 data_limit.json
-rw-r--r-- 1 simko simko   181 Jun 17 11:20 data_limit_muone.json
-rw-r--r-- 1 simko simko 14477 Jun 17 11:20 plot_limit.png
-rw-r--r-- 1 simko simko 10095 Jun 17 11:20 post.png
-rw-r--r-- 1 simko simko 10007 Jun 17 11:20 pre.png

@lukasheinrich What about moving them to the main docs directory as is the usual for REANA examples? Do you prefer to keep them inside in order to better emulate the multi-repo setup used for exotics decat searches?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.