Notes on a conversation with @MikeHart85:

It would be nice if these components were usable as:

1. Bash scripts that you can run directly inside a VM, such as a CI VM, or even on a local machine if you want.
2. Containers
3. Pods

In particular encoding the low-level steps as

buildah run ...
buildah run ...
buildah run ...

does not seem to add anything compared to a single

buildah run ... bash_script_with_the_actual_steps.sh

I followed all instructions to build and launch everything, when running bash launch_bluesky_headless.sh I see the following error:

mhanwell@unobtanium ~/src/bluesky-pods (main) $ bash launch_bluesky_headless.sh
+ '[' '' '!=' '' ']'
+ imagename=bluesky
++ pwd
+ podman run --pod acquisition -ti --rm -v /home/mhanwell/src/bluesky-pods:/app -w /app -v ./bluesky_config/ipython:/usr/local/share/ipython -v ./bluesky_config/databroker:/usr/local/share/intake -v ./bluesky_config/happi:/usr/local/share/happi -e EPICS_CA_ADDR_LIST=10.0.2.255 -e EPICS_CA_AUTO_ADDR_LIST=no bluesky ipython3 --ipython-dir=/usr/local/share/ipython
Python 3.8.5 (default, Aug 12 2020, 00:00:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.
[TerminalIPythonApp] WARNING | Unknown error in handling startup files:
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/usr/lib/python3.8/site-packages/IPython/core/shellapp.py in _exec_file(self, fname, shell_futures)
    335                     else:
    336                         # default to python, even without extension
--> 337                         self.shell.safe_execfile(full_filename,
    338                                                  self.shell.user_ns,
    339                                                  shell_futures=shell_futures,

/usr/lib/python3.8/site-packages/IPython/core/interactiveshell.py in safe_execfile(self, fname, exit_ignore, raise_exceptions, shell_futures, *where)
   2718             try:
   2719                 glob, loc = (where + (None, ))[:2]
-> 2720                 py3compat.execfile(
   2721                     fname, glob, loc,
   2722                     self.compile if shell_futures else None)

/usr/lib/python3.8/site-packages/IPython/utils/py3compat.py in execfile(fname, glob, loc, compiler)
    166     with open(fname, 'rb') as f:
    167         compiler = compiler or compile
--> 168         exec(compiler(f.read(), fname, 'exec'), glob, loc)
    169 
    170 # Refactor print statements in doctests.

/usr/local/share/ipython/profile_default/startup/00-base.py in <module>
     19 from bluesky_adaptive.per_start import adaptive_plan
     20 
---> 21 from bluesky_queueserver.plan import configure_plan
     22 
     23 import databroker

ModuleNotFoundError: No module named 'bluesky_queueserver.plan'

We should at least add a table to the readme of what containers are expected to be spun up, how they interact with each other, and what images they depend on. I can imagine the majority of applications will just execute the bash script and poke around, but a few may want to do some repurposing or extension. Even walking through the compose file, it took a bit of time for me to understand what was getting spun up, and from where.

Both ADsim and the data-replay IOC.

So that we can write files on one side and read them on the other.

This is a longer-term concern and is not on the critical path for our MVP.

Just to record some thoughts from separate conversations I've had with @stuartcampbell and @tacaswell:

We currently have a couple very large pods. This meets our needs at present and works quite well, so I'm no great hurry to change it, but I think we should consider restructuring it in the future.

1. We like to use tools the way they a meant to be used. We are abusing the pod abstraction here by stuffing so many services into one pod. To zeroth order, a pod should have one container. To first order, pods can have a container and additional containers running support services whose data stays local to the pod (i.e. worker processes, an nginx proxy). Large services like MongoDB or Kafka whose data is of interest to multiple other services should in general get their own pods. Some external validation on that opinion:

   The primary purpose of a multi-container Pod is to support co-located, co-managed helper processes for a primary application.

   Source: https://www.mirantis.com/blog/multi-container-pods-and-container-communication-in-kubernetes/

   You might say our current structure is using "pod" to mean "private network" rather than "pod".

2. When pods are small, restarting one is less disruptive to the rest of the system.

3. If we ever want to build something comparable in production (which is still very much an open question) we will definitely not want large pods because we'll want the large services on dedicated nodes. To maintain correspondence between dev and production, we should use a more idiomatic pod structure.

One possible pod grouping:

• Beamline Pod: {IOCs + queue server parts + local redis}
• Message Broker Pod: {zookeeper + kafka}
• Database Pod {mongo + mongoconsumer}
• Data Broker Server Pod {databroker server + nginx}

So each service can drop the fragemnet it needs there rather than all of the config living in the nginx role.

https://github.com/ornl-epics/dbwr so we can see what PVs are inside.

archiver, saverestore, etc

https://docs.redpanda.com/current/get-started/quick-start/

It looks like we can replace zookeeper + kafka with a single container (which would reduce some of the random startup issues) and get a web UI to look at them to boot.

We do use librdkafka based kafka bindings, but do not think we are using any of the fancy features.

Currently can only copy with one bluesky profile, we probably want a couple of profiles to enable different ioc setups.

The Spawner will use the podman API to launch user containers (and pods?). Useful links:

• https://www.redhat.com/sysadmin/podman-python-bash
• https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/

In previous work I have used docker-compose with mounting of git repositories to facilitate development, and imagine a similar workflow could be achieved with podman. In the development of either the bluesky or databroker stacks I imagine the need for:

• Server running in one container, with reload enabled for development
• Client running in another container with reload enabled for development
• Server to combine the server and client behind a single port (probably NGINX)

There would be a (probably default) production version where it would not enable reload, and would simply build the web application into an optimized static bundle using everything within the container/image.

There appears to be include https://docs.docker.com/compose/multiple-compose-files/include/ andextends https://docs.docker.com/compose/multiple-compose-files/extends/ directives in the vocabulary and a way to merge https://docs.docker.com/compose/multiple-compose-files/merge/ compose files when invoking up (or automagically by putting semantics in filenames 🤯 ).

This needs a bit of ivestigation, but my current thinking is we want to use include so that we can have shared compose file fore:

• core data services (mongo, postgres, kafka, tiled, jlab)
• epics serivces (archiver, saverestore, ...)
• set(s) of IOCs
• bluesky / qs / CSS configuration

So that we can have a couple of variations {just ophyd.sim, a bunch of mock caproto IOCs, ADsim + motorsim, beamline-analog, blackhole-IOC, ...} without having to copy-paste a lot of yaml.

There are a couple of custom images built, these should be published so they can be pulled.