esteinig / brick Goto Github PK

It's a bit of a mess at the moment, not very informative for users. Particularly the form action error handling must be improved (on the return function, which currently checks for pydantic specific validation errors in a less than optimal way)

Extension of the basic BRIG - like rings and annotations to some more specific applications:

• Antimicrobial resistance predictions for automatic generation of labels with abritAMR
• Probability of viral, plasmid or chromosomal origin along segments of the reference genome with geNomad
• Add styling and provide arc line specific options, add geNomad prediction class selection

For long running tasks we want to:

• Celery: adjust timeout for task result requests to 10min - if these timeout, the data is still processed and dumped into the database model, meaning while e.g. a ring is not added, on reload the updated session model is visualised
• Celery: test different settings for file uploads and long running tasks
• App: perhaps we need a job table which contains a log of submitted and completed/failed jobs maybe at a later stage, working relatively smoothly now
• Server: keep a copy of the output tagged by reference sequence or similar to avoid redundant re-computing
• Server: limit threads in case of high load on these tasks, both on Celery workers and as setting for subprocesses
• Server: limit worker container memory for tasks, or implement proper workflow scheduler for intensive pipelines
• Server Profile geNomad memory consumption to set limits on worker container

Adding the --splits argument to genomad execution to reduce memory footprint for now. See if timeout limits have to be adjusted. These arguments can all set through the FastAPI settings and when deploying the stack through the docker/brick.env stack configuration variables.

Intensive bioinformatics tasks are not meant to run in task schedulers, especially considering resource management. Need a better solution also for other projects, any suggestions welcome!

Form action returns inside components are a bad implementation, because they prevent navigation to other sites or components and do not run when the component is not rendered on the DOM anymore, for example when inside a conditional block that switches.

Instead, we can manually dispatch the form submission data to parent components and bubble them up, and therefore run the requests from BrickInterface.svelte which is always rendered. Much better user experience and less code repetition.

Web deployment docker-compose.yml uses the default secret files (but obviously not the default secrets) in docker/mongodb - this is not ideal for continuous deployment - changing this to a root level .secrets folder which is in the .gitignore.

Local deployment (docker-compose.yml) is not sensitive around security like the web-application (unless specified by user) and it should be fine to just use the default secret files in the repository.

Local deployment via reverse-proxy to http://brick.localhost in docker-compose.localhost.yml will also change to the .secrets folder. I assume users that use this configuration are capable of changing the location of the files.

Also adding a strip line ending to the secrets validators since it has tripped me up a couple of times now.

At the moment BRICK is not optimized for mobile phones... definitely not necessary, but who knows what people are up to on their phones, maybe they do lots of genomics now?

Low priority, will likely only need some minor adjustments and testing of mobile resolution classes with Tailwind

Needs to comply with conventional commit specs for cocogitto changelog ops

This is going to be a little complicated. Labels may overlap in densely annotated regions, it would be good to:

• enable individual style editing of labels including re-positioning of line-text point to better space labels
• perhaps automatically style them to not overlap?
• add options and label database updates

Automatic overlap resolution is quite difficult, going with manual editing and styling of labels now, which was needed anyway.

Paul raised this for his metagenome assembly work 🙏

At the moment we can only use a single sequence from a reference sequence file - this was largely for convenience and getting an initial version of the application running. I will add support for combining multiple assembled contigs into a single reference.

Skeleton UI is amazing for generating page themes using tokens - but the default themes are not that pretty. Make some new ones from the palettes and include light/dark mode.

Things to do before launching the web application:

• app modifications 🥇
  • session save and store - probably better to do atomic request updates
  • zoom user config
  • layout plot settings
  • fix initial load function
  • fix rings not appearing on reload (due to getting the filtered store in the ring panel not the data panel)
  • add session re-hydration, consider what to do with files, see also: #6
  • file deletion in table
  • implement reference ring as an endpoint for consistency
  • blastn configuration options in ring
  • set timeout of celery task checks in docker compose env var
  • fix indexing of label ring and added new outer ring'
  • fix png export opacity animation
  • fix title font styling
  • fix default values on plotConfig
  • positioning of labels on line around circle
  • bug: all labels are added again (RingStore)
  • labels responsive to changes in ring settings
  • default visualisation on landing page
  • add version to session, ring and file models backwards compatibility targeted for major release
  • change all form actions to dispatchers, see also: #11
  • improve error message, see also: #9
  • save session and new session clipboards
• replace bind mount with volume as work directory
• fix package and software versions
• production-grade deployment and containers
• dev and prod docker compose profiles
• cli task and timed data deletion instead of basic script
• cocogitto bump tests and pre-bump hooks 🥉
  • package.json
  • brick.api.core.config
  • pyproject.toml
• MongoDB setup and container cfg
• CI/CD GitHub actions
  • action workflow for testing on test/** branches dev branch
  • action workflow for releases on release branch - i'm not sure if cocogitto is quite stable yet, there seems to be a bug (to be submitted after more testing) when bumping versions in a branch that is carried over into other branches including main
  • action workflow for remote update and deployment
• reverse proxy setup and configuration 🥈
• README

Drag and drop rings into new order - for next release, this is somewhat complex

George suggested the following, which is a neat idea:

As a feature, could you have a ring or maybe color the reference by the levels of sequence conservation, summaring all BLAST rings.... Say, if there were 20 samples shown, some sites in the genome may be absent/variable in only one sample, while others may often be absent/variable

Resolving a Cloudflare error

Figure out if this is possible with Uvicorn

Some error in the computation of the reference position based on cursor position. Seems more severe in the working view than the landing page visualization, where it appears only correct when cursor is on the outermost ring. Need to investigate.

Set cookie or similar to save user preference for palettes?

There seems to be a bug in the server-side database logic which introduces a second label ring when labels should be added to an existing label ring. This occurs only when adding another ring other than RingType.LABEL after creating a label ring AND adding a second label ring?

• cannot reproduce with changes to prevent ghost labels as per description below

TTL session save in MongoDB should the user request it - can then be linked into sharing. Question is whether to retain files for this time as well?

Title is changed to default from landing page visualization, because the reset is triggered on click of the Create Figure button. Needs to be done after session page load.

Think about how best to do this - either as "presentation" without ability to modify figure - this would be for sharing the interactive site for exploration with colleagues / additional comments / descriptions of the data if desired. Otherwise, the link can simply be shared as long as it has not expired, and the figure can be fully modified by colleagues

List of minor but annoying style issues I have not figured out how to modify:

• modals (e.g. save or new) focus on first focusable element with a red outline
• cursor icon change on mouseup when grabbing the svg element is not possible at the moment because the d3 zoom function suppresses the mouseup event will not fix this for now as this would take some time for very little improvement

Whole chromsome annotations can sometimes be difficult to discern without zooming in a lot. It should be fairly straight forward to subset a selected reference and all rings through the control panel to focus on specific regions e.g. 100000 - 200000 bp

• Display session limits to user on FileTable
• Delete files if they reach session limits in FileTable