tamulib / mod-camunda Goto Github PK

This project forked from folio-org/mod-camunda

License: Apache License 2.0

Shell 3.46% Java 87.91% JavaScript 0.67% Groovy 0.10% Python 0.11% Ruby 0.10% Dockerfile 0.47% Raku 0.11% mIRC Script 7.07%

mod-camunda's Introduction

mod-camunda

This software is distributed under the terms of the Apache License, Version 2.0. See the file "LICENSE" for more information.

Docker deployment
1. Publish docker image
Camunda Module Dependencies
Workflow Project Structure
App Deployment
Camunda APIs
ActiveMQ Message Broker
FOLIO Integration
Additional Information
Issue Tracker

Docker deployment

cd ..
git clone https://github.com/TAMULib/mod-workflow.git
cd mod-workflow
mvn clean install

cd mod-camunda
docker build -t folio/mod-camunda .
docker run -d -p 9000:9000 folio/mod-camunda

Publish docker image

docker login [docker repo]
docker build -t [docker repo]/folio/mod-camunda:[version] .
docker push [docker repo]/folio/mod-camunda:[version]

Camunda Module Dependencies

This module extends spring-module-core and brings in Camunda BPM to enable workflow capabilities. Camunda is an open-source BPM platform that is embedded in this module via the following dependencies.

# --- VERSIONS ---
<camunda.version>7.9.0</camunda.version>
<camunda.spring.boot.version>3.0.0</camunda.spring.boot.version>

# --- DEPENDENCY MANAGEMENT ---
<dependency>
  <!-- Import dependency management from Camunda -->
  <groupId>org.camunda.bpm</groupId>
  <artifactId>camunda-bom</artifactId>
  <version>${camunda.version}</version>
  <scope>import</scope>
  <type>pom</type>
</dependency>

# --- DEPENDENCIES ---
<dependency>
  <groupId>org.camunda.bpm.springboot</groupId>
  <artifactId>camunda-bpm-spring-boot-starter</artifactId>
  <version>${camunda.spring.boot.version}</version>
</dependency>

<dependency>
  <groupId>org.camunda.bpm.springboot</groupId>
  <artifactId>camunda-bpm-spring-boot-starter-webapp</artifactId>
  <version>${camunda.spring.boot.version}</version>
</dependency>

<dependency>
  <groupId>org.camunda.bpm.springboot</groupId>
  <artifactId>camunda-bpm-spring-boot-starter-rest</artifactId>
  <version>${camunda.spring.boot.version}</version>
</dependency>

camunda-bpm-spring-boot-starter
- Adds the Camunda engine (v7.9)
- https://docs.camunda.org/manual/develop/user-guide/spring-boot-integration/
- https://github.com/camunda/camunda-bpm-spring-boot-starter
- The Camunda engine requires a database schema to be configured on startup
  - Work is in progress to allow the module to start without any database creation and have the tenant creation perform the necessary table creation and initial data import
  - Details on the process engine database schema configuration can be foune here
camunda-bpm-spring-boot-starter-webapp
- Enables Web Applications such as Camunda Cockpit and Tasklist
- https://docs.camunda.org/manual/develop/user-guide/spring-boot-integration/webapps/
camunda-bpm-spring-boot-starter-rest
- Enables the Camunda REST API
- https://docs.camunda.org/manual/develop/user-guide/spring-boot-integration/rest-api/
- https://docs.camunda.org/manual/7.9/reference/rest/
- The Camunda REST API uses Jersey so we use spring boot's common application properties to configure the path to be /camunda in the application.yml file
  - spring.jersey.application-path=camunda

Workflow Project Structure

Business Process Models and Decision Models are built using the Camunda Modeler which impelements BPMN 2.0 and DMN 1.1 specifications.

.bpmn files are stored in /src/main/java/resources/workflows
.dmn files are stored in /src/main/java/resources/decisions

Any Java code that is executed in the context of a process is usually written in a Java Delegate. These classes are stored in /src/main/java/org/folio/rest/delegate/

Deploy and run the application

Run the application mvn clean spring-boot:run
1. Note there is a hard dependency on ActiveMQ. If running without ActiveMQ, be sure to comment out activemq.broker-url: tcp://localhost:61616 in the application.yml
Deploy all the processes by running scripts/deploy.sh file
Navigate to Camunda Portal localhost:9000/app/welcome/default/#/welcome
Log in as admin username: admin, password: admin

Camunda APIs

Process/Decision Deployment
- https://docs.camunda.org/manual/7.9/reference/rest/deployment/
- GET
  - /camunda/deployment
  - /camunda/deployment/{id}
- POST
  - /camunda/deployment/create
- DELETE
  - /camunda/deployment/{id}
Process Definition
- https://docs.camunda.org/manual/7.9/reference/rest/process-definition/
- GET
  - /camunda/process-definition
  - /camunda/process-definition/{id}
- POST
  - /camunda/process-definition/{id}/start
  - /camunda/process-definition/key/{key}/tenant-id/{tenant-id}/start
Decision Definition
- https://docs.camunda.org/manual/7.9/reference/rest/decision-definition/
- GET
  - /camunda/decision-definition
  - /camunda/decision-definition/{id}
Tasks
- https://docs.camunda.org/manual/7.9/reference/rest/task/
- GET
  - /camunda/task
  - /camunda/task/{id}
- POST
  - /camunda/task/{id}/claim
  - /camunda/task/{id}/unclaim
  - /camunda/task/{id}/complete
Message Events
- https://docs.camunda.org/manual/7.9/reference/rest/message/
- POST
  - /camunda/message

ActiveMQ Message Broker

We are using ActiveMQ to consume messages. Currently we are only consuming, not producing messages. This is a hard dependency when running the application, so if you want to run the application without a message broker, comment out activemq.broker-url: tcp://localhost:61616 in the application.yml

FOLIO Integration

For detailed information to bring up a FOLIO instance refer to https://github.com/folio-org/folio-ansible.

The following requires Vagrant 1.9.6 or above.

vagrant up
# wait

When finished Okapi will be running with mod-workflow and mod-camunda deployed under the diku tenant. mod-camunda will have its port forwarded for access to the Camunda webapps. FOLIO UI will be accessible at http://localhost:3000; username: diku_admin, password: admin.

Okapi is being built and redeployed from within this vagrant. Eventually this will not need to happen. If a specific branch of either mod-camunda or mod-workflow is desired to be deployed, modify the Vagrantfile git checkout main to the desired branch and restart vagrant. vagrant destroy, vagrant up

Development

In order to facilitate development on mod-camunda in the context of Okapi, there is a sync directory from the host machine to the guest machine. The host directory is at .vagrant/sync and it will contain okapi, mod-camunda, and mod-workflow. The development and git branch management can be done on the host machine. The guest directory is at /sync. The redeployment of a module must be done from the guest machine.

vagrant ssh
sudo su
cd /sync
# kill mod-camunda running on port 9000
kill $(lsof -t -i :9000)
cd mod-camunda
mvn clean install
nohup java -jar target/mod-camunda-1.1.0.jar &

Login

curl -v -H "Content-Type: application/json" -H "X-Okapi-Tenant: diku" http://localhost:9130/authn/login -d '{"username": "diku_admin", "password": "admin"}'

The response headers of interest are X-Okapi-Token and refreshToken.

Refresh Token

curl -v -H "X-Okapi-Tenant: diku" -H "Content-Type: application/json" http://localhost:9130/refresh -d '{"refreshToken": "`[refresh token goes here]`"}'

The body of this response will contain the new X-Okapi-Token.

Workflow Module Triggers

The Trigger entity from mod-workflow is used to select which request-response events from Okapi are to be published to the event.queue that mod-camunda can subscribe to. In order to create the Triggers we have to provide the correct permissions to the diku_admin. The vagrant will create an example trigger for when a user is created.

Permissions

In order to call mod-camunda and mod-workflow through the Okapi gateway a user will need the appropriate permissions. In order to accommodate this the Vagrantfile runs a shell script in which updates permissions for diku_admin. Providing him with all permissions to all interfaces of mod-camunda and mod-workflow.

Cleanup

When finished it will be desired to cleanup as the FOLIO vagrant uses a lot of resources.

vagrant destroy
rm -rf .vagrant/sync

Removing the synced directory, .vagrant/sync, on the host will remove any changes during development that have not been pushed.

Additional information

Issue tracker

See project FOLIO at the FOLIO issue tracker.

mod-camunda's People

Contributors

Watchers

mod-camunda's Issues

The StreamingResponseDelegate (formerly the TestStreamDelegate) should support multiple streams.

In its current implementation the StreamingResponseDelegate (refactored from TestStreamDelegate) utilizes the StreamService to register its stream.

The StreamService can only store and provide access to one Flux stream at a time. The StreamService should change to support a map of Flux streams. The StreamingResponseDelegate should supply a unique key (or one could be generated by the service itself) and that key should be placed on the execution context to be used in retrieving that flux stream in subsequent delegates.

The map solution used should support concurrent access.

We will need a strategy in place for cleaning a flux from the map when its stream is complete.

Mod Camunda's EnhancementExtractor should do comparisons based on multiple comparators pairs

Email Java Delegate

Spend a few minutes researching existing functionality in Camunda.

Otherwise, create Java delegate that is provided email configuration, to, cc, bcc, from, subject, body template.

The Enhancement delegate should support the new `enhancementComparisons` and `enhancementMappings` properties of an EnhancementTask

All streaming delegates should be shown to be running until their stream is complete.

Currently each of our streaming delegates manages a stream and then finalizes. The streams process lives on past the delegate. It would be ideal if the delegate did not complete until the stream it is managing is finalized.

The StreamService should support a map of Fluxes

The StreamService should support a Map<UUID, Flux> to keep track of multiple streams.

Mod Camunda should provide multiple extractor delegates

Given the completion of this issue in Mod Workflow and this local issue:

Mod Camunda should provide an abstract extractor delegate.

Mod Camunda should provide specific extensions of the abstract extractor delegate for extraction strategies corresponding to the values of the MergeStrategy enum in Mod Workflow.

The enumeration of MergeStrategy mergeStrategy found on any incoming Workflow Task of type ExtractorTask should be used to implicate a specific extensions of the abstract extractor delegate when creating the BPMN representation of that workflow.

Use milliseconds instead of seconds for flux element delay

It would be better appreciated to have finer-grained control over how long the delay is.

For modules, like external reference resolver, higher speeds may be potentially supported and a smaller delay would be preferred, especially for large data sets.
(example: 1 million records at 1 second delay vs 1 million records at 400ms delay.)

This would require either changing duration to milliseconds OR allow custom parsing such that "ms" (or something similar) is added to inform the parser to use milliseconds.

The former would require updating all appropriate tasks in fm-workflow.
The latter would require updating only what needs to use milliseconds, but at the cost of adding more processing logic to determine when to use seconds or milliseconds.

fm-workflows will need to be updated based on the decision made here.

Mod-workflow may also need to be changed, and also ensure that the data type is consistent between camunda and workflow (e.g. long vs int).

Mod Camunda should provide a start workflow interface

Mod Camunda's module descriptor should be modified to include an endpoint for the manual start of a workflow, designated by the workflows process identifier.

It should also provide a controller implementation of for this endpoint which will start the designated workflow.

** This may already exist **

Mod Camunda should provide an activate/deactivate interface

Mod Camunda should expose an API for activating workflows.

Mod Camunda should have its module descriptor modified to provide an interface for workflow-engine/workflow/activate and workflow-engine/workflow/deactivate, with all relevant permissions.

Both endpoints should accept a JSON representation of a workflow as defined by Mod Workflow. We should explore making mod-workflow a dependency of mod camunda, so that the workflow model defined in mod workflow can be available in mod camunda for deserialization.

The implementation of the activation and deactivation should be a recapitulation of the logic found in ModCamundaService and BpmnModelFactory of Mod Workflow.

Analogous service ought to be created in Mod Camunda to handle the activation.

Mod camunda should provide a reporting delegate

Mod camunda should provide a delegate for compiling a report from a provided template.

The EnhancementDelegate should support lexicographical comparisons

Destroy Database Connection Java Delegate

Singleton service for connection registry. Will connect and continue workflow passing connection key.

StreamCreateForEach (formerly TestCreateForEachDelegate) should allow restriction of creation by multiple properties.

Currently the StreamCreateForEach delegate (refactored from TestCreateForEachDelegate) allows for the optional property String uniqueBy. When not null this property is used to query the target location to see if any entities are already with the same key value pair as the current potential entity being created.

It might be the case that the potential entity being created is needed to be unique by multiple keys. To support this potential use case the property uniqueBy ought to be changed to be a String[] and the query generated from it should check for an existing entity which matches on each key value pair.

The delegates package should be decluttered

Currently there are quite a few delegates in place from previous proof of concept work on workflows. These delegates should be deprecated and refactored into a separate package, such as delegate.examples or delegates.poc.

Only those delegates needed to perform the migration workflows should reside at the top level of the delegate package.

Any documentation references to these classes will need to reflect this refactor.

Mod-Camunda should have a StreamConsumer Delegate

This delegate should utilize the stream service to obtain the primary stream and subscribe to. Th BPMN model factory should ensure that this delegate is always called from a service task at the end of the workflow, if any task on that workflow return true from isStreaming

Mod Camunda should generate BPMN for the StreamingRequestTask

File I/O Read Delegate

We need a delegate to read from a path and read JSON files into a workflow. Should have a few properties; path to request JSON files and delay to provide ability to throttle.

Mod Camunda should provide a delegate for originating a flux stream

This delegate would utilize the StreamingService to originate a stream. This delegate should automatically be the initial delegate for any workflow where isStreaming = true.

Determining this will depend on the completion of
Mod Workflow issue 13 and Mod Camunda issue 1

Script utilities should include techniques for getting marc fields

The StreamAccumulationDelegate (formerly TestAccumulationDelegate) should have its request functionality moved into a new PoolRequestDelegate

Currently the StreamAccumulationDelegate (refactored from the TestAccumulationDelegate) both accumulates a stream into a list, and then makes a creation request for each object within the list.

This is poor separation of concerns. Instead the StreamAccumulationDelegate should only collect a stream into a list of size n, and then pass that list forward in the workflow. The request functionality that currently exists should be implemented in a PoolRequestDelegate.

Logging should be handled by a logger and not system out. #2

Currently much of the logging has been handle using System Out. This should be changed to use a logger, and Log Level should be added to the docker file as a configurable environment variable.

Dockerfile should be tracked in this repo

Enhancing Extractor Delegate Should Enhance All Matches

Enhancing Extractor Delegate should continue to enhance all matches until no match is found. This should include respecting a boolean for isMultiple, placing enhanced properties within an object and placing that object in an array. The enhancement delegate will need to name the key of the destination array based on the mapAs value.

The BPMN model factory will need to be updated to respect the isMultiple and mapAs properties on the enhancement task.

Migration workflows should use a login delegate

Currently login is occurring within the TestStreamDelegate. This should not happen in the context of any of the workflow delegates, but should instead be handled by a login delegate.

The FolioLoginDelegate and the ResetLoginTokenDelegate are examples of how the can be handled.

These two classes may be usable with slight modifications.

TestCreateForEachDelegate should be refactored to StreamCreateForEach

TestCreateForEachDelegate should be refactored to be StreamCreateForEach and should be at the top level of the delegate package.

The BPMN model factory should support the sort mode property of the EnhancementTask

TAMULib/mod-workflow#45

Mod Camunda script utilities should provide for UUID creation.

Investigate potential blocking changes to the Enhancing Delegate

The Streaming Process Delegate should support multiple scripting languages

Currently full support of multiple scripting languages has not been full implemented.

The ScriptEngineService utilizes a script template written in javascript. This should have a map of templates for all supported languages and should select the correct template based on the scriptType value.

The script utils have only been expressed in javascript. This file should be refactored to be src/main/resources/scripts/utils.js. Then analogous implementations should be created in each of the supported scripting languages. The script utility resources should be loaded from these files using the scriptType as an extension.

finally, the implementation of each utility function should be in Java, and each script utility should expose a call to the Java method as a native function call. Here is an example of calling java from javascript in the javax scripting engine.

TestAccumulatorDelegate should be refactored to StreamAccumulationDelegate

TestAccumulatorDelegate should be refactored to be StreamAccumulationDelegate and exist at the top level of the delegate package.

Mod Camunda's delegates should support a migration of Voyager users into FOLIO organizations

Mod Camunda should provide a EnhancingExtractorDelegate

This delegate will create a new stream from a specified source and merge the tuples of the new stream onto those of the primary stream, adding specified data points from the new tuples to the primary stream's tuples.

Database Query Java Delegate

Requires database key from registry, query, and result mapping.

The StreamingRequestDelegate should map the incoming stream

Currently the streaming request delegate subscribes to the primary stream. This should be changed to a map of the stream, returning the incoming string changed. (This can possibly be done with a for each or some-other method.)

Mod Camunda should provide a PoolRequestDelegate

A new PoolRequestDelegate should be created and placed at the top level of the delegate package. It should receive a list of objects and properties Expression concurentRequest
and Expression requestEndpoint, where concurentRequest should be parsable into an integer representing the desired number of concurrent request and requestEndpoint is a string representing the url to which the request should be made.

The delegate should pop n number of objects from the incoming array of objects, and for each make a request to the endpoint designated by requestEndpoint. wThe body of each request is a JSON representation of the object.

This delegate should continue to pop object from the incoming array and make requests from those object, while ensuring that the total number of pending requests never accedes concurentRequest. As each pending request resolves, a new request should be made until all objects in the incoming array have been exhausted.