kestra-io / kestra Goto Github PK

Default value are not exposed for now on documentation.
This will keep user blind on this part

Changing the namespace or the id will led to a 404 on the update from the webui.
It's a normal behaviour on the FlowControler we ask and update on a flow that don't exists.

This led to 2 issues :

• 404 are not catch and loading bar keep growing, this must be capture and displayed as error.
• How can we rename a flow ? is this a possible action ?

Since there is a lot of "foreign keys" (see #60), we need to disabled the edition of namespace and flow id for now (waiting #60)

If an invalid flow, all server failed to start.
Bean validation exception is throw and will be catch by picocli that will only display invalid flow-path
Need to have a proper error

When a taskRun has no output nor is restartable, the "hamburger menu" is still displayed and contains nothing.

depends on: #54

Since there is a lot of "foreign keys" on flow namespace + flow id, we need to implement a special method to move a flow and change the name.

The ui must be like on github with its "danger zone" (it will break all app using the flow).
The move method must changed the flow namespace & id, since the operation can be very long, it must be async.

Also, when a user want to change the namespace it encounters multiple “Page not found” before understanding that it wasn’t possible. We should at least improve the error message.

Seems that with Kafka runner, some very long line will kill a consuming worker silently.
No error will be in logs and the consumer is remove for Kafka silently.

Really need to investigate

On the main page, add a inputs table below the actual with key / value of inputs.
If possible, add a download link on any inputs of type file

running a simple flow ./kestra test flow.yaml that open a connection to a server for example (keep resource open) will hang the command test (reproduced with SlackExecution command)

We need to find a way to close resource :

• Keep resource open for a frequent tasks can lead to some performance improvements.
• But that can lead to overflow the destination server with many connection open and never used, also that can lead to some exception when the server close the connection.

I think the proper way will to close all connection for now.

For now, some exception can be throw during evaluation of any method of FlowableTask interface.
If this exception occured, it will break the flow and maybe break the application.

We need to properly capture these exception to be resilient in case of an invalid flow

Task create temporary file like that :

        // temp file
        File tempFile = File.createTempFile(this.getClass().getSimpleName().toLowerCase() + "_", ".csv");
        ObjectOutputStream output = new ObjectOutputStream(new FileOutputStream(tempFile));

It's mostly done for now to handle in order to forward to StorageInterface with a : runContext.putFile(tempFile).getUri()

We need to find a better way to have StorageObject directly without intermediate temporary files or at least delete the temporary files.

on 49f829e, we introduced a new feature that return last execution if terminated on executions/{executionId}/follow, we don't need anymore to call executions/{executionId}

For now only the message exception is saved on execution.
Capture recursively cause message and add the full stacktrace is TRACE log

Seems to be have an context scope that don't work on routeInfo(), since on the FlowRoot.Vue we include the Executions.Vue, the execution replace the breadcrumb.

On file upload in ui , display a progress bar real time that displays progression %

id: errors
namespace: org.floworc.tests

tasks:
- id: failed
  type: org.floworc.core.tasks.scripts.Bash
  commands:
  - 'exit 1'
  errors:
  - id: 2nd
    type: org.floworc.core.tasks.debugs.Echo
    format: second {{task.id}}

The errors will not be triggers, need to be global to work or on FlowableTask

Flow like sequential-with-local-errors on reporting for now status SUCCESS on parent-seq tasks.

Hello~
Is there a guide to start working via source code? I couldn't find in document...

Thanks.

Task fields are not indicated as mandatory or not so user cannot know what he must/should/may provide ...

For now, log are emitted only at the end of the current task.
They must be real time and emitted as soon as possible to be displayed on the web ui

Currently the uri generated by StorageInterface have to many slash, example :
floworc:////org/floworc/tests/inputs/executions/test/inputs/file/application.yml.

We need to have only : floworc:///org/floworc/tests/inputs/executions/test/inputs/file/application.yml. like that : scheme://{nohost}/org/floworc

On EachSequential, the output is unreachable with handlebars :
https://github.com/tchiotludo/floworc/blob/cb93c3d93622193835a466fa0e0b63d7286f10a6/core/src/main/java/org/floworc/core/models/executions/Execution.java#L200
since we add a uid() with TaskRunId.
Find a proper way to add a the value of all childs

allow submit file with DnD

Allow to configure :

• errors
• timeout
• retry

for :

• global configuration
• per tasks type configuration
• per namespace
• per flow

When using dynamic tasks, we don't see the value, so we are enabled to locate a specific tasks.
The link from topology also must need to link to actual tasks with value

If I click on the side menu bottom arrow, it must not overflow the content.
Also it must be keep on local storage preference.

In some metrics, namespace is in a label "namespace" (eg: kestra_org_kestra_core_tasks_debugs_return_duration_seconds_count) and in other, it is in the label "namespace_id" (eg: kestra_executor_execution_duration_seconds_count)

for every command on cli, this code from TestCommand will be run :

    @SuppressWarnings("unused")
    public static Map<String, Object> propertiesOverrides() {
        try {
            Path tempDirectory = Files.createTempDirectory(TestCommand.class.getSimpleName());

            return ImmutableMap.of(
                "kestra.repository.type", "memory",
                "kestra.queue.type", "memory",
                "kestra.storage.type", "local",
                "kestra.storage.local.base-path", tempDirectory.toAbsolutePath().toString()
            );
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

and will create a temp folders, must be created only on test command

When we create the execution from the webui, the execution is created and return directly.
But the indexer could not have the time to index the execution on the backend.
This lead by a 404 on /api/v1/executions/{{executionId}} and the ui freeze here.

Difficult use case here, I think the better will be to use only the execution return during the creation and reach the updated information from the listen url (without trying to reach the url below).
This will lead to no dependency on indexer on this part of the webui.

I need to reboot my server if I want to add a plugin, it is not very friendly

For now, we don't see attempts detail on the webui.
We need to add this informations

Refactor the RunContext, the object is mutable right now and will not be usable in whole case.
For example : ApplicationContext is need for some method and can be null.

Maybe implement a WorkerRunContext that extends RunContext

Add a progress bar for input uploads to enhance UX. Users large files upload will come soon.

After triggering a flow, the Ui calls 2 http routes, first executions/{executionId}/trigger then executions/{executionId}/follow

This problem occurs sometimes when an execution is so fast that the second call (follow) is done when the execution is already terminated.

Currently the follow route, does not handle this case and always wait for the passed execution to terminate ...

What must be done :

• Update the follow route to handle this case by checking the execution state and returning without waiting ...

Hi,

I wanted to reach out over email, but your site is under development. I'm curious about one thing -

Why are most of the orchestration engines I see on GitHub built using Java? This seems to be a common trait among them.

Could you please detail some of the reasons why you chose Java over, say, go or python? Is it just that you are more familiar with the language?

FlowRepository implements a method :

    List<Flow> findRevisions(String namespace, String id);

It would be great to be able to use it 😄 (I need to find a culprit for wasting 3 hours of my life - might be me, probably me, but it least I'll be able to sleep on both ears)

Prerequisite for #62

Add a "restartable" property to TaskRun.

Currently the fact that a taskRun can be restarted for an execution is computed on Ui side ...

Ui Side : See 'Restart' component

If there isn't any possibility in the Switch FlowableTask, the task throw an NPE, and the task remain "running"

2019-12-26 15:12:01,892 INFO  mory-queue-4 l.parent-seq [execution: 3Pf6uUWsqTo2jjynz1T9As] [taskrun: 2M4fE75in0WRObmtLX8lTJ] Task parent-seq (type: Switch) started
Exception in thread "memory-queue-7" java.lang.NullPointerException
	at org.kestra.core.runners.FlowableUtils.resolveState(FlowableUtils.java:76)
	at org.kestra.core.tasks.flows.Switch.resolveState(Switch.java:58)
	at org.kestra.core.runners.AbstractExecutor.childWorkerTaskResult(AbstractExecutor.java:60)
	at org.kestra.runner.memory.MemoryExecutor.lambda$handlChild$5(MemoryExecutor.java:133)
	at java.base/java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:441)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.kestra.runner.memory.MemoryExecutor.handlChild(MemoryExecutor.java:136)
	at org.kestra.runner.memory.MemoryExecutor.lambda$run$1(MemoryExecutor.java:59)
	at org.kestra.runner.memory.MemoryQueue.lambda$emit$0(MemoryQueue.java:45)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

This must failed the task properly

Maybe use a structured text language like Markdown

Actually, some files have version hardcoded :

• org/kestra/webserver/Application.java
• org/kestra/cli/App.java
• package.json

We need to find a way to only rely on gradle.properties

Every package provided by Kestra use Lombok and cause this :

Solution seems to be delombok during build :
mplushnikov/lombok-intellij-plugin#438 (comment)

plugins {
    id "io.freefair.lombok" version "3.2.1"
}

task sourcesJar(type: Jar) {
    from 'build/delombok/main'
    archiveClassifier = 'sources'
}

tasks.getByName("publish").dependsOn(delombok)

Display a new tab on flow with a list of revision.
On click one revision, display a popup with comparison between previous & selected version.
Add a select on top in order to allow user to change revision for comparison.

Switching between tabs on flow page lead to an empty page :

We need to press F5 to have the current page displayed well

Each flowable tasks at the parentId on the output making it unreachable with handlebar vars

Doing these implementation that supports lucene queries , pagination and sorting fields will make it possible to provide unit tests for find routes that are actually untested.

for now, generic collection are shown as List<T>, array as T[].
Use a consistent syntax for both of them.
Also Expose property for T, for now, no way to know what is T

For now, the filter act as an LogLevel == value, must be the min loglevel wanted.
We can setup by default at INFO