Git Product home page Git Product logo

orleanscontrib / orleans.activities Goto Github PK

View Code? Open in Web Editor NEW
82.0 15.0 16.0 1.97 MB

Workflow Foundation (.Net 4.x System.Activities workflows) over Microsoft Orleans framework, providing stable, long-running, extremely scalable processes with XAML designer support.

License: Apache License 2.0

C# 99.95% Batchfile 0.05%
orleans orleans-framework workflow workflow-foundation virtual-actors actor-model actor-framework workflow-activities microsoft-orleans distributed-actors

orleans.activities's Introduction

Orleans logo Orleans.Activities

Workflow Foundation (.Net 4.x System.Activities workflows) over Microsoft Orleans framework to provide stable, long-running, extremely scalable processes with XAML designer support.

Gitter Waffle

Stable: GitHub version NuGet version

Master: Build status AppVeyor project feed (NuGet source)

Guidelines

This project is licensed under the Apache License.

Important

Only in case of projects with designed XAML workflows!!!

  • Don’t use Microsoft.NET.Sdk, use old project format (Sdk has no XamlAppDef BuildAction)

  • Use at least VS 15.6 (older VS 15.x Activity designer crashes when Microsoft.Orleans.Core.Abstractions is installed via NuGet 4.x PackageReference tag)

Documentation

  • see the Samples below, they come with tutorial level, detailed descriptions
  • or see my Presentations repo

Concept

Why?

The key concepts behind Workflow Foundation and Microsoft Orleans are very similar, but Microsoft Orleans solves the pain points of Workflow Foundation (WCF and SQL oriented, non-scalable).

Workflow Foundation Microsoft Orleans
Single threaded
Persistent reminders
Stateful
😐 Communication
😐 Storage
Scalability

How?

We kept only the the "good parts" of Workflow Foundation, the WorkflowInstance (a mini Virtual Machine executing the Activities) and replaced everything else with Microsoft Orleans, ie. we replaced WorkflowServiceHost with Microsoft Orleans Grains (stateful actors).

Concept-How

Integrated:

  • Persistence (compatible with legacy workflow extensions)
  • Reminders (compatible with legacy Delay activities)
  • Tracking
  • Designer & Debugger support
  • Nearly all legacy activities are supported (except TransactionScope and WCF messaging activities)

Result?

A typical workflow grain ("domain service" grain) manages operations in other normal grains ("aggregate root" grains) and handles only the process specific data in it's own state.

  • Normal grains typically have long life, they can have event sourced states or can use short-lived Orleans transactions on their interfaces.
  • Workflow grains have a one-shot lifetime, the long-lived transaction they implement.

Concept-Result

Implementation

Overview

This is a very high level view:

  • Each WorkflowGrain is indistinguishable from a normal grain and backed by a WorkflowHost.
  • The WorkflowHost is responsible to handle the lifecycle of the WorkflowInstance, mainly recreate it from a previous persisted state when it aborts.
  • The communication between the WorkflowGrain and the WorkflowHost is based on 2 developer defined interfaces for the incoming and outgoing requests (TWorkflowInterface and TWorkflowCallbackInterface).
    • These interfaces provide the type safe communication with the workflow.
    • Their methods can be referenced from the workflow activities to accept incoming or to initiate outgoing requests.
  • The methods of the TWorkflowInterface and TWorkflowCallbackInterface are independent from the grain's external public interface, you can merge different public requests into one method or vice versa. Or a reentrant grain even can execute (read-only) public interface methods independently from the current running workflow operations.
  • The method's signatures are restricted, their parameters and return values are lazy, async delegates with 1 optional parameter/return value. The delegates executed by the workflow activities if/when they accept them (command pattern).
  • There are design-, build- and static-run-time checks to keep the interfaces and the workflows in sync.
  • Though you can execute complete workflows as methods also.

The goal, is to keep the C# code in the grain, and use the workflow only to decide what to do next. This way we can avoid a steep learning curve to use workflows: the developer doesn't need to write or to understand anything about activities, he/she can build workflows with the provided activities in a designer.

Functionality

Integrated:

  • Persistence (compatible with legacy workflow extensions)
  • Reminders (compatible with legacy Delay activities)
  • Tracking
  • Designer & Debugger support
  • Nearly all legacy activities are supported (except TransactionScope and WCF messaging activities)

Extra implemented features:

  • TAP async API
  • Optionally idempotent request processing for forward recovery
  • Automatic reactivation after failure
  • Workflow can be persisted during processing an incoming request (ReceiveRequestSendResponseScope is not an implicit NoPersistScope)
  • Executing code "in the background" on the tail of the request after the request returns it's response
  • Workflow is informed whether it is running in a reloaded state after failure (to determine necessary recovery)
  • Notification participant extensions (to get notified when the workflow is idle)

Under construction:

  • Tests (currently semi manual, semi automatic MSTest, don't even look at them)
  • More elaborate sample with
    • DI/Autofac
    • Strategy and Humble Object patterns, to show an architecture, where the application logic can be tested independently from Orleans and from Orleans.Activities workflows

Not implemented, help wanted (for design and for implementation):

  • DynamicUpdateMap support (updating loaded workflows to a newer definition version), though the separation of the application logic (the plain C# delegates) and the process (the diagram) results in a very simple workflow diagram, that has a big chance you won't need to update when it runs
  • See all Help Wanted issues

Samples

With tutorial level, detailed descriptions!

HelloWorld - How to communicate with the workflow through custom interfaces.

Arithmetical - How to execute the complete workflow like a method.

Details

You don't need to understand this to use the project! This is for those who want to dig in to the source!

This is still an overview, all the details of the classes are hidden. The goal is to give a map to understand the relations between the classes.

The 2 generated proxies translate the calls between the TWorkflowInterface and TWorkflowCallbackInterface interfaces and the workflow's API, where the methods are identified by their names.

For more details see the detailed comments in the source!

Overview

orleans.activities's People

Contributors

lmagyar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

orleans.activities's Issues

Semi-optimistic concurrency to handle multiple grain activations

Currently in case of double activation, with optimistic concurrency, the first writer wins.
The original workflow SQL persistence provider used a "last reader" strategy: it "locked" the persisted row in database (instances in memory should refresh the lock in each 2 minutes to prove they are live), and the "etag" was incremented on each read. In case of double activation, only the last reader with the latest "etag" were able to write back the state.
This can be implemented with the blob provider, with a lease/break pair.

DynamicUpdateMap support

Updating loaded workflows to a newer definition version. Though the separation of the application logic (the plain C# delegates) and the process (the diagram) results in a very simple workflow diagram, that has a big chance you won't need to update when it runs.

Better workflow state serialization

Currently the whole Dictionary<XName, InstanceValue> in IWorkflowState is serialized with NetDataContractSerializer and zipped with GZipStream. NetDataContractSerializer uses XML format (XmlDictionaryWriter). We should try to use Orleans or any other serializers.
NOTE: One of the special values in InstanceValue is the ActivityExecutor itself, and it's allegedly works only with NetDataContractSerializer.

Always add reactivation reminder

Even when in runnable state and there is already a reminder (based on a delay). That reminder can be far in the future. This way in case of failure the workflow will be reactivated sooner.

Test scheduling - sync context

When activities execute grain delegates, sync context should be restored to original grain. Should? Isn't it restored yet?

Be able to load Completed workflows

The original WF persistence doesn't allow to load completed workflows, if we try to rehydrate an executor, the WorkflowInstance.InitializeCore will throw a NRE... But this is essential, if the client repeats a request due to a failure and we have idempotent responses, we should use them.

Workaround:

  • we shouldn't store the executor in completed state
  • we should always store the completion state, output arguments and termination exception
  • let save and load the extensions
  • add workarounds for every incoming method (operation and reminder resumption, Run and RunToCompletion, Deactivate, etc.)

Even if we can answer idempotent requests in completed state, we can't re-fire a normal outgoing WorkflowCallbackInterface operation or we shouldn't re-fire the OnCompleted event, because if those hadn't been successful the WF wouldn't have completed at all. So it's OK, if RunToCompletion's workaround can read the output arguments without the OnCompleted event, it's like an idempotent operation.

Test scheduling - stack dive

We should check to not make stack dives!!!
TaskCompletionSource-s for operations are created without TaskCreationOptions.RunContinuationsAsynchronously, and nearly all ContinueWith uses TaskContinuationOptions.ExecuteSynchronously.
When we go from sync OnNotifyUnhandledException and OnNotifyPaused to TAP async equivalents, there are two await Task.Yield(), these should break most of the loops.
NOTE: SendResponse's TaskCompletionSource.Task should continue ASAP, and not wait for the other activities in the WF, this way WF can run on the tail of the operations!
See eg.: http://stackoverflow.com/questions/28321457/taskcontinuationoptions-runcontinuationsasynchronously-and-stack-dives

Remove any logic from OnActivateAsync

Currently the workflow is "preparated" ie. loaded (and started if it was saved in Runnable state) during OnActivateAsync(). It can cause exceptions that are propagated back to OnActivateAsync(), causing activation to fail. Will this cause reactivation in an infinite loop? Or waste silo resources?
The alternative is to delay the workflow preparation for the real incoming request or Reminder after activation, if it fails, the grain remains active and will reload and rerun the workflow on the next incoming request or Reminder, ie. will retry later without deactivation.

Refactor Tests

Currently semi manual, semi automatic MSTest.
Refactor it and use xUnit and Orleans scheduler.
Create/use something like MS.Activities.Test? (Ron Jacobs)
+Somewhere there is a Timeout in a parallel activity, it should be in a pick.
+Measure code coverage.

More elaborate sample

With:

  • DI/Autofac
  • Strategy and Humble Object patterns, to show an architecture, where the application logic can be tested independently from Orleans and from Orleans.Activities workflows

Constraint on Operation names

Add constraint to check whether the current operation name is in the list. Currently if the interface changes, the combobox is empty meantime the property contains the value, without any error message.

Add activity support for DeactivateOnIdle, DelayDeactivation

Currently it can be implemented through any TWorkflowCallbackInterface method, but a built in support would be easier, and it is better to see this kind of operations in the workflow designer and not hide inside operation delegates.
Or IdleDeactivationMode parameter like the IdlePersistenceMode parameter???

Test parallel incoming request processing

Reentrant grain, 2 ReceiveRequest activity in parallel, both parallel branch contains some Delay activity before SendResponse to emulate internal work.
Both request has to accepted and the workflow must idle in both Delay activity and send the responses in parallel.

Cancellable operations

To wait for the incoming request is cancellable, but the execution of the parameter delegate is not.
Also the outgoing requests return delegate is not cancellable.
The outgoing request is awaited by the scope without cancellation, this is a third place for an optional cancellable Task.

WorkflowHost should handle multiple active requests

Currently only one taskCompletionSource can be protected with redirecting unhandled exceptions to it. WorkflowHost is reentrant, multiple parallel requests can be active.
This is a problem when eg. the workflow aborts, and we have to complete all taskCompletionSources.

Rename Affector and Effector

The difference is only 1 letter, it's confusing. "that's a semantics deathtrap" @Aaronontheweb :) I suggest:

  • IWorkflowInstance - OK
  • IWorkflowHost -> IWorkflowInstanceCallback because de facto this is a callback interface
  • IAffector - IWorkflowHost
  • IEffector - IWorkflowHostCallback
  • TAffector - TWorkflowInterface
  • TEffector - TWorkflowCallbackInterface

Test scheduling - being active

Test that when workflow runs on the tail of the already replied incoming request, Orleans scheduler really treat these delegates/tasks as belonging to the message.
Eg. non-reentrant grain will not accept another incoming request until the workflow goes idle.
Be aware, a delay in workflow makes it idle, it's tricky to test.

CancelWorkflow should complete WF in Canceled and not in Closed state

When the workflow cancels itself due to an unhandled exception in a ReceiveRequestSendResponseScope, the completion state of the workflow is Closed nod Canceled.

If we let the exception to buble up, and a real OnUnhandledException event occurs, that schedules workflow cancellation after the Idle event, the completion state will be correct.

I suppose, there is missing a context.MarkCanceled(); statement somewhere in this activity that prevents the default cancellation of the root activity to finally make Canceled it's state instead of Closed. The TaskAsyncNativeActivity turns off default Cancellation, if cancellation finally happens, MarkCanceled should be called somewhere.

See: http://blogs.msdn.com/b/carlos/archive/2013/02/06/wf4-workflow-foundation-4-understanding-cancellation.aspx

ClearStateAsync support on Completed state

To directly clear state when completed is dangerous, it removes idempotent informations also.
It would be better to register for a later (days, months) state clear, or enable some external archivation of the database with storing these info in eg. separate column.

NuGet package

And also separate samples into their own solutions based on NuGet.

Base template activity

Currently workflows are transaction scripts. Make them DSL.
In a template activity you can specify one or more activities as arguments (like the try/catch/finally arguments in TryCatchFinally activity).
And a body, where you can use these arguments.
This way you can build template activities based on this base like:

  • "execute this activity in a loop in case of exceptions"
  • "wait for this incoming request with a timeout and throw exception when it expires"

Make factories dependent on grain state

Because IMHO this is the simplest way to make them optionally dependent on runtime (practically incoming request's) values.
Eg. a public grain method can set an "immutable" property in grain state and factories executed later can use it.
The immutability can be ensured by code contracts.

State consistency after workflow abort

Currently the workflow is not reloaded until the next incoming workflow related request or reactivation reminder (2 minutes), until this the non workflow related requests will read an obsolete state, that will be reloaded later.
Should we reload State after abort immediately (but not run, that can cause abort again), or let non workflow related requests to prepare the workflow state and implicitly the grain state?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.