Customer reported:

I'm upgrading my team's service to use cadencefx instead of a non-standard implementation from before. I was recommended using ReplayWorkflow to ensure that nothing breaks.
I was able to get the workflow history as JSON but I get this error:
nondeterministic workflow: missing replay decision for WorkflowExecutionCompleted: (Result:[], DecisionTaskCompletedEventId:26)
These are the last two events in the history:

  {
        "eventId": 26,
        "timestamp": 1537253999278302610,
        "eventType": "DecisionTaskCompleted",
        "decisionTaskCompletedEventAttributes": {
            "scheduledEventId": 24,
            "startedEventId": 25,
            "identity": "128@compute3435-sjc1@deferred-dispatch"
        }
    },
    {
        "eventId": 27,
        "timestamp": 1537253999278309160,
        "eventType": "WorkflowExecutionCompleted",
        "workflowExecutionCompletedEventAttributes": {
            "decisionTaskCompletedEventId": 26
        }
    }

Thrift?
Compatible Json?

For high rate workflows using blocking model with a thread pool might be prohibitive
Anther option is to use Kotlin coroutines for such use cases.

Currently to pass configuration to a workflow it has to be passed as an argument, loaded in an activity or in a SideEffect (which can access only global static methods).

The proposal is to support passing a configuration object instance to a workflow instance.
The possible API:

WorkerOptions o = new WorkerOptions.Builder().setConfig(config).build();

and inside a workflow code:

ConfigClass config = Workflow.getConfig(ConfigClass.class);

or even inject it into a workflow instance field.

ConcurrentModificationException will be thrown when user tries to cancel workflow and workflow has uncatched exception. More specifically, DeterministicRunnerImpl.close() loops all failedPromises and call promise.get(). CompletablePromiseImpl get will try to call forgetFailedPromise and modify the set on failure. Hence the exception.

When this happens decision task will fail and the workflow will stuck.

We need to avoid promise de-register on DeterministicRunnerImpl.close, or use iterator to loop through the set so remove is allowed.

Call stack

ERROR [2018-08-06 18:59:01,881] com.uber.cadence.internal.worker.PollerOptions: uncaught exception
! java.util.ConcurrentModificationException: null
! at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
! at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
! at com.uber.cadence.internal.sync.DeterministicRunnerImpl.close(DeterministicRunnerImpl.java:280)
! at com.uber.cadence.internal.sync.SyncWorkflow.close(SyncWorkflow.java:111)
! at com.uber.cadence.internal.replay.ReplayDecider.decideImpl(ReplayDecider.java:391)
! at com.uber.cadence.internal.replay.ReplayDecider.decide(ReplayDecider.java:355)
! at com.uber.cadence.internal.replay.ReplayDecisionTaskHandler.handleDecisionTaskImpl(ReplayDecisionTaskHandler.java:120)
! at com.uber.cadence.internal.replay.ReplayDecisionTaskHandler.handleDecisionTask(ReplayDecisionTaskHandler.java:62)
! at com.uber.cadence.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:200)
! at com.uber.cadence.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:181)
! at com.uber.cadence.internal.worker.PollTask.lambda$run$0(PollTask.java:102)
! ... 3 common frames omitted
! Causing: java.lang.RuntimeException: Failure processing decision task. WorkflowID=ucd-demo, RunID=281a9949-7dc0-4f13-b823-03dbd4e8b389
! at com.uber.cadence.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:278)
! at com.uber.cadence.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:181)
! at com.uber.cadence.internal.worker.PollTask.lambda$run$0(PollTask.java:107)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
! at java.lang.Thread.run(Thread.java:748)

https://github.com/uber/cadence-java-client/blob/master/CHANGELOG.md

Currently specifying the wrong parameters to scheduling activity or a child workflow causes an exception that is thrown to a workflow code. While it sounds reasonable a bad code change might cause failures of workflows that might be running for a long time. The solution is to not to throw an application level exception, but an Error which leads to a decision failure. When decisions keep failing the workflows get stuck until the bug is fixed. This way a bad deployment is going to block workflow executions instead of failing them.

Cadence server includes a field 'attempt' on response to PollForDecisionTask which can be used to drop decisions and let them timeout if they are repeatedly failing.

Replaying history is currently done via the query apis. It'll fetch the full history from the server. There's no support for replaying history from a file. This is useful for repro-ing issues that happened in prod

Have a separate threads in a worker that heartbeat on behalf of activities.

Java Client Support for: uber-go/cadence-client#421

Determinister Runner waits on tasks inside locks on close. If the tasks have not yielded by the time close is a deadlock is triggered. This is more likely to happen with workflows with more concurrent activities.

When the worker shuts down and the interval passed to shutdown method is longer than the activity execution time there should not be activity timeouts.

What about from Go?

Currently when there's a cache miss (we receive partial history for a workflow that's been evicted) we fail the decision. And the server has to dispatch the decision again. This causes a significant slowdown in performance (~10x)
Proposed solution: immediately fetch the full history when a cache miss happens so that the impact would be equivalent to not using a cache.

Currently the test workflow service ignores any tokens/max page size on the request and will always return the full set of events. This is needed to unit test pagination

From Go client:

WorkerOptions struct {
                ...
                 // Optional: Sets how decision worker deals with non-deterministic history events
		// (presumably arising from non-deterministic workflow definitions or non-backward compatible workflow definition changes).
		// default: NonDeterministicWorkflowPolicyBlockWorkflow, which just logs error but reply nothing back to server
		NonDeterministicWorkflowPolicy NonDeterministicWorkflowPolicy
                 ...
}

Also consider to make it workflow type level config (may be even a property on @WorkflowMethod.)

Currently parameter of type List doesn't deserialize correctly. It happens because gson used to implement DataConverter doesn't have enough information about the type of the argument during deserialization. Current interface uses Class as a type passed to gson. The solution is to pass Type object returned from Method.getGenericParameterTypes(). The following code works:

  @Test
  public void testUUIDListGson() throws NoSuchMethodException {
    Method m = JsonDataConverterTest.class.getDeclaredMethod("foo", List.class);
    Type arg = m.getGenericParameterTypes()[0];
    GsonBuilder gsonBuilder =
        new GsonBuilder()
            .serializeNulls();
    Gson gson = gsonBuilder.create();

    List<UUID> list = new ArrayList<>();
    for (int i = 0; i < 2; i++) {
      list.add(UUID.randomUUID());
    }
    String data = gson.toJson(list);
    List<UUID> result = gson.fromJson(data, arg);
    assertEquals(result.toString(), list, result);
  }

The proposal is to change DataConverter API to either accept Method as a parameter or list of Type objects.

Currently each heartbeat call reaches service.

For relatively short running workflows keeping around multiple versions is a simpler model for developers than using Workflow#getVersion. In Java classloaders could be used to load multiple versions of the same class in the same process. This way a workflow would be packaged into an WAR like archive which would be loaded by workflow worker hosting process on demand.