dotnet / dotnext Goto Github PK
View Code? Open in Web Editor NEWNext generation API for .NET
Home Page: https://dotnet.github.io/dotNext/
License: MIT License
Next generation API for .NET
Home Page: https://dotnet.github.io/dotNext/
License: MIT License
Is the DotNext.Net.Cluster.Consensus.Raft cluster capable of allowing non-specified members to join the cluster?
e.g. cluster boots with 3 seed nodes, then additional nodes join in against one of those, and the cluster become aware of the newly joined nodes?
Or is it purely capable of consensus between the pre-specified members?
In https://github.com/AsynkronIT/protoactor-dotnet , we have externalized all cluster management.
e.g. we use Consul or Kubernetes API to get the cluster topology.
I would very much like to give it a try to use dotNext Raft as an alternative cluster provider, as it wouldn't require users to have some specific infrastructure in place.
public class Test
{
[Fact]
public async Task AsyncLambdaDoesntHangInTryBlock()
{
Expression<Func<Task<string>>> exprThrowException = () => ThrowException();
Expression<Func<Task<string>>> exprReprocess = () => Reprocess();
var asyncTryCatchExpression = AsyncLambda<Func<Task<string>>>(
(lambdaContext, result) =>
{
Try(() =>
{
Assign(result, Expression.Invoke(exprThrowException).Await());
Return(result);
})
.Catch<Exception>(() =>
{
Assign(result, Expression.Invoke(exprReprocess).Await());
Return(result);
})
.End();
});
var asyncTryCatchFunc = asyncTryCatchExpression.Compile();
var asyncTryCatchTask = asyncTryCatchFunc();
var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await await Task.WhenAny(asyncTryCatchTask, Task.Delay(-1, cancellationTokenSource.Token));
}
private static async Task<string> ThrowException()
{
await Task.Delay(TimeSpan.FromSeconds(1));
throw new Exception();
}
private static Task<string> Reprocess()
{
return Task.FromResult(string.Empty);
}
}
Thank you very much for providing a raft C # implementation. I now want to build a distributed kv system. Can I use your framework to achieve this?
Hello,
I am trying to change my cluster to run in containers and am having a problem with configuring the members. Since docker assigns them their own IPs and makes the nodes available through a dns system (eg https://mydockerraftnodeservice), I have configured the members collection to use the service name. However, when I attempt to bring up the cluster, the constructor of the RaftClusterMember class tries to convert the member to an endpoint and fails since it is not a loopback endpoint:
internal static IPEndPoint? ToEndPoint(this Uri memberUri)
{
switch (memberUri.HostNameType)
{
case UriHostNameType.IPv4:
case UriHostNameType.IPv6:
return new IPEndPoint(IPAddress.Parse(memberUri.Host), memberUri.Port);
case UriHostNameType.Dns:
return memberUri.IsLoopback ? new IPEndPoint(IPAddress.Loopback, memberUri.Port) : null;
default:
return null;
}
}
This is a problem since I do not know the IPs of the containers ahead of time and since the standard way to refer to the containers is through dns.
Do you have any ideas?
I understand wanting to keep LexicalScope an internal implementation detail.
But allowing adding statements manually instead of using CodeGenerator would be really helpful when there is a lot of existing code using standard System.Linq.Expressions
patterns.
If I could manually call LexicalScope.Current.AddStatement
, it would save me a lot of code rewriting. And make gradual migrations to the dotNext LinqEx handling.
Preferably I would like to be able to do the following:
LexicalScope
/AsyncLambdaExpression
using manually specified ParameterExpression[]
, instead of just System.Type[]
LexicalScope
, specifically using ILexicalScope.AddStatement
Background:
I'm trying to use your Raft implementation to implement a distributed lock service in an Asp.Net Core application. I'm using the IMessageBus.LeaderRouter to send messages to the leader where the lock is entered into the consensus store and, if needed, it waits for the lock to become available (up to a supplied timeout value).
Seems to work reasonably well as long as there is no contention... but if there were no contention I wouldn't need a lock!
I think the issue I'm running into is that the timeout is specified for all requests at the HttpClient level, using a timeout value that is only appropriate for the built-in election messages.
Reading the documentation led me to this blurb in the Microsoft docs for HttpClient:
The same timeout will apply for all requests using this HttpClient instance. You may also set different timeouts for individual requests using a CancellationTokenSource on a task. Note that only the shorter of the two timeouts will apply.
Before I go in and hack up the code to try to make this work for me, am I missing something?
See suggestions in #56 .
Tasks:
SkipAsync
to IAsyncBinaryReader
PersistentState
in a way that allows to do the compaction in parallel with writes or readsfsync
for every write must be disabled by default (FileOptions.WriteThrough
flag)The documentation guide is missing at this link: https://sakno.github.io/dotNext/migration/2.html
Hi,
I found that if I want to persist the config on file should use the class PersistentClusterConfigurationStorage. I don't know is correct or not but from tracing the source I create the class
`
internal class PersistentClusterConfigurationStorage : PersistentClusterConfigurationStorage
{
public PersistentClusterConfigurationStorage(string path, int fileBufferSize = 512, MemoryAllocator allocator = null) : base(path, fileBufferSize, allocator)
{
}
protected override void Encode(IPEndPoint address, ref BufferWriterSlim output)
{
output.WriteEndPoint(address);
}
protected override IPEndPoint Decode(ref SequenceReader reader)
{
return (IPEndPoint)reader.ReadEndPoint();
}
}
and using for persisting data
static async ValueTask AddMembersToCluster(AppSettingConfig serverConfig, PersistentClusterConfigurationStorage storage)
{
if (serverConfig.Members == null) return;
for (int i = 0; i < serverConfig.Members.Length; i++)
{
var t = Dns.GetHostAddresses(serverConfig.Members[i].EndPoint);
var ipEndPint = new IPEndPoint(t[0], serverConfig.Members[i]?.Port ?? serverConfig.PeerCommunicationPort);
await storage.AddMemberAsync(ClusterMemberId.FromEndPoint(ipEndPint), ipEndPint);
}
}
`
but when the running application LeaderChanged is not called and can't understand what is false.
Hi,
I try to start example/RaftNode and saw what some nodes can stop participating in leader elections.
What can lead to this behavior?
.NET Core SDK (reflecting any global.json):
Version: 3.1.401
Commit: 5b6f5e5005
Runtime Environment:
OS Name: Windows
OS Version: 10.0.17763
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.401\
Host (useful for support):
Version: 3.1.7
Commit: fcfdef8d6b
The host also has the Hyper-V role installed, if relevant.
node1:
cmd: RaftNode.exe tcp 3262
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 911. Election timeout 00:00:00.2750000
Consensus cannot be reached
Term of local cluster member is 911. Election timeout 00:00:00.2750000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 913. Election timeout 00:00:00.2330000
Consensus cannot be reached
Term of local cluster member is 913. Election timeout 00:00:00.2330000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 919. Election timeout 00:00:00.2880000
node2:
cmd: RaftNode.exe tcp 3263
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 911. Election timeout 00:00:00.2720000
Consensus cannot be reached
Term of local cluster member is 911. Election timeout 00:00:00.2720000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 913. Election timeout 00:00:00.2720000
Consensus cannot be reached
Term of local cluster member is 913. Election timeout 00:00:00.2720000
New cluster leader is elected. Leader address is 127.0.0.1:3264
Term of local cluster member is 919. Election timeout 00:00:00.1820000
node3: Problem node
cmd: RaftNode.exe tcp 3264
Consensus cannot be reached
Term of local cluster member is 465. Election timeout 00:00:00.1930000
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 469. Election timeout 00:00:00.2030000
Consensus cannot be reached
Term of local cluster member is 469. Election timeout 00:00:00.2030000
In fact, I have seen the same behavior when trying to use RaftCluster with TCP transport in our project and cannot find the problem. So I tried to reproduce the problem with the example project.
Another problem is that after a while, the worker node stops with an assertion error:
Process terminated. Assertion failed.
at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ClientExchange.ProcessInboundMessageAsync(PacketHeaders headers, ReadOnlyMemory`1 payload, EndPoint sender, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\TransportServices\ClientExchange.cs:line 70
at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.ClientNetworkStream.Exchange(IExchange exchange, Memory`1 buffer, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\Tcp\TcpClient.cs:line 59
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.ClientNetworkStream.Exchange(IExchange exchange, Memory`1 buffer, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.Enqueue(IExchange exchange, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\Tcp\TcpClient.cs:line 139
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpClient.Enqueue(IExchange exchange, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ExchangePeer.SendAsync[TResult,TExchange](TExchange exchange, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\TransportServices\ExchangePeer.cs:line 48
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ExchangePeer.SendAsync[TResult,TExchange](TExchange exchange, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.TransportServices.ExchangePeer.VoteAsync(Int64 term, Int64 lastLogIndex, Int64 lastLogTerm, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\TransportServices\ExchangePeer.cs:line 68
at DotNext.Net.Cluster.Consensus.Raft.RaftClusterMember.DotNext.Net.Cluster.Consensus.Raft.IRaftClusterMember.VoteAsync(Int64 term, Int64 lastLogIndex, Int64 lastLogTerm, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftClusterMember.cs:line 78
at DotNext.Net.Cluster.Consensus.Raft.CandidateState.VotingState.VoteAsync(IRaftClusterMember voter, Int64 term, IAuditTrail`1 auditTrail, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\CandidateState.cs:line 35
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.CandidateState.VotingState.VoteAsync(IRaftClusterMember voter, Int64 term, IAuditTrail`1 auditTrail, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.CandidateState.VotingState..ctor(IRaftClusterMember voter, Int64 term, IAuditTrail`1 auditTrail, CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\CandidateState.cs:line 55
at DotNext.Net.Cluster.Consensus.Raft.CandidateState.StartVoting(Int32 timeout, IAuditTrail`1 auditTrail) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\CandidateState.cs:line 131
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState() in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.cs:line 661
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState()
at DotNext.Net.Cluster.Consensus.Raft.FollowerState.Track(TimeSpan timeout, IAsyncEvent refreshEvent, Action candidateState, CancellationToken[] tokens) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\FollowerState.cs:line 34
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at DotNext.Threading.QueuedSynchronizer.WaitAsync(WaitNode node, TimeSpan timeout, CancellationToken token) in g:\work\VSNET\github\dotNext\src\DotNext.Threading\Threading\QueuedSynchronizer.cs:line 121
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskFactory.CompleteOnInvokePromise.Invoke(Task completingTask)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetResult()
at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut()
at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
I noticed that Synchronizer uses internal state quite liberally and would fail in a multithreaded environment. Code like this
return node is null ? CompletedTask<bool, BooleanConst.True>.Task : node.Task.WaitAsync(timeout, token);
will fail with NullReferenceException if other thread calls AsyncManualResetEvent.Set(false)
. With a small modification example code here would trigger this error.
Hello, I am seeing a strange behavior. My state (and therefore my snapshot) is a dictionary object. The last logentry before a snapshot builder is requested enters a particular key into the dictionary. The snapshotbuilder then creates a snapshot with the entire dictionary.
Now, when I bring down the nodes and bring them up again, a replay occurs in which the snapshot is (obviously) applied first, but then the logentry which was seen previous to the building of the snapshot is applied after which causes a duplicate key exception. It seems as if this is a race condition between the last entry before the snapshot and the snapshot record itself.
Any ideas?
Hi,
I would like to implement a gRPC version of RaftCluster in the DotNext.Net.Cluster package. However, I can't find any samples or detailed information on how to go about this. I have looked at the HTTP version but it seems very different than the transport-agnostic implementation. Do you have a working sample using RaftCluster?
Thanks.
When handling vote requests in the following method, the election timeout when rejecting vote requests (when line 544 is false):
Note that this happens both explicitely in line 533, and indirectly in line 529. It seems to be problematic in both cases, although in the later when it is a leader it does need to start a new election timeout (it is supposed to change though, instead of remaining the same which seems to be the case following the implementation of StepDown).
Some of this is not explicit at all in the raft papers, not even the phd dissertation. However, one can simulate various scenarios in the raft visualization by sending requests/stopping/resuming/timing out nodes and seeing how exactly it deals with those cases https://raft.github.io/
More info on some of the scenarios related to the votes being rejected with the checks in line 529:
Consider how the later scenario can play when there are only 2 out of 3 nodes remaining. One of them has an entry the other one does not. If one is very unlucky with timings then the node without the extra entries keeps becoming a candidate first on many rounds. This alone does not match the many rounds I have seen it take, but this might combine with other implementation details in dotnext raft, as maybe an unlucky election timeout might keep getting reused.
Originally posted by AntonWerenberg October 28, 2021
When attempting to run the unmodified RaftNode example for tcp nodes I encounter problems.
all running are in the leader state and appear not to interact with each other.
Upon closer inspection, I have found that the memberlist of each node only contains itself. I suspect that the problem may be because the node is configured with coldstart = true. Could this be the case?
Upon reading the documentation I am in doubt whether only a single node should be configured with coldstart = true.
When running the example with http configuration everything runs flawlessly.
commands run in seperate teminals when starting example in tcp mode
~/dotNext/src/examples/RaftNode$ dotnet run --framework net6.0 tcp 3262 node1
~/dotNext/src/examples/RaftNode$ dotnet run --framework net6.0 tcp 3263 node2
~/dotNext/src/examples/RaftNode$ dotnet run --framework net6.0 tcp 3264 node3
Hi @sakno,
Starting with version 3, the hosted host is not being disposed. I'm including screenshot that will explain everything:
Hey! I use the PooledBufferWriter class from the DotNext v 3.2.1 package, after converting to Stream, then call the CopyTo method and get an exception NotSupportedException
I wanted to know if it was so conceived or am I doing something wrong?
using var writer = new PooledBufferWriter<byte>(ArrayPool<byte>.Shared.ToAllocator());
using Stream writeStream = StreamSource.AsStream(writer);
using MemoryStream memoryStream = new MemoryStream();
memoryStream.Write(new byte[1024]);
writeStream.CopyTo(memoryStream); // thow exception
Hello,
I am currently trying to run the 4.2.0-beta.1
version and I have come across a specific issue regarding the log entry writing with Raft and the events associated with it
After step 3 and inspecting my logs, it seems like the leader steps down to follower but does not fire the IRaftCluster.LeaderChanged
event. IRaftCluster.Members
will still report the local (Remote == false) node 1 as leader on node 1, but attempting to write to the log now will result in
System.InvalidOperationException: The local cluster member is not a leader
at DotNext.Threading.Tasks.ValueTaskCompletionSource`1.GetResult(Int16 token) in /_/src/DotNext.Threading/Threading/Tasks/ValueTaskCompletionSource.T.cs:line 272
at DotNext.Threading.Tasks.ValueTaskCompletionSource`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/DotNext.Threading/Threading/Tasks/ValueTaskCompletionSource.T.cs:line 279
at DotNext.Net.Cluster.Consensus.Raft.LeaderState.ReplicationCallback.Invoke() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/LeaderState.Replication.cs:line 177
--- End of stack trace from previous location ---
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 818
IClusterMember.MemberStatusChanged
will correctly fire and set node 2 to Unavailable during step 3.
This behavior was not present before the upgrade to 4.2.0-beta.1
.
After step 4 is done, (on node 1) IClusterMember.MemberStatusChanged
will correctly fire and mark node 2 as available again. However, node 1 is still unable to write to the log as it seemingly isn't leader anymore. Only some time after this step, node 1 will correctly fire IRaftCluster.LeaderChanged
to NO LEADER, and then fire it again but WITH A LEADER.
Hi,
I am currently trying to use the Raft feature, and got members to join a cluster. However when I try to get their metadata via GetMetadataAsync
the receiving instance (who is also the leader) just throws a middleware exception about not being able to properly handle the json data that is used internally to communicate.
I have tried setting no meta data, I have tried setting a simple "hello" -> "world" pair, nothing seems to make a difference.
Can someone please check if this function is currently working for them? I am using the latest stable release 4.1.3
Thank you!
This is all the information given to me. Visual studio is unable to catch the error, there is no more output besides this:
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
System.InvalidCastException: Unable to cast object of type 'System.Text.Json.Serialization.Converters.ObjectConverter' to type 'System.Text.Json.Serialization.JsonConverter`1[System.String]'.
at System.Text.Json.Serialization.JsonDictionaryConverter`3.GetConverter[T](JsonTypeInfo typeInfo)
at System.Text.Json.Serialization.Converters.DictionaryDefaultConverter`3.OnWriteResume(Utf8JsonWriter writer, TDictionary value, JsonSerializerOptions options, WriteStack& state)
at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryWrite(Utf8JsonWriter writer, TDictionary dictionary, JsonSerializerOptions options, WriteStack& state)
at System.Text.Json.Serialization.Converters.JsonMetadataServicesConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state)
at System.Text.Json.JsonSerializer.WriteStreamAsync[TValue](Stream utf8Json, TValue value, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
at System.Text.Json.JsonSerializer.WriteStreamAsync[TValue](Stream utf8Json, TValue value, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
at System.Text.Json.JsonSerializer.WriteStreamAsync[TValue](Stream utf8Json, TValue value, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
Hello,
I am seeing a strange behavior on some of my nodes but not on others. My first node comes up first and successfully polls for the other members. The other two members, however, come up, try to connect, get a 403 resonse from the first node and crash with the following error:
info: Microsoft.Hosting.Lifetime[0]
Now listening on: https://0.0.0.0:443
info: CRB.DistributedServices.LockServer.LockServerNodeLifeTime[0]
Cluster member initialized.
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: QA
info: Microsoft.Hosting.Lifetime[0]
Content root path: /app
dbug: DotNext.Net.Cluster.Consensus.Raft.Http.Embedding.RaftEmbeddedCluster[0]
Sending request of type PreVote to member Unspecified/lockserver01-ingress-svc.lockserver-ns.svc.cluster.local:443
dbug: DotNext.Net.Cluster.Consensus.Raft.Http.Embedding.RaftEmbeddedCluster[0]
Sending request of type PreVote to member Unspecified/lockserver03-ingress-svc.lockserver-ns.svc.cluster.local:443
warn: DotNext.Net.Cluster.Consensus.Raft.Http.Embedding.RaftEmbeddedCluster[0]
Cluster member Unspecified/lockserver03-ingress-svc.lockserver-ns.svc.cluster.local:443 is unavailable
System.Net.Http.HttpRequestException: Name or service not known
---> System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftClusterMember.SendAsync[TResult,TMessage](TMessage message, CancellationToken token) in /_/src/cluster/DotNext.AspNetCore.Cluster/Net/Cluster/Consensus/Raft/Http/RaftClusterMember.cs:line 96
dbug: CRB.DistributedServices.LockServer.LockServerMetricsCollector[0]
ReportResponseTime: 00:00:00.0433675
dbug: CRB.DistributedServices.LockServer.LockServerMetricsCollector[0]
ReportResponseTime: 00:00:00.1943564
Unhandled exception. DotNext.Net.Cluster.Consensus.Raft.Http.UnexpectedStatusCodeException: Response status code does not indicate success: 403 (Forbidden).
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftClusterMember.SendAsync[TResult,TMessage](TMessage message, CancellationToken token) in /_/src/cluster/DotNext.AspNetCore.Cluster/Net/Cluster/Consensus/Raft/Http/RaftClusterMember.cs:line 115
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState>g__PreVoteAsync|74_0(Int64 currentTerm) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 740
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.DotNext.Net.Cluster.Consensus.Raft.IRaftStateMachine.MoveToCandidateState() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 694
at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
I am not sure why the first node is returning a 403 (I never configure any authorization) or why such a response would cause a crash of the node.
I wanted to try and use the cluster library for a project, using the provided playground as an example, but I hit a roadblock.
Multiple methods in the example have type of content that can be used for consensus limited to unmanaged types. I wish to send a more complex object, or at least a string. What am I missing? And if I am not asking for too much, can a minimal example be provided?
Thank you
Partially related to my other issue (#14), when I fixed the issue in my code the state machine seems to become invalid at some point (I guess the return might not be valid). I get this exception + stack trace:
at void DotNext.Runtime.CompilerServices.AsyncStateMachine<TState, TResult>.System.Runtime.CompilerServices.IAsyncStateMachine.MoveNext() in /_/src/DotNext.Metaprogramming/Runtime/CompilerServices/AsyncStateMachine.cs:line 380
at void System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox.MoveNext(Thread threadPoolThread)
at Action System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents(Task task, Action continuation)+(Action innerContinuation, Task innerTask) => { }
at void System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, object state, ref Task currentTask)
at void System.Threading.Tasks.Task.ThrowAsync(Exception exception, SynchronizationContext targetContext)+(object state) => { }
at void System.Threading.QueueUserWorkItemCallback.s_executionContextShim(QueueUserWorkItemCallback quwi)
at void System.Threading.QueueUserWorkItemCallback.Execute()
at bool System.Threading.ThreadPoolWorkQueue.Dispatch()
This is my code for generating the lambda:
var funcSignature = typeof(Func<,,,>).MakeGenericType(writerType, typeof(EntityBase),
typeof(CancellationToken), typeof(Task<>).MakeGenericType(typeof(byte[])));
var method = typeof(CodeGenerator).GetMethod(nameof(CodeGenerator.AsyncLambda))
.MakeGenericMethod(funcSignature).Unreflect();
var compileMethod = typeof(Expression<>).MakeGenericType(funcSignature)
.GetMethod(nameof(LambdaExpression.Compile), BindingFlags.Public | BindingFlags.Instance, null,
CallingConventions.Any, Type.EmptyTypes, null).Unreflect();
finalExpression = method(null, (Action<LambdaContext>)GenerateAsyncLambda);
finalExpression = compileMethod(finalExpression);
void GenerateAsyncLambda(LambdaContext c)
{
var localWriter = DeclareVariable("writer", c[0]);
var localEntity = DeclareVariable("e", c[1]);
var localToken = DeclareVariable("token", c[2]);
var actualEntity = DeclareVariable(entityType, "actualEntity");
Assign(actualEntity, Expression.Convert(localEntity, entityType));
ParameterReplacer replacer = new ParameterReplacer(localWriter, actualEntity, localToken, entityType);
foreach (var expr in writerExpressions)
Await(replacer.Visit(expr), true);
}
EDIT: I tried adding Return()
at the end, but that's not working either.
At this moment, async lambda function compiler implemented by Metaprogramming library supports only Task
and Task<R>
as valid return types. ValueTask
is not supported. To do that, any custom async state machine builder should be recognizable by compiler with help of AsyncMethodBuilderAttribute
I can't get AsyncBarrier
to work at all, even under trivial usage.
The following code never completes - the Console.WriteLine
is never reached. Putting breakpoints on the return
statements reveals that one of them is reached, but the other never is.
[Fact]
public async Task Test1()
{
var barrier = new AsyncBarrier(2);
var results = await Task.WhenAll(
Task.Run(async () =>
{
await barrier.SignalAndWaitAsync();
return 24;
}),
Task.Run(async () =>
{
await barrier.SignalAndWaitAsync();
return 42;
})
);
Console.WriteLine(string.Join(',', results));
}
This project is transferring to .NET Foundation. Location of this repository will be changed. Due to transfer, we need to do the following tasks:
Originally posted by AntonWerenberg November 8, 2021
I've been experimenting with the raft node example code for some time, and one issue keeps being present: Instability in the cluster.
The nodes will often run fine in the beginning, but sooner or later they will start to go out of sync and fail. One node will show the tcp error warning:
"warn: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74022]
Request has timed out"
and others will start running elections, but will not reach consensus.
the behaviour can be seen in the attached image.
I'm running with default election timeout settings from the example.
I'm running with default election timeout settings from the example.
I'm not a great programmer and I'm really having trouble seeing which direction to go to get to the bottom of this?
My main concern is currently if this could be due to my own setup, something not configured correctly, or something like that.
I measured Broadcasttime using Metrics collector. It is showing broadcast times of around 3 ms.
I'm testing your Raft implementation and I observe issues when cluster configuration changes in runtime. The point is to simulate behavior in a Kubernetes cluster where containers are always recreated with new IP addresses.
I have modified the RaftNode project to be able to update the configuration. I introduced appsettings.json and moved the in-memory configuration from Program.cs into it. This allows me to update the config file at any time (the members
collection) and that change is picked up by all three running instances of RaftNode.
When there are no configuration changes, the cluster is very stable - I can restart any RaftNode instance (both the leader or any of the followers) and it immediately joins the cluster - if a new leader election is needed it usually happens within one term. However, when I stop a node and change its port number in the config file and then start it again with the new port, I sometimes observe these issues:
PersistentState.ApplyAsync(long startIndex, CancellationToken token)
on the line with Debug.Fail.I'm running this on Windows, using HTTP for communication and I use persistent storage - the node with the modified port reuses the storage.
Run this test. On my computer, it deadlocks after about 6 seconds
[TestMethod]
public async Task DotNextAsyncAutoResetEvent()
{
DotNext.Threading.AsyncAutoResetEvent autoResetEvent = new(false);
var loopCount = 0;
var setCount = 0;
var consumedCount = 0;
var timer =Stopwatch.StartNew();
var lastSecondReported = 0;
var producerTask = Task.Run(() => {
while (true)
{
loopCount++;
var didSet = autoResetEvent.Set();
if (didSet)
{
setCount++;
}
if (timer.Elapsed > TimeSpan.FromSeconds(lastSecondReported))
{
var tup1 = new { loopCount };
var tup = new { loopCount, setCount, consumedCount };
Console.WriteLine($"t={lastSecondReported}sec: {new { loopCount, setCount, consumedCount }}");
lastSecondReported++;
}
if (timer.Elapsed > TimeSpan.FromSeconds(30))
{
break;
}
}
});
var consumerTask = Task.Run(async () => {
while (true)
{
var success = await autoResetEvent.WaitAsync(TimeSpan.FromMilliseconds(1));
if (success)
{
consumedCount++;
}
if (producerTask.IsCompleted)
{
break;
}
}
});
await producerTask;
autoResetEvent.Set();
await consumerTask;
}
using DotNext 4.0.0-beta.8 from Nuget
In docs not enough information about Snapshot. How I understand snapshot is simple command (implementing serialization/desirialization in CommandFormatter etc). How concatinate WriteToAsync in inherit from SnapshotBuilder and Command?
Hi,
I read your documentation and try to enter in this web link: https://github.com/sakno/dotNext/tree/develop/src/examples/RaftNode , but is show 404 error. I would like to know if you have some simple example project to use consensus for some data.
Greetings,
Luis
For best performance, we use TcpConfiguration and create a simple k/v Store, that uses a simple dictionary. And not a configuration snapshot in PersistentState. we found that the optimistic way to create log entries is using fixed struct size like example. but I get 136ms avg latency that is very high for me.
SqlServer instance give me 40ms avg latency for the same job! and we need 1-3ms avg latency.
I wanna know we could reach 3ms with this library or should use other libraries?
Hello,
Thank you for sharing your work with the community.
The documentation of .NEXT is very detailed - going through it right now.
I'd be lying if I said that I understand all the complexities involved in what you've built.
That being said, I'm currently evaluating options to fulfill the following use case:
So I thought that some kind of leader election would get me there.
I've used distributed locks (via a DB) in the past, but I thought I could do better.
Would you say I'm on the right path with .NEXT Raft?
The other question is: are you currently using .NEXT Raft in production or know of case histories I might share with the higher-ups, so I can promote the usage of this library?
Thank you again for your contribution to the community and for your time.
Best,
Rob
If my cluster is configured to have the following members:
localhost:1000, localhost:1001, localhost:1002
The following code will not redirect to the leader properly since the port that is used for the redirect is not the leader port:
internal Task Redirect(HttpContext context)
{
if (context.Request.Path.StartsWithSegments(pathMatch, StringComparison.OrdinalIgnoreCase))
{
var leader = cluster.Leader;
if (leader is null)
{
context.Response.StatusCode = StatusCodes.Status503ServiceUnavailable;
return Task.CompletedTask;
}
else if (leader.IsRemote)
{
return redirection(context.Response, new UriBuilder(context.Request.GetEncodedUrl()) { Host = leader.Endpoint.Address.ToString(), Port = applicationPortHint ?? context.Connection.LocalPort }.Uri);
}
}
return next(context);
}
should this not use something like the following?
new UriBuilder(context.Request.GetEncodedUrl()) { Host = leader.Endpoint.Address.ToString(), Port = leader.Endpoint.Port }.Uri)
Hi, is there support here for an untyped async lambda expression (akin to LambdaExpression
in the standard library)? I see there's AsyncLambdaExpression<TDelegate>
but I don't know the input/return types at the time of constructing the expression in my use case. Thanks!
Hi,
Do you have a example to use IMemberDiscoveryService?
I would like to add members dynamically in the cluster...
Best regards,
Luis
Microsoft.CodeAnalysis.FxCopAnalyzers used throughout NEXT is 3.3.1, 3.3.2 is the latest version and it is marked as deprecated. Consider removing/replacing this library.
According to the docs on Async Lambda:
AsyncLambda
factory method is overloaded by two versions of this method:
AsyncLambda((fun, result) => { })
introduces special variableresult
that can be used to assign result of the function. This approach is similar to Result variable in Pascal programming language.
AsyncLambda(fun => { })
doesn't introduce special variable for the function result and control transfer to the caller is provided byCodeGenerator.Return
method.
However, it appears that the first overload (the one with result
) does not exist in the source code. The only overload I can find is the second one here. It appears that to return values from an AsyncLambda
, one needs to explicitly call CodeGenerator.Return
instead.
Perhaps the discrepancy is because the overload with result
is yet to be implemented?
Hello, I have setup my cluster in embedded http mode (using DotNext.Net.Cluster.Consensus.Raft.Http.Embedding;
) and am able to achieve consensus of all the nodes. However, when I try to setup my own controller to respond to requests on the cluster, All calls to it come back as "The requested URL can't be reached". There is no response code.
Here is my cluster configuration:
{
"Kestrel": {
"EndpointDefaults": {
"Protocols": "Http2"
}
},
"partitioning": false,
"lowerElectionTimeout": 500,
"upperElectionTimeout": 1000,
"members": [ "https://localhost:3262", "https://localhost:3263", "https://localhost:3264" ],
"protocolVersion": "Http2",
"metadata": {
"key": "value"
},
"logLocation": "c:\\locks",
"recordsPerPartition": 10000,
"allowedNetworks": [ "::1", "127.0.0.0", "255.255.0.0/16", "2001:0db9::1/64" ],
"hostAddressHint": "192.168.0.1",
"requestJournal": {
"memoryLimit": 5,
"expiration": "00:00:10",
"pollingInterval": "00:01:00"
},
"resourcePath": "/cluster-consensus/raft",
"port": 3262,
"heartbeatThreshold": 0.5,
"requestTimeout": "00:01:00",
"keepAliveTimeout": "00:02:00",
"requestHeadersTimeout": "00:00:30"
}
Here is my project startup:
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder => {
webBuilder.UseStartup<Startup>();
})
.JoinCluster();
In the app configuration I call app.UseConsensusProtocolHandler()
Finally, here is what my controller looks like:
using DotNext.Net.Cluster.Consensus.Raft;
using DotNext.Net.Cluster.Replication;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;
using System;
namespace CRB.DistributedServices.LockServer
{
[Route("[controller]")]
public class MyController : ControllerBase
{
private readonly IPersistentState _persistentState;
private readonly IReplicationCluster<LockMessageLogEntry> _replicationCluster;
private readonly ILogger<MyController> _logger;
public MyController(IPersistentState persistentState, IReplicationCluster<LockMessageLogEntry> replicationCluster, ILogger<MyController> logger)
{
_persistentState = persistentState ?? throw new ArgumentNullException(nameof(persistentState));
_replicationCluster = replicationCluster ?? throw new ArgumentNullException(nameof(replicationCluster));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
[HttpGet]
public ActionResult Get()
{
_logger.LogInformation("Got");
return Ok();
}
}
}
Do you have any idea if the cluster might be blocking my controller?
I'm trying to set up a system where things function more or less normally no matter how many members are online and no matter how many members are part of the cluster. Looking at the documentation I thought AllowPartitioning would provide that functionality, but it wasn't working and looking at the test cases (SingleNodeWithoutConsensus
) it appears it isn't meant to work the way I want... but I'm a little bit confused about why the current behavior is desired.
As best as I can tell, the only real reference to the option is when in the leader state, you won't bump out of leader state upon discovering that a quorum is not present. The state machine cannot make commits, and non-leaders will not become leaders.
I understand that strictly speaking Raft doesn't allow commits without consensus... so maybe what I want isn't really Raft, but it feels like what you've got is so close to being what I need. What do you think?
I had to do quite a bit of work to get my code running, but so far it does generate perfectly well working asynchronous methods for my automatic serialization of entities. But while there is no compilation error and it does execute like expected, I realized that exceptions that are thrown inside the generated delegate don't propagate. I catch an exception in the async method I call via the AsyncLambda and rethrow it so it bubbles up my pipeline. But it never reaches the code calling the "AsyncLambda". It seems like the exception just get lost and execution stops. This is a serious issue as I can't recover from this state and don't get feedback when any exception occurs.
If this is a known problem, I'd like to hear if there's any progress to fixing that. If you need an example or more thorough explanation I'd give that preferrably in private.
This task is the next step of improvements introduced in #56 :
This is a minor documentation issue.
According to the docs, CodeGenerator.ForEach
can be used as follows:
Lambda<Action<string>>(fun =>
{
ForEach(fun[0], ch =>
{
WriteLine(ch);
});
});
But this code throws System.ArgumentException
because CodeGenerator.WriteLine
does not work for System.Char
. In fact it appears it doesn't work for any primitive value type. Perhaps I can just update the docs from string
to List<string>
instead?
When an exception is thrown in CodeGenerator.AwaitForEach
which contains another CodeGenerator.AwaitForEach
(nested), you'll receive an NullReferenceException
instead of the thrown exception.
Note: with a single CodeGenerator.AwaitForEach
the exception that was being thrown in the body is being thrown, as expected.
.NET 5
DotNext.Metaprogramming 2.12.1
Nested: https://dotnetfiddle.net/yfswYy (invalid, throws NullReferenceException)
Single: https://dotnetfiddle.net/DPnQrZ (valid, throws user exception)
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using static DotNext.Metaprogramming.CodeGenerator;
public class Program
{
public static async Task Main()
{
var result = AsyncLambda<Func<IAsyncEnumerable<char>, IAsyncEnumerable<char>, Task>>(fun =>
{
AwaitForEach(fun[0], ch =>
{
WriteLine(ch);
AwaitForEach(fun[1], ch => WriteLine(ch));
});
}).Compile();
await result(GetData('1'), GetData('2'));
}
public static async IAsyncEnumerable<char> GetData(char data)
{
await Task.Yield();
yield return data;
throw new Exception("Expected exception");
}
}
1
2
Unhandled exception. System.Exception: Expected exception
1
2
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at lambda_method2(Closure , AsyncStateMachine`1& )
at DotNext.Runtime.CompilerServices.AsyncStateMachine`1.System.Runtime.CompilerServices.IAsyncStateMachine.MoveNext() in /_/src/DotNext.Metaprogramming/Runtime/CompilerServices/AsyncStateMachine.cs:line 124
--- End of stack trace from previous location ---
at Program.Main()
at Program.<Main>()
I encountered this issue when trying to build some fairly complicated async expressions. A minimal reproducible example is here:
class Program
{
private static readonly PropertyInfo _propertyInfo = typeof(TestClass).GetProperties().First();
static async Task Main(string[] args)
{
var innerExp = GetTestExpression(false);
var outerExp = CodeGenerator.AsyncLambda<Func<TestClass, Task<TestClass>>>(context =>
{
var output = innerExp.Invoke(context[0]).Await();
CodeGenerator.Assign(output, _propertyInfo, Expression.Constant("updated", typeof(string)));
CodeGenerator.Return(output);
});
var dlg = outerExp.Compile();
var result = await dlg(new TestClass("original"));
}
private static Expression<Func<TestClass, Task<TestClass>>> GetTestExpression(bool useCompilerGeneratedExpression)
{
if (useCompilerGeneratedExpression)
{
return v => Task.FromResult(v);
}
return CodeGenerator.AsyncLambda<Func<TestClass, Task<TestClass>>>(context =>
{
CodeGenerator.Return(context[0]);
});
}
public class TestClass
{
public TestClass(string testString)
{
TestString = testString;
}
public string TestString { get; set; }
}
}
This code crashes with a System.ArgumentException
on .NET Core 3.1. The exception message is as follows:
Expression of type 'System.Void' cannot be used for return type 'System.Threading.Tasks.Task`1[MetaprogrammingBug.Program+TestClass]
If I remove the CodeGenerator.Assign
call in the outer expression, then the code runs with no problem. Also, if I set useCompilerGeneratedExpression
to true
, it runs with no problem as well. I believe the compiler-generated expression should be semantically identical to the one generated with CodeGenerator.AsyncLambda
, so the latter fact most certainty indicates that there is a bug somewhere.
I have not read the source code for CodeGenerator.AsyncLambda
so I don't have the slightest idea what could have gone wrong... Would greatly appreciate if you can help with this issue!
This is more like a question.
Shouldn't the Wait method of the Synchronizer class have the Async suffix? I learnt somewhere that every async method or every method returning a task should have that suffix. Are there logical exceptions?
Are there any plans to update everything to .NET5?
Hi,
I ran into some issue while configuring Kestrel through the application config file using the server.urls
parameter.
If the StartUp class takes longer than RaftHttCluster to start, it cannot find the server address and the application stops with the message:
g:\work\VSNET\github\dotNext\src\examples\RaftNode\bin\Debug\netcoreapp3.1>RaftNode.exe http 3262
Unhandled exception. DotNext.Net.Cluster.Consensus.Raft.RaftProtocolException: Configured list of members in cluster doesn't contain address of entire application
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.StartAsync(CancellationToken token) in g:\work\VSNET\github\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.cs:line 138
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at RaftNode.Program.Main(String[] args) in g:\work\VSNET\github\dotNext\src\examples\RaftNode\Program.cs:line 143
at RaftNode.Program.<Main>(String[] args)
I think this is related to the issue aspnet/Hosting#1390 which says that server.Features.Get<IServerAddresses>()
gets populated after StartUp is done.
It's easy to reproduce:
{"server.urls", "http: // localhost: 3262"}
UseKestrel()
System.Threading.Thread.Sleep (10000)
The only workaround is setting up a Kestrel listener address and port while building the host, isn't it?
Currently it's possible to create a foreach-statement with CodeGenerator.ForEach
, but it's not possible to create an async foreach-statement.
Create a new method in CodeGenerator
named AsyncForEach
that does the same as ForEach
but for IAsyncEnumerable
.
C#
IAsyncEnumerable<string> elements;
async foreach (var element in elements)
{
// use element
}
DotNext.Linq.Expressions
Expression elements;
CodeGenerator.AsyncForEach(elements, element =>
{
// use element
});
Via ltrzesniewski/InlineIL.Fody#6 ?
Or msbuild task with roslyn code generator?
I am working on an implementation of Raft using this library. I have 2 implementations of my own that give me the exception: 'Configured list of members in cluster doesn't contain address of entire application'.
Thinking about it, I thought I probably made a mistake on my implementation and tried to run the RaftNode example that is part of the repository. However it still throws the same exception in the same place.
I hope anyone can give me a hand with this, I am stuck not knowing what I am missing and/or doing wrong.
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.