Comments (14)
sakno
commented on August 16, 2024
1
I found an issue. CommitAsync method doesn't check range of the end index in the transaction log. As a result, the commit index can be larger than the index of the last added log record. This fact turns the log into inconsistent state. When node started, it trying to replay all committed log entries. Due to this bug, the condition commitIndex > lastIndex leads to reading of log record which is out of range.
from dotnext.
sakno
commented on August 16, 2024
1
@potrusil-osi , could you please check again?
from dotnext.
sakno
commented on August 16, 2024
1
It was just for tests. No specific requirements for the partition size now. I'll publish a new version tonight.
from dotnext.
sakno
commented on August 16, 2024
@potrusil-osi , ensure that the following conditions are met for docker/k8s:
- Container is stateless so if the storage for persistent state is not mounted from the host you will get an issue because fresh node need to replicate the state from scratch
- Every node should have exact collection of members in the configuration. As a result, it's not enough to change the membership only for one node. The change must be applied to all nodes, otherwise the existing nodes will not be able to recognize a new member. Therefore, underlying configuration provider for ASP.NET Core must support change tracking and be connected to some distributed configuration system such as Consul/Etcd.
from dotnext.
sakno
commented on August 16, 2024
Could you please provide full stack trace for exception caused by Debug.Fail?
from dotnext.
potrusil-osi
commented on August 16, 2024
I didn't perform the tests in containers yet. I just executed RaftNode.exe from my Windows command line like RaftNode.exe http 3262 node1. That means all three instances read their configuration from the same file which I was modifying.
from dotnext.
potrusil-osi
commented on August 16, 2024
Process terminated. Assertion failed.
Log entry with index 2250 doesn't have partition
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ApplyAsync(Int64 startIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 752
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ApplyAsync(Int64 startIndex, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ReplayAsync(CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 791
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ReplayAsync(CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.InitializeAsync(CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 810
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.StartAsync(CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.cs:line 326
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.StartAsync(CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.StartAsync(CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.cs:line 144
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at RaftNode.Program.UseAspNetCoreHost(Int32 port, String persistentStorage) in C:\Users\Tomas\source\repos\dotNext\src\examples\RaftNode\Program.cs:line 40
at RaftNode.Program.StartNode(String protocol, Int32 port, String persistentStorage) in C:\Users\Tomas\source\repos\dotNext\src\examples\RaftNode\Program.cs:line 128
at RaftNode.Program.Main(String[] args) in C:\Users\Tomas\source\repos\dotNext\src\examples\RaftNode\Program.cs:line 154
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at RaftNode.Program.Main(String[] args)
at RaftNode.Program.<Main>(String[] args)
from dotnext.
potrusil-osi
commented on August 16, 2024
When I was going through the command line history I saw one more failure. I originally had not paid any attention to it because the node was able to recover from relatively quickly, but I'm pasting it here for completeness:
System.InvalidOperationException: Nullable object must have a value.
at System.Nullable`1.get_Value()
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ApplyAsync(Int64 startIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 745
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.CommitAsync(Nullable`1 endIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 685
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReceiveEntriesAsync[TEntry](TMember sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.cs:line 515
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.ReceiveEntries(HttpRequest request, HttpResponse response, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.Messaging.cs:line 250
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.ReceiveEntries(HttpRequest request, HttpResponse response, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.Messaging.cs:line 251
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
from dotnext.
sakno
commented on August 16, 2024
@potrusil-osi , these exceptions indicate that transaction log is broken. Persistent state records all log entries to the disk and trying to replay all saved records at the time of node start. In your case, all files stored withing node folder. Could you please share the contents of this folder?
And one more thing. When you changing the address of the node you must clean the directory with transaction log for this node.
from dotnext.
sakno
commented on August 16, 2024
The issue was in reconf logic. The commit above contains the fix. Here is my test environment:
- Each node shares the same
appsettings.jsonfile. The file contains member configuration except the path to the persistent storage. This path must be unique for each node. - Decrease partition size from 50 to 10 specified by
SimplePersistentStateconstructor inside ofRaftNodeexample. - Start three nodes
- Stop the random node
- Change the port number of the stopped node inside of
appsettings.json - Erasing the folder with persistent state is up to you.
- Launch the stopped node again with the new port
from dotnext.
potrusil-osi
commented on August 16, 2024
My testing environment has been exactly the same except the partition size (more about it later).
So I got the develop branch and just ran the nodes (I didn't change the partition size to 10 and kept the old folders with persistent state) and the behavior after reconfiguration appeared to be very similar as before. This is the output of the node that was reconfigured (the // comments were added):
> RaftNode.exe http 3264 node3 // previously the port was 3265
Accepting value 4181500
... // replaying the log entries
Accepting value 4192000
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 2673. Election timeout 00:00:00.2430000
Consensus cannot be reached
Term of local cluster member is 2673. Election timeout 00:00:00.2430000
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
System.InvalidOperationException: Nullable object must have a value.
at System.Nullable`1.get_Value()
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ApplyAsync(Int64 startIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 745
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.CommitAsync(Nullable`1 endIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 685
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReceiveEntriesAsync[TEntry](TMember sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.cs:line 514
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.ReceiveEntries(HttpRequest request, HttpResponse response, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.Messaging.cs:line 250
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.ReceiveEntries(HttpRequest request, HttpResponse response, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.Messaging.cs:line 251
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 2675. Election timeout 00:00:00.2960000
Receiving a snapshot. // I added this message into the code myself when a snapshot is received from the leader
Receiving a snapshot.
Receiving a snapshot.
Consensus cannot be reached
Term of local cluster member is 2675. Election timeout 00:00:00.2960000
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 2677. Election timeout 00:00:00.2150000
Receiving a snapshot.
Receiving a snapshot.
Consensus cannot be reached
Term of local cluster member is 2677. Election timeout 00:00:00.2150000
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 2679. Election timeout 00:00:00.2090000
Receiving a snapshot.
Consensus cannot be reached
Term of local cluster member is 2679. Election timeout 00:00:00.2090000
Receiving a snapshot.
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 2681. Election timeout 00:00:00.1990000
Receiving a snapshot.
Receiving a snapshot.
Consensus cannot be reached
Term of local cluster member is 2681. Election timeout 00:00:00.1990000
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 2683. Election timeout 00:00:00.2910000
Receiving a snapshot.
Receiving a snapshot.
Consensus cannot be reached
Term of local cluster member is 2683. Election timeout 00:00:00.2910000
Receiving a snapshot.
New cluster leader is elected. Leader address is 127.0.0.1:3263
Term of local cluster member is 2685. Election timeout 00:00:00.2400000
Receiving a snapshot.
Accepting value 4204000
Snapshot received and appended.
Accepting value 4204500
Accepting value 4205000
... // everything seems to work fine now
After this exercise I changed the partition size to 10 and deleted the folders with persistent state. And almost immediately I ran into the Debug.Fail issue. Here is the call stack:
Process terminated. Assertion failed.
Log entry with index 40 doesn't have partition
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ApplyAsync(Int64 startIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 752
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ApplyAsync(Int64 startIndex, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.ApplyAsync(CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 761
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.CommitAsync(Nullable`1 endIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 685
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.CommitAsync(Nullable`1 endIndex, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.CommitAsync(Int64 endIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\PersistentState.cs:line 709
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReceiveEntriesAsync[TEntry](TMember sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\RaftCluster.cs:line 514
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReceiveEntriesAsync[TEntry](TMember sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.ReceiveEntries(HttpRequest request, HttpResponse response, CancellationToken token) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.Messaging.cs:line 250
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.ReceiveEntries(HttpRequest request, HttpResponse response, CancellationToken token)
at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.ProcessRequest(HttpContext context) in C:\Users\Tomas\source\repos\dotNext\src\cluster\DotNext.AspNetCore.Cluster\Net\Cluster\Consensus\Raft\Http\RaftHttpCluster.Messaging.cs:line 293
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Hosting.HostingApplication.ProcessRequestAsync(Context context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnectionMiddleware`1.OnConnectionAsync(ConnectionContext connectionContext)
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.InnerOnConnectionAsync(ConnectionContext context)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
at System.Threading.Tasks.TaskFactory`1.<>c__DisplayClass41_0`2.<FromAsyncImpl>b__0(IAsyncResult iar)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Security.SslStream.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
at System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
I also have the content of the folder if you are interested.
from dotnext.
potrusil-osi
commented on August 16, 2024
Everything looks great after your latest change! The reconfigured node joins the cluster almost instantly now and I don't observe any failures at all.
from dotnext.
potrusil-osi
commented on August 16, 2024
Btw. I even changed the log partition size back to 50 and everything is still stable. Was there any specific reason why you suggested to lower it to 10?
from dotnext.
sakno
commented on August 16, 2024
Closing issue. Release is published on NuGet: https://www.nuget.org/packages/DotNext.AspNetCore.Cluster/2.12.1
from dotnext.
Related Issues (20)
- Cluster node status HOT 1
- DotNext vs Community toolkit
- How to use Optional<string> as query parameter? HOT 5
- Support for non-LTS .NET releases HOT 1
- Various trimming warnings in DotNext.Metaprogramming HOT 15
- Potential addition of an `OrderedDictionary<TKey, TValue>` type HOT 7
- raft leader loses leadership and the node gets stuck HOT 19
- Metaprogramming: Try-catch not catching exception HOT 2
- ask for set up workaround to make AOT works HOT 4
- Directly reference algorithm used in int sqrt HOT 3
- Not seeing expected improvement in throughput of RaftCluster.ReplicateAsync method when cluster minority is inaccessible HOT 20
- Seemingly random NullReferenceException in async state machine HOT 13
- Cache bound by weighted count HOT 1
- AsyncReaderWriterLock overload parity between Enter and Acquire HOT 1
- Add `IDisposable`-returning extension method for upgrading read lock to write lock HOT 7
- DotNext.Net.Cluster crash in production since I think version 5.4.0 HOT 24
- DotNext.Net.Cluster: System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'length') HOT 44
- If ColdStarted node is down, new leader is not elected HOT 1
- Did you remove "IEnumerable<T>.FirstOrNone"? HOT 11
- Suggestions about UserDataStorage.CopyTo() and UserDataStorage.Clear() HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dotnext.