pheymann / scala.bft Goto Github PK

From time to time the duration is ignored while waiting for a result in the specs. Thus leading to failing tests/builds.

What happens: The Await.result instantaneously throws a TimeoutException.

19:04:50.243 [INFO ][a.event.slf4j.Slf4jLogger] Slf4jLogger started
19:04:50.243 [INFO ][c.g.p.s.b.c.LeaderConsensus] {1,0,[0]}.aborted
19:04:50.243 [DEBUG][c.g.p.s.b.c.PrepareRound] {1,0,[0]}.prepare.consensus.messages: 1
19:04:50.246 [DEBUG][c.g.p.s.b.c.CommitRound] {1,0,[0]}.commit.consensus.messages: 1
19:04:50.246 [DEBUG][c.g.p.s.b.c.CommitRound] {1,0,[0]}.commit.consensus.messages: 2
19:04:50.246 [INFO ][c.g.p.s.b.c.CommitRound] {1,0,[0]}.commit.consensus.reached
19:04:50.246 [INFO ][c.g.p.s.b.c.PrepareRound] {1,0,[0]}.prepare.consensus.reached
19:04:50.246 [INFO ][c.g.p.s.b.c.PrepareRound] {1,0,[0]}.prepare.start
19:04:50.246 [DEBUG][c.g.p.s.b.c.PrepareRound] {1,0,[0]}.prepare.start.consensus.reached
19:04:50.246 [INFO ][c.g.p.s.b.c.CommitRound] {1,0,[0]}.commit.start
19:04:50.246 [DEBUG][c.g.p.s.b.c.CommitRound] {1,0,[0]}.commit.start.consensus.reached

As the storage system can be implemented in multiple ways (in-memory, persistent (RDB, key-value store, ...)) a flexible api is needed. This can be achieved by using akka actors. The current LogStorage extension shall not provide the storage code directly but send messages to the available implementation. Therefore, a number of messages are defined which have to be used by the storage modules:

• start consensus for request: stores request r
• add pre-prepare: stores pre-prepare message
• add prepare: stores prepare message
• add commit: stores commit message

The api has to wait until the storage process is completed for each message before continuing.

The determine the implementation during runtime the names of the actors receiving the above messages have to be set (configured) in code or config file.

See parameter description here.

As described in the overview the consensus consists of three rounds:

• pre-prepare: ((PRE_PREPARE, v, n, D(r)), r)
  
  Leader sends the client's request r to all followers. When the request is delivered to all replicas the consensus proceeds with the prepare phase.
• prepare: (PREPARE, v, n, D(r), Id(R_i))
  
  All replicas send a PREPARE message to all other replicas waiting for 2f acknowledgments. When received the messages the consensus proceeds with the commit phase.
• commit: (COMMIT, v, n, D(r), Id(R_i))
  
  All replicas send a COMMIT message to all other replicas waiting for 2f + 1 acknowledgements. When the messages are received the request operation is executed. For this issue the execution code is only available as api call.

This first implementation shall use seed replicas to integrate new instances into the running system. The protocol looks like follows:

1. there are k replicas R known (host and port defined in a config) and called seeds
2. when a new instances is added to the system it sends a join message to the seed nodes
3. the seeds send a cluster update message to all other replicas including the new instance

note: All replicas could send the cluster update message so that the probability of replicas which do not receive the message decreases (config).

The reference to the remote replicas is stored as remote actor references.

All seed replicas could be faulty and return wrong results. That doesn't work.

Instead:
Use a static solution using a predefined number of replicas with known hosts and ports.

Messages and request needs to be authenticated to hold the consistence guarantees. Thereby, authentication has to be separated into:

• lightweight MACs (Message Authentication codes) for requests and messages which aren't of the view change type
• heavyweight digital signature for view change messages

MAC Generation

prerequisite: every node (including clients) have a 16 Byte session key they share with each replica

MACs are created as follows:

1. compute a MD5 hash h_r from the client request r
2. concatenate h_r with a session key and apply MD5 again to get h_rk
3. use the 10 least significant bytes from h_rk

Scenarios:

1. messages are sent to a single client: just one MAC is needed using the clients session key
2. messages are sent to replicas: one MAC per replica/session key is generated; all are applied as vector to the message (or just add the MAC of those replica which will receive the message)

View Change Messages

They have to be signed with a strong cryptographic strategy like RSA.

Use Scala cats library to apply functional programming design and pattern on the current state (feature/authentication). This includes:

• applying interpreter pattern to encapsulate IO operations
• use Actors for IO only

Message transmission is separated into two types:

• consensus message: These messages are lightweight and can be sent by using the akka messages directly.
• client request: These kind of messages can be heavyweight and need to be chunked into smaller units (size config). Therefore the request:
  1. is represented as Byte Array and partitioned into n units regarding to the chunk size,
  2. each unit gets a order number assigned describing its position in the original array, the client id and client sequence number,
  3. the units are distributed over k sender actors and sent to the replicas. Multiple sender actors can serve one replica (config) to speed up transmission,
  4. the remote replicas collect all units and bring them in order using the order numbers -> request array reconstructed. Thereby, a unit is accepted when client id and sequence number are valid.