tonyzhangnd / goovid Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 0.0 2.45 MB

Framework for composing distributed agents

License: MIT License

Python 15.49% Shell 1.91% Go 82.60%

distributed-computing distributed-storage distributed-systems

goovid's People

Stargazers

goovid's Issues

Concurrency bugs

Go panics with

Box 127.0.0.1:8100 : Agent 100 : Client 100 ack (100, 0, client 100 request 0) committed
Box 127.0.0.1:8101 : Agent 101 : Client 101 ack (101, 0, client 101 request 0) committed
Box 127.0.0.1:8102 : Agent 102 : Client 102 ack (102, 0, client 102 request 0) committed
Box 127.0.0.1:8101 : Agent 101 : Client 101 ack (101, 1, client 101 request 1) committed
Box 127.0.0.1:8100 : Agent 100 : Client 100 ack (100, 1, client 100 request 1) committed
Box 127.0.0.1:8102 : Agent 102 : Client 102 ack (102, 1, client 102 request 1) committed
fatal error: concurrent map writes

goroutine 89 [running]:
runtime.throw(0x117c873, 0x15)
	/usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc00026baf0 sp=0xc00026bac0 pc=0x102c1b2
runtime.mapassign_fast64(0x1147f40, 0xc0000c7380, 0x2, 0x40)
	/usr/local/go/src/runtime/map_fast64.go:101 +0x35f fp=0xc00026bb30 sp=0xc00026baf0 pc=0x1011c8f
github.com/TonyZhangND/GoOvid/agents/paxos_chatroom.(*ReplicaAgent).handleDecision(0xc000138000, 0xc0006240f0, 0x25)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/agents/paxos_chatroom/replica.go:217 +0x2a6 fp=0xc00026bc68 sp=0xc00026bb30 pc=0x11194d6
github.com/TonyZhangND/GoOvid/agents/paxos_chatroom.(*ReplicaAgent).Deliver(0xc000138000, 0xc0006240f0, 0x25, 0xc000090001)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/agents/paxos_chatroom/replica.go:102 +0x40d fp=0xc00026bce0 sp=0xc00026bc68 pc=0x111803d
github.com/TonyZhangND/GoOvid/server.send(0x100020002, 0xc0006240f0, 0x25)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/server/server.go:61 +0x337 fp=0xc00026bde0 sp=0xc00026bce0 pc=0x111ff67
github.com/TonyZhangND/GoOvid/server.initAgents.func1.1(0xc000260002, 0xc0006240f0, 0x25)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/server/server.go:185 +0x136 fp=0xc00026be18 sp=0xc00026bde0 pc=0x11223e6
github.com/TonyZhangND/GoOvid/agents/paxos_chatroom.(*ReplicaAgent).spawnCommander(0xc000138000, 0xc000254220, 0xc00041e000, 0xc000256420)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/agents/paxos_chatroom/leader.go:197 +0x840 fp=0xc00026bfc0 sp=0xc00026be18 pc=0x1116200
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00026bfc8 sp=0xc00026bfc0 pc=0x1057831
created by github.com/TonyZhangND/GoOvid/agents/paxos_chatroom.(*ReplicaAgent).runLeader
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/agents/paxos_chatroom/leader.go:74 +0x96f

goroutine 1 [runnable]:
syscall.syscall(0x1090810, 0xd, 0xc0003ad8e8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/sys_darwin.go:63 +0x2e
syscall.Fstat(0xd, 0xc0003ad8e8, 0xc0003ad8f0, 0x1034cb4)
	/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:1823 +0x62
os.newFile(0xd, 0xc0000161e0, 0x1a, 0x1, 0xd)
	/usr/local/go/src/os/file_unix.go:150 +0x1f5
os.openFileNolog(0xc0000161e0, 0x1a, 0x209, 0x1a4, 0x133a108, 0x7, 0xc000016100)
	/usr/local/go/src/os/file_unix.go:227 +0x1f9
os.OpenFile(0xc0000161e0, 0x1a, 0x209, 0x1a4, 0x129b980, 0xc0000d2c80, 0xc0001d2000)
	/usr/local/go/src/os/file.go:284 +0x5f
github.com/TonyZhangND/GoOvid/server.debugPrintf(0xc000016100, 0x15, 0xc0000a01d0, 0x1, 0x1)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/server/serverCommons.go:31 +0x84
github.com/TonyZhangND/GoOvid/server.initAgents.func3.1(0x117a773, 0xb, 0xc0000a01d0, 0x1, 0x1)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/server/server.go:201 +0xfd
github.com/TonyZhangND/GoOvid/agents/paxos_chatroom.(*ReplicaAgent).handleP2b(0xc000138000, 0xc0002ee02a, 0xb)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/agents/paxos_chatroom/leader.go:222 +0x19f
github.com/TonyZhangND/GoOvid/agents/paxos_chatroom.(*ReplicaAgent).Deliver(0xc000138000, 0xc0002ee02a, 0xb, 0xc000090001)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/agents/paxos_chatroom/replica.go:110 +0x43a
github.com/TonyZhangND/GoOvid/server.handleServerMsg(0xc0002ee024, 0x11)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/server/server.go:134 +0x355
github.com/TonyZhangND/GoOvid/server.InitAndRunServer(0xc000018310, 0xe, 0xc0000c6e70, 0x0, 0x0)
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/server/server.go:279 +0x33b
main.main()
	/Users/zhang/go/src/github.com/TonyZhangND/GoOvid/ovid.go:48 +0x386

Since issue is when things are being committed, a suspect is some map that is accessed upon commit

Add message loss parameter to servers

Add a message loss parameter p to each server in [0, 1] where p is the fraction of outgoing messages a server will drop

Implement individual start mode

Implement a switch in paxos.sh that does not start each replica and client automatically.
Then have a command in controller to start those nodes

Implement all Paxos frontend agent stubs

This is to test that the configuration works

Loss rate seems to be not working

No messages between clients and replicas seem to be dropped, even when loss rate is set to 0.99

Implement controller logic

This includes

Sending commands to clients
Sending commands to replicas
And the logic in the clients and replicas for receiving those commands

Application state not updated properly

Given that all Paxos logs are the same, application states are observed to diverge.
Low priority because project requires comparing paxos log, not application state

Implement replica agent

Replica as in the "replica" in Paxos Made Moderately Simple

Debug print feature

Use the value injection technique to make debug a global variable in commons.go

Implement client logic

Implement the logic for the clients of the Paxos cluster

Process manual mode requests
Script mode autopilot operation
Replica stub to process client request

Front end needs needs to handle log mode

Right now, Paxos.sh does not care about logging, and the Paxos config generator take log mode as always true

Suspected deadlock

Some replicas appear to hang when f =3 and nClients = 9

Implement a kvs service (non-distributed)

Implements a key-value-store with a "put" and "get" API.

Three agents are needed

tty agent: parses and issues user commands, prints responses to terminal
client agent: sends the commands from tty to the kvs agent, and forward responses to tty.
kvs agent: Implements the datastore.

The kvs agent ensures durability by maintaining an append-only log, and for each put request, it only responds to the client after the value is durably committed.

A space optimization feature that saves an image of the datastore and truncate the log may be implemented in the future

Test config generator

After completing issues #9, #10, #12, run Ovid on a generated configuration to test that the Paxos config generator works as expected.

Implement some checks for rejecting bad configurations

Examples of bad configurations:

Routes that go to non-existed agents, leading to null pointer exceptions
Invalid agent types
Invalid box

Chat crashes the recipient when sending empty message

Make sure that routing table multiplexing works

SAFETY ISSUE: Inconsistency observed with multiple clients issuing concurrent requests

Handle message loss

Implement timeouts when messages are lost or when message takes a long time to deliver. The only timeout that is needed is in the client.
Each time the client times out, double the wait time

Implement starter script that starts the Paxos service

Implement a script that starts a Paxos cluster, given the input parameters

Number of Clients
Number of tolerated failures $f$
Client mode (script/manual)
Loss rate

Handle proper Multi-paxos leader election

Optimization: Newly elected leader does not need to re-propose for slots that it already know the decision

Implement all Paxos backend agent stubs

This is so that a configuration can be parsed and tested by GoOvid

Some outputs missing on `dump`

Not all output files show up upon dump command

Implement kill and skip commands

Deadlock issues

Following from #23, there are more deadlocks to be fixed.
Symptom: Scouts have issues obtaining from p1b in chan p1b messages that have been received.

In the initial implementation, clients block until a response is received for each request.
We want to remove this restriction, but ensure that each request is processed in FIFO order, and be able to match responses to requests by adding a number to requests.

tonyzhangnd / goovid Goto Github PK

goovid's People

Stargazers

goovid's Issues

Recommend Projects

Recommend Topics

Recommend Org