Git Product home page Git Product logo

6.824's Introduction

MIT-6.824

Labs of MIT 6.824: Distributed Systems.

Lab 1 MapReduce

  • MapReduce input and output
  • Single-worker word count
  • Distributing MapReduce tasks
  • Handling worker failures
  • Inverted index generation

Lab 2 Raft

  • 2A
  • 2B
  • 2C

Lab 3 Fault-tolerant Key/Value Service

  • Key/value service without log compaction
  • Key/value service with log compaction

Lab 4 Sharded Key/Value Service

  • The Shard Master
  • Sharded Key/Value Server
    • Garbage collection of state
    • Client requests during configuration changes

6.824's People

Contributors

kophy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

6.824's Issues

About Lab4 ShardKV Challenge 2

Challenge 2 descirbe a situation

Say that some replica group G3, when transitioning to C, needs shard S1 from G1, and shard S2 from G2. We really want G3 > to immediately start serving a shard once it has received the necessary state, even if it is still waiting for some other shards. > For example, if G1 is down, G3 should still start serving requests for S2 once it receives the appropriate data from G2,
despite the transition to C not yet having completed.

In your code, the reonfiguration code is an "atomic" operation. you should get all fetched shard data, and then write into raft and apply. And then it can serve for client.

Personally, it cannot do the above situation.

By the way, thanks for your sharing code which helped me a lot.

func (kv *ShardKV) getReconfigureEntry(nextConfig shardmaster.Config) (Op, bool) {
	entry := Op{}
	entry.Command = "reconfigure"
	entry.Config = nextConfig
	for i := 0; i < shardmaster.NShards; i++ {
		entry.Data[i] = make(map[string]string)
	}
	entry.Ack = make(map[int64]int64)
	ok := true

	var ackMu sync.Mutex
	var wg sync.WaitGroup

	transferShards := kv.getShardsToTransfer(nextConfig)
	for gid, shardIds := range transferShards {
		wg.Add(1)

		go func(gid int, args TransferShardArgs, reply TransferShardReply) {
			defer wg.Done()

			if kv.sendTransferShard(gid, &args, &reply) {
				ackMu.Lock()
				// copy only shards requested from that replica group to reconfigure args
				for _, shardId := range args.ShardIds {
					shardData := reply.Data[shardId]
					for k, v := range shardData {
						entry.Data[shardId][k] = v
					}
				}
				// merge ack map from that replica group to reconfigure args
				for clientId := range reply.Ack {
					if _, ok := entry.Ack[clientId]; !ok || entry.Ack[clientId] < reply.Ack[clientId] {
						entry.Ack[clientId] = reply.Ack[clientId]
					}
				}
				ackMu.Unlock()
			} else {
				ok = false
			}
		}(gid, TransferShardArgs{Num: nextConfig.Num, ShardIds: shardIds}, TransferShardReply{})
	}
	wg.Wait()
	return entry, ok
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.