Git Product home page Git Product logo

levelupdb's Introduction

CS-739-P4: LevelUpDB - A Cloud-Native version of LevelDB

Locating the code

Server

Our server code includes a key value server, communicates with the loaf balancer through heartbeats and implements Raft. Source code: src/server/*

  1. class KeyValueOpsServiceImpl: responds to client's Get(), Put() requests
  2. class LBNodeCommClient: communicates with the load balancer for hearbeats and leader asserts their leadership
  3. class RaftServer: implements RequestVote() and AppendEntries()

Client

Our client side implementation library resides here. Source code : src/client/*

  1. class KeyValueClient: has the client implementation for making Get() and Put() calls that is shared by the user and the load balancer
  2. class KeyValueService: has the implementation for the load balancer

Test Code

Automates our test cases for Get() and Put(). Source code: test/

  1. read.cc: Tests Get()
  2. write.cc: Tests Put()
  3. run_read_script.sh: Runs the tests for Get()
  4. run_write_script.sh: Runs the tests for Put()

Utilities

Source code: src/util/*

  1. LevelDB Wrapper code: levelDBWrapper.cc
  2. Locks code: locks.cc
  3. Replicated Log Helper code : replicated_log_persistent.cc and replicated_log_volatile.cc
  4. Voting state Helper code: term_vote_persistent.cc and term_vote_volatile.cc
  5. Volatile state code: state_volatile.cc
  6. State Helper code: state_helper.cc

Note: For the persistent states of Raft, we also implemented volatile versions for fast lookups. The persistent state is stored in the storage/ folder.

Setup

Folder structure

|_src
|____third_party
|_______________leveldb
|_______________snappy
|_storage

gRPC Installation

Follow these steps to install gRPC lib using cmake [here] (https://grpc.io/docs/languages/cpp/quickstart/#setup) :warning: make sure to limit the processes by passing number(e.g. 4) during make -j command.

for example, instead of make -j use make -j 4

LevelDB Installation

Clone and follow the commands mentioned here and here

Snappy Installation

Clone and follow the commands mentioned here.

Build

Building src

  1. cd src/
  2. chmod 755 build.sh
  3. chmod 755 clean.sh
  4. ./clean.sh
  5. ./build.sh

Building test

  1. cd test/
  2. chmod 755 build.sh
  3. chmod 755 clean.sh
  4. ./clean.sh
  5. ./build.sh

Run

Running src

We need to start the loadbalancer and server

cd src/cmake/build
./loadbalancer <ip with port for client to connect> <ip with port for servers to send heartbeat>
./server <my kv ip with port> <my raft ip with port> <lb ip with port>

Running test

To run read (calls get API) test

cd test
chmod run_read_test.sh
chmod run_write_test.sh
./run_read_test

To run write (calls put API) test

cd test
chmod run_read_test.sh
chmod run_write_test.sh
./run_write_test

Results will be redirected to /results Note: The run_read_test and run_write_test assume that the loadbalancer kv ip is 0.0.0.0:50051.

Deliverables

  1. Presentation here
  2. Demos here
  3. Report here

levelupdb's People

Contributors

benitakbritto avatar reetuparna avatar shreyansh21sharma avatar ganeriwalk avatar

Stargazers

Devansh Goenka avatar  avatar

Watchers

James Cloos avatar  avatar  avatar  avatar

levelupdb's Issues

Test cases for measurement

  • Single read
  • Concurrent read same key
  • Concurrent read diff key
  • Single write
  • Concurrent write same key
  • Concurrent write different key
  • Run

RAFT: Log Helper

  • replicated log persistent
  • replicated log volatile
  • term vote log persistent
  • term vote log volatile
  • other state volatile
  • Cmake

Addition of node (on demand)

  • Establish heartbeat with LB
  • Establish connection with leader
  • Get state from leader
    Should be identical to crash recovery cases? Corner cases?

Load Balancer

  • Route request based on consistency level
  • Heartbeat

Initial VM Setup

  • Create 4 machines for development on Azure
  • Open all network inbound and outbound ports
  • Ping machines
  • Install gRPC
  • Clone repository

Crash Recovery

  • Fetch logs from leader
  • Reconcile state with leader
  • Stop serving requests during recovery

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.