Git Product home page Git Product logo

fournodedistributedarchitecture's Introduction

PowerloomAssignment

Usage: export PYTHONPATH=src; redis-server &; python3 src/server.py &; python3 simulate.py;

  • Peer (Node):

    • A Peer is a unanimous node, which is independently in itself both client and server.
    • A Peer does not need to be dependent some central system to function.
    • For now I will implement peer node's as just sockets which are bound to different ports on the same machine.
  • Implementation of a Node:

    • Each peer is represented by a class Node. This class will have a socket which will be binded to the local host with a unique port number.
    • A socket bind represents that the peer is ready to accept connections (tcp handshake) at that port.
    • A socket connect represents that the peer is ready to initiate the tcp handshake with the target peer.
  • Trusted Server (Tracker):

    • The trusted server is an api through which each peer can querry it for different types of information.
    • Peers can upload a payload string to the server through the API, and the server returns a payloadId back to the peer which is unique to that string
    • Each peer reports itself to the server, and the server maintains a list of all the peers that have reported to it.

TODO:

- [x] Successfully test the api by sending a register post request to the server
- [x] Send a payload string from a peer and get back the payloadId
- [x] API calls to register and querry for registered peers
- [x] Implement the hashing function that is required for the payload load string
- [x] Implement the payload Querry API and test it.
- [x] Create interface for a peer to querry with other peer about the chunk position
- [x] Make the threads daemon
- [x] A peer cannot accept a connection from other peer while it is in communication a previous peer (Implement using thread.join() call). This will work in convention with p2pchannels thread where the main peer thread will wait for p2pchannels thread to finish.
- [x] Make sure that each (host,port) combination is not repeated in the server for the list of peers connected to it.
- [x] Each peer needs to have a deregister function that will erase their entry from the peer list in the server
- [x] Log all the activities/communications into a log file
- [x] Use counters for each of the peers to make sure all their activities are measured.
- [x] Build functions for peers to querry them about their metrics of all the activities.
- [x] Peer and socket shutdown functions need to be created
- [x] Write proper simualtion code
- [X] Pre-deadline last commit: 7385171 (no alphabets in this hash)

# POST DEADLINE FIXES:
- [x] Error: pyaloadId missing
- [x] Reorganize folder into tests and src folder
- [x] Integrate github actions.
- [x] Fully integrated and tested all kinds of actions for both peers and server using github actions. If any error comes up after 
commits, it will be caught.

fournodedistributedarchitecture's People

Contributors

raghavendragaleppa avatar

Stargazers

 avatar

Watchers

Swaroop Hegde avatar  avatar

fournodedistributedarchitecture's Issues

Simulate script fails after few iterations

The simulate.py script fails after a few iterations where it tries to close a socket connection which already seems to have been unbound/closed.

The error dump usually looks like this:

====================Creating Peers====================
(('127.0.0.1', 1060)): Created a peer with ID: 201102_194301382849
INFO:     127.0.0.1:61173 - "POST /register HTTP/1.1" 307 Temporary Redirect
User ip: 127.0.0.1
INFO:     127.0.0.1:61173 - "POST /register/ HTTP/1.1" 200 OK
(('127.0.0.1', 1060)): {"message":"Successfully Registered"}
(('127.0.0.1', 1061)): Created a peer with ID: 201102_194301397055
INFO:     127.0.0.1:61175 - "POST /register HTTP/1.1" 307 Temporary Redirect
User ip: 127.0.0.1
INFO:     127.0.0.1:61175 - "POST /register/ HTTP/1.1" 200 OK
(('127.0.0.1', 1061)): {"message":"Successfully Registered"}
(('127.0.0.1', 1062)): Created a peer with ID: 201102_194301407516
INFO:     127.0.0.1:61177 - "POST /register HTTP/1.1" 307 Temporary Redirect
User ip: 127.0.0.1
INFO:     127.0.0.1:61177 - "POST /register/ HTTP/1.1" 200 OK
(('127.0.0.1', 1062)): {"message":"Successfully Registered"}
(('127.0.0.1', 1063)): Created a peer with ID: 201102_194301417877
INFO:     127.0.0.1:61179 - "POST /register HTTP/1.1" 307 Temporary Redirect
User ip: 127.0.0.1
INFO:     127.0.0.1:61179 - "POST /register/ HTTP/1.1" 200 OK
(('127.0.0.1', 1063)): {"message":"Successfully Registered"}
Deregister a peer
INFO:     127.0.0.1:61180 - "DELETE /deregister HTTP/1.1" 200 OK
(('127.0.0.1', 1063)): {"message":"The Peer 201102_194301417877 has been successfully deregistered"}
================================================================================

Verify Payload
INFO:     127.0.0.1:61185 - "POST /upload_payload HTTP/1.1" 307 Temporary Redirect
INFO:     127.0.0.1:61185 - "POST /upload_payload/ HTTP/1.1" 200 OK
(('127.0.0.1', 1062)): Successfully upload payload
(('127.0.0.1', 1062)): PayloadId: payload_FFAUAPYDAPPF
No payload exists to verify, uploading a payload
INFO:     127.0.0.1:61186 - "POST /get_payload HTTP/1.1" 307 Temporary Redirect
INFO:     127.0.0.1:61186 - "POST /get_payload/ HTTP/1.1" 200 OK
(('127.0.0.1', 1061)): Successfully Querry of payload_FFAUAPYDAPPF to the server
Using payload: payload_FFAUAPYDAPPF for verifying chunks
(('127.0.0.1', 1061)): IDs exchanged with: ('127.0.0.1', 61187)
(('127.0.0.1', 1062)): IDs exchanged with: ('127.0.0.1', 1061)
(('127.0.0.1', 1061))P2P: Started P2P channel with node: ('127.0.0.1', 61187)
(('127.0.0.1', 1062))P2P: Started P2P channel with node: ('127.0.0.1', 1061)
Verifying in proper order
(('127.0.0.1', 1062))P2P: Message sent to 201102_194301397055
{'results': [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True], 'chunk_available': True}
================================================================================

Querry a payload and get chunks
INFO:     127.0.0.1:61192 - "POST /get_payload HTTP/1.1" 307 Temporary Redirect
INFO:     127.0.0.1:61192 - "POST /get_payload/ HTTP/1.1" 200 OK
(('127.0.0.1', 1062)): Successfully Querry of payload_FFAUAPYDAPPF to the server
claimedString: 7z6N8mQOUrojIUUoRwByHJW3J9xcLaBHaBiQq2QgTakvnFueJZubFChcuL4iqkrP2CgYxYg
Root Hash: 2e62acad7221fcd328e759179f19af0dd2c9013594caf772a018e973
================================================================================

INFO:     127.0.0.1:61197 - "DELETE /deregister HTTP/1.1" 200 OK
(('127.0.0.1', 1061)): {"message":"The Peer 201102_194301397055 has been successfully deregistered"}
Traceback (most recent call last):
  File "simulate.py", line 47, in <module>
    node_a.kill()
  File "/Users/anomit/workspace/recruitment/powerloom_ta_2020/RaghavendraGaleppa/src/node.py", line 113, in kill
    self.shutdown()
  File "/Users/anomit/workspace/recruitment/powerloom_ta_2020/RaghavendraGaleppa/src/node.py", line 102, in shutdown
    self.sock.shutdown(socket.SHUT_RD)
OSError: [Errno 57] Socket is not connected

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.