Git Product home page Git Product logo

memcached-fuse-filesystem's Introduction

Memcached FUSE File System

Following file system uses memcached to operate on files and directories as if it was on real disk. Let's start overviewing memcached class

  1. Memcached Class overview

        typedef struct memcached{
            char *addr;
            int port;
            int fd;
        } memcached;

    Memcached class is responsible for sending and recieving data over network. struct memcached is one that saves file descriptor that is needed to send and recieve data. each of given function returns integer type value, whis is one of the following:

    ENUM {STORED, DELETED, NOT_FOUND, NOT_STORED, ERROR };

    following enumerations are used to check errors while communicating to memcached. main error checking is done on the lowest level. so we can avoid any of unnecessary bugs that may have occured.

    with given memcached struct we also have struct which saves information about requested key

    typedef struct mm_data_info{
        char *key,*value;
        int flags, ttl, size;
    } mm_data_info;

    following mm_data_info struct is responsiible for saving all needed parameters that memcached needs to get/save value. In this class malloc()-s are rarely used because it is a must for the lowest class to work as fast as possible

  2. Chunk Class overview

    typedef struct chunk
    {
        int ind;
        char data[DATA_SIZE];
        char key[MAX_FNAME];
    } chunk;

    chunk class is responsible for saving some part of data. each chunk can save up to DATA_SIZE bytes. also each of them have variable ind. which is indicator of current size that is filled in this chunk.

  3. Content Class overview

    typedef struct content
    {
        char path[MAX_FNAME];
        int size;
    } content;

    content class is responsible to save full data of some directory or file. each content is assigned to some file or directory. which we have information from path variable. size is indicator of how much bytes are written in this current content

  4. (1) File Class overview

    typedef struct file{
        int _NOT_USED;
        char file_name[MAX_FNAME];
        int is_hardlink;
        int hardlink_count;
        char hardlink_name[MAX_FNAME];
        content ex_cn;
        content cn;
        int is_linked;
        uid_t uid;
        gid_t gid;
        mode_t mode;
    } file; 

    file class is responsible for creating files and operating over it. File name is full path of current file. which is the key to the memcached. also here we have two content classes 'cn' is responsible forsaving real data, as for 'ex_cn' is is used to save external attributes.

  5. (2) Directory Class overview

    typedef struct dir
    {
        char dir_name[MAX_FNAME];
        content ex_cn;
        content cn;
        int is_linked;
        uid_t uid;
        gid_t gid;
        mode_t mode;
    } dir;

    Directory class is responsible for creating and saving directories. same structure as files (with least parameters) but implementation is different.

    Key Details

    1). Hierarchy : current file system's main key is that it can be easily refactored or modified. Each of the class is on its own and only know what it really needs to know about lower classes.for example chunk class has no idea about content and Content has no idea what are directories or files. same goes to memcached. It is only interface that we use to save chunks and informations and thus have no idea that we are using it for file system. This architecture strongly follows decoupling of classes and dependancy injection.

      _______________
     |               |
     |   DIRECTORY   |_____
     |_______________|     |      _______________       ___________
                           |     |               |     |           |
                           |====>|    CONTENT    |====>|   CHUNK   |
      _______________      |     |_______________|     |___________|
     |               |_____|
     |     FILE      |
     |_______________| 
    

    2). Random Acces Memory :

    Every chunk operation in this file system in in o(1). each chunk is indexed using file name and saved in the memcached as follows : "$(index_of_chunk)$(file_full_path)"

    Example:

    We create file named test.txt inside directory dir1. full path would be : /dir1/test.txt now imagine this file's size is 1 GB. and we want to access nth byte we just calculate index knowing that each chunk contains daa of DATA_SIZE bytes: index = n / DATA_SIZE needed chunk key = "$(index)/dir1/test.txt"

    3).Saving Data (Xattr)

    problem here was to save data without delimiters and using as small space as possible. This solution was inspired from DNS protocol, which sends data as follows: first it writes how many bytes should you read. and then that number of bytes. after that again how many bytes you want and so on...

    Example:

    We want to save data {key:value} we would save in chunk next thing 3key5value

    but as you might have thought this would not work for strings which have more than 10 bytes. So I decided to create unique saving and created constant length number, so {key:value} would be saved as 003key005value

memcached-fuse-filesystem's People

Contributors

csenshi avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.