Git Product home page Git Product logo

blackinkgj / flash-translation-layer Goto Github PK

View Code? Open in Web Editor NEW
15.0 5.0 13.0 334 KB

:thinking: This repository aims to make a generic FTLs interface and device interface for SSD on the Linux system.

Home Page: https://blackinkgj.github.io/Flash-Translation-Layer-Documents/index.html

License: MIT License

C 94.62% Makefile 5.05% Dockerfile 0.33%
ssd ftl linux user-space-driver interface glib nand embedded zns zoned

flash-translation-layer's Introduction

Flash Translation Layer

build-and-testCodacy Badge

Overview

This repository contains the simple Flash Translation Layers made on the Linux Environments. Furthermore, this repository works on the Ramdisk and various devices(e.g., Zoned Block Device, bluedbm, etc.).

If you have any questions about this project, don't hesitate to contact the maintainer ([email protected]).

Build

These instructions are based on the Ubuntu(>=16.04) environments

Prerequisite

Before you build this repository, you must install some packages from the package manager.

sudo apt update -y
sudo apt install -y git make gcc g++ libglib2.0-dev libiberty-dev

After you download the packages, you must receive this project code using git clone like below.

git clone --recursive ${REPOSITORY_URL}

Now you move to this repository's project directory root by using mv.

If you want to use a module like the Zoned Block Device module, you must set the value of the USE_ZONE_DEVICE variable to 1 in the Makefile.

Additionally, you must install a valid library for each module. Each module's requirements are as follows.

Moreover, before you run the Zoned Block Device-based program, You must check that you give the super-user privileges to run the Zoned Block Device-based programs.

Test

Before you execute the test and related things, you must install the below tools.

You can check the source code status by using make check. If you want to generate test files, execute the below command.

make clean && make -j$(nproc) test USE_LOG_SILENT=1

After the build finish, you can get the various test files from the results. Run those files to test whether the project's module works correctly.

Execution

Suppose you want to generate a release program through the main.c, then you must execute the below commands.

make clean && make -j$(nproc)

Now, you can run that program by using ./a.out. Note that this repository is main.c file conducts the integration test of this project.

Installation

This project also supports generating the static library for using the external project. If you want to use it, please follow the below commands.

make clean
make -j$(nproc)
sudo make install

For building in the macOS

We are not providing native support for the macOS but providing Docker-based support for the macOS.

First, you need to create the builder image for building this project.

make docker-builder

After you create a docker image for building, run commands using make docker-make-${TARGET_RULE}. For example, you can use like:

make docker-make-test # same as `make test` in the Linux
make docker-make-integration-test # same as `make integration-test` in the Linux
make docker-make-all
make docker-make-check
make docker-make-flow

Example

After installing our shared library on your system, you can make your own program. You can refer to making your program from the example directory.

You can run this example as follows:

pushd example
make
./rw_example
make clean
popd

Benchmark

Build benchmark program by using:

make benchmark.out

See its usage by using:

./benchmark.out -h

For example, if you want to see sequential write performance on the ramdisk, type like:

./benchmark.out -m pgftl -d ramdisk -t write -j 4 -b 1048576 -n 100

It shows results like:

INFO:[interface/module.c:module_init(49)] flash initialize success
INFO:[interface/module.c:module_init(55)] submodule initialize success
INFO:[device/ramdisk/ramdisk.c:ramdisk_open(60)] ramdisk generated (size: 1073741824 bytes)
INFO:[device/ramdisk/ramdisk.c:ramdisk_open(77)] bitmap generated (size: 16392 bytes)
[parameters]
        - modules     pgftl
        - devices     ramdisk
        - workloads   write
        - jobs        4
        - block size  1048576
        - # of block  100
        - io size     100MiB
        - path        (null)
fill data start!
ready to read!
Processing: 100.00% [1829.49 MiB/s]
finish thread 0
finish thread 1
finish thread 2
finish thread 3
[job information]
id  time(s)   bw(MiB/s) iops      avg(ms)   max(ms)   min(ms)
=====
0   0.0644    1552.6847 100       0.6440    2.0855    0.1686
1   0.0530    1886.6408 100       0.5300    2.0855    0.1624
2   0.0522    1915.5273 100       0.5220    2.1562    0.1530
3   0.0541    1847.9744 100       0.5411    2.1562    0.1530
[crc status]
crc check success
INFO:[interface/module.c:module_exit(75)] submodule deallocates success
INFO:[interface/module.c:module_exit(83)] flash deallocates success
[parameters]
        - modules     pgftl
        - devices     ramdisk
        - workloads   write
        - jobs        4
        - block size  1048576
        - # of block  100
        - io size     100MiB
        - path        (null)

If you encounter a random-related error, please run commands as follows:

make benchmark.out USE_LEGACY_RANDOM=1

How to get this project's documents

You can get this program's documentation file by using doxygen -s Doxyfile. Also, you can get the flow of each function using make flow.

flash-translation-layer's People

Contributors

blackinkgj avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

pnuoslab pnu-oslab-org eric613-chan jsy0831 jhcho0114 allenk iliefu yummuywang wenjieqihust zhangqiang880

flash-translation-layer's Issues

BlueDBM driver may have an out-of-index problem

bluedbm_erase_end_request() does not check the counter array's boundary. It may cause an access validation error in the array.

Usually, the out-of-index error is raised from the operating system. However, bluedbm_erase_end_request()'s access validation might not be handled because it runs under the Bluedbm device.

Therefore, we must handle this problem.

Flash-Translation-Layer/device/bluedbm/bluedbm.c

Lines 35 to 47 in 8da8c10

static void bluedbm_erase_end_request(uint64_t segnum, uint8_t is_bad)
{
if (g_badseg_counter == NULL || g_erase_counter == NULL) {
pr_err("NULL pointer detected\n");
pr_err("\tg_badseg_counter: %p\n", g_badseg_counter);
pr_err("\tg_erase_counter: %p\n", g_erase_counter);
return;
}
g_atomic_int_inc(&g_erase_counter[segnum]);
if (is_bad) {
g_atomic_int_inc(&g_badseg_counter[segnum]);
}
}

Add recovery logic for page FTL

Current FTL does not have the recovery logic. In other words, if you turn off the system, all page mapping information and bad block table information will be deleted. Therefore, we must implement this.

Implement the cache feature for Page FTL

Page FTL has not been finished implementing the cache feature. It needs to implement.

Moreover, its architecture's cache shares read and write routines. The cache only can be updated by the write routine. In other words, the current cache implementation cannot efficiently absorb the I/O performance degradation.

Flash-Translation-Layer/ftl/page/page-read.c

Lines 82 to 92 in 964e3b5

#ifdef PAGE_FTL_USE_CACHE
cached = (struct device_request *)lru_get(pgftl->cache, lpn);
if (cached) {
memcpy(request->data, &((char *)cached->data)[offset],
request->data_len);
ret = request->data_len;
device_free_request(request);
pthread_mutex_unlock(&pgftl->mutex);
goto exception;
}
#endif

Adding wiki page

Add a wiki page to make it easier to understand our program architecture, implementations, and usage.

Support FUSE for making users can easily use this FTL

Supporting FUSE might make users can more easily use this. Moreover, users can benchmark the performance using FIO or so on.

However, as far as we knew, FUSE generates some I/O performance issue1.

Footnotes

  1. Bijlani, A., & Ramachandran, U. (2019). Extension framework for file systems in user space. In 2019 USENIX Annual Technical Conference (USENIX ATC 19) (pp. 121-134). ↩

Removing glib in the code

Due to its Glib existence, Arduino or other embedded boards cannot use this module. Therefore, I need to remove the Glib in the code.

Need to make SIMD abstract layer for memory functions

Although this project executes too much memory copying, it does not use the SIMD feature in the CPU. So, I created the ARM SIMD usage sample version and proved the performance enhanced. However, it does not take a generic one.

Therefore, we must abstract the memcpy or memset to support the generic SIMD regardless of the architectures.

Remove CI for ARM

Flash-Translation-Layer/.github/workflows/build.yml

Lines 37 to 58 in d74f061

build-and-test-arm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
submodules: recursive
- uses: pguyot/arm-runner-action@v2
with:
commands: |
grep MemTotal /proc/meminfo
sudo apt-get update -y && sudo apt-get clean
sudo apt-get install --no-install-recommends -y git make gcc g++ libglib2.0-dev libiberty-dev
sudo apt-get install --no-install-recommends -y cppcheck flawfinder cflow
sudo apt-get install --no-install-recommends -y python3-pip
sudo pip3 install lizard==1.17.0
make check
make flow
make
make clean && make test USE_LOG_SILENT=1
make clean && make integration-test
make clean && make benchmark.out
make clean && make benchmark.out USE_LEGACY_RANDOM=1

Add an example for using a shared library form

Some users request to know how to use a shared library form exactly. Therefore, we need to create an example directory for users can easily understand how to bind their program with this shared library.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.