Git Product home page Git Product logo

hiliev / py-zfs-rescue Goto Github PK

View Code? Open in Web Editor NEW
22.0 3.0 3.0 74 KB

A very minimal user-space implementation of ZFS in Python and a tool for salvaging data from broken pools

Home Page: https://hiliev.eu/post/recovering-datasets-from-broken-zfs-raidz-pools/

License: BSD 3-Clause "New" or "Revised" License

Python 97.98% Makefile 0.57% Shell 1.45%
zfs zfs-pool rescue python dataset salvage data-recovery filesystem filesystem-library zfs-raid

py-zfs-rescue's Introduction

py-zfs-rescue

A very minimal implementation in Python 3 of ZFS in user-space for pool recovery purposes.

Background

This project evolved from a set of Python scripts for reading and displaying on-disk structures that the ZFS debugger zdb would not show. It is the culmination of the effort to salvage the data from a severly broken raidz1 array. More background information is available in this blog post.

What it is?

zfs_rescue is a Python 3 script that is able to read the structure of a ZFS pool provided an initial device that belongs to the pool and to extract various types of information from the pool:

  • list the accessible datasets with their sizes
  • recursively list the files in all or some of the datasets found
  • archive the content of all or some regular files in a given dataset

The code was developed specifically against a broken ZFS raidz1 pool created by an old Solaris 10 x86 system and thus handles:

  • ZFS version 10 on little-endian systems
  • pools that consist of a single mirror or raidz1 vdev (the mirror code should be able to handle single devices too)
  • for raidz1 the parity information is used to recreate the data from the failed device, if any
  • directories with small to moderately large number of elements
  • access to remote disks via a simple TCP/IP protocol

Thanks to the work of @eiselekd, additional support was added for:

  • LZ4 compression
  • Fletcher checksum validation
  • modern ZFS attributes

What it is not?

This is not a generic rescue tool or a filesystem debugger per se. It provides no command-line interface and all configuration is done by altering the source code. The output is quite technical and requires some understanding of the ZFS internals and on-disk structure.

The ZFS implementation is minimal and incomplete. It is basically in a "works for me" state. Notably the following features are missing:

  • support for really large directories (it could be implemented relatively easily)
  • validation of the block checksums -- currently the tool relies on all metadata being compressed and the LZJB decompressor failing with garbled input data
  • LZ4 and GZIP decompression
  • support for pools created on big-endian systems

There is minimal to no error recovery and encountering an unsupported object will abort the program. This is intentional as it helps easily spot unimplemented features and deviations from the specification.

How to use it?

Documentation is currently a WiP.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.