Git Product home page Git Product logo

lakefs's Introduction

Apache License Go tests status Node tests status nessie integration tests status Artifact HUB code of conduct

What is lakeFS

lakeFS is an open source tool that transforms your object storage into a Git-like repository. It enables you to manage your data lake the way you manage your code.

With lakeFS you can build repeatable, atomic and versioned data lake operations - from complex ETL jobs to data science and analytics.

lakeFS supports AWS S3, Azure Blob Storage and Google Cloud Storage as its underlying storage service. It is API compatible with S3, and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, Presto, etc.

For more information see the official lakeFS documentation.

Capabilities

In Development

  • Experiment - try new tools, upgrade versions, and evaluate code changes in isolation. By creating a branch of the data you get an isolated snapshot to run and compare experiments over, while others are not exposed.
  • Debug - checkout specific commits in a repository’s commit history to materialize consistent, historical versions of your data.
  • Collaborate - leverage isolated branches managed by metadata (not copies of files) to work in parallel.

During Deployment

  • Version Control - deploy data safely with CI/CD workflows borrowed from software engineering best practices. Ingest new data onto an isolated branch, perform data validations, then add to production through a merge operation.
  • Test - define pre-merge and pre-commit hooks to run tests that enforce schema and validate properties of the data to catch issues before they reach production.

In Production

  • Roll Back - recover from errors by instantly reverting data to a former, consistent snapshot of the data lake. Choose any commit in a repository’s commit history to revert in one atomic action.
  • Troubleshoot - investigate production errors by starting with a snapshot of the inputs to the failed process.
  • Cross-collection Consistency - provide consumers multiple synchronized collections of data in one atomic, revertable action.

Getting Started

Docker (MacOS, Linux)

  1. Ensure you have Docker & Docker Compose installed on your computer.

  2. Run the following command:

    curl https://compose.lakefs.io | docker-compose -f - up
  3. Open http://127.0.0.1:8000/setup in your web browser to set up an initial admin user, used to login and send API requests.

Docker (Windows)

  1. Ensure you have Docker installed.

  2. Run the following command in PowerShell:

    Invoke-WebRequest https://compose.lakefs.io | Select-Object -ExpandProperty Content | docker-compose -f - up
  3. Open http://127.0.0.1:8000/setup in your web browser to set up an initial admin user, used to login and send API requests.

Download the Binary

Alternatively, you can download the lakeFS binaries and run them directly.

Binaries are available at https://github.com/treeverse/lakeFS/releases.

Setting up a repository

Please follow the Guide to Get Started to set up your local lakeFS installation.

For more detailed information on how to set up lakeFS, please visit the lakeFS documentation.

Community

Stay up to date and get lakeFS support via:

  • Slack (to get help from our team and other users).
  • Twitter (follow for updates and news)
  • YouTube (learn from video tutorials)
  • Contact us (for anything)

More information

Licensing

lakeFS is completely free and open source and licensed under the Apache 2.0 License.

lakefs's People

Contributors

adunuthulan avatar ari-hacks avatar arielshaqed avatar daniel-shuy avatar daniloff200 avatar dependabot[bot] avatar eden-ohana avatar einato-zz avatar eylonronen avatar guy-har avatar itaiad200 avatar itaidavid avatar jbampton avatar johnnyaug avatar karentamrazyan avatar kzuri avatar lynnro314 avatar mofolactic avatar nopcoder avatar ozkatz avatar peacing avatar rohansahana avatar sarathsp06 avatar shamikakumar avatar sinithh avatar sufiyan1997 avatar talsofer avatar tzahij avatar xubofei1983 avatar yaelriv avatar

Stargazers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.