Git Product home page Git Product logo

ghdfs's Introduction

Build Status

GHDFS

Works with HDFS for common operations and Scala compatibility.

Installation

Package is under com.globo.bigdata.ghdfs

  • Include in your dependencies:
    "com.globo.bigdata" %% "ghdfs" % "0.0.13"

Usage

    val hdfs = HdfsManager(Properties.envOrNone("HADOOP_CONF_DIR"))
    
    hdfs.write(Path)
    
    hdfs.write(Path, InputStream)

    hdfs.read(Path)
    
    hdfs.status(Path)
    
    hdfs.move(Path, Path)
    
    hdfs.listFiles(Path, recursive = false).foreach(...)
    
    hdfs.delete(Path, recursive = true)

    etc...

Get Filesystem Instance

    hdfs.getFS.exists(hadoopPath)

Contribute

For development and contributing, please follow Contributing Guide and ALWAYS respect the Code of Conduct

ghdfs's People

Contributors

adinizsilva avatar fgrutsch avatar victormazevedo avatar felipe-sbatista avatar regismachado avatar dmvieira avatar renanoliveira avatar luanabm avatar

Stargazers

avatar Glaucia Esppenchutz avatar Joel Melo avatar avatar Gabriel Cardoso avatar

Watchers

Marcello Azambuja avatar Rafael Floriano da Silva avatar avatar Tiago Albineli Motta avatar Diego Pinheiro avatar Claudio Braga Leite avatar Kenji Yamamoto avatar avatar Gustavo Rezende avatar avatar Vinícius das Chagas Silva avatar Rafael Gonçalves avatar Igor Macaubas avatar Cainã Nunes avatar avatar Felipe Ramos Ferreira (Cabuto) avatar Hélio Correia avatar Guilherme Rezende avatar Romulo Costa Tavares avatar Cláudio Luz avatar avatar Gustavo Luz avatar Rubens Azambuja Neves Neto avatar Rodrigo Veiga avatar Marcio M. avatar Danilo Moura avatar Victor de Oliveira Areas avatar Wilson Júnior avatar Ismael Moreira avatar Sergio Jorge avatar James Cloos avatar avatar André Luiz Diniz avatar Mauricio Antunes avatar Antenor do Váu Cabrerisso avatar Rodrigo Lourenço avatar Ricardo Dias avatar avatar Guilherme Balestieri Bedin avatar Vivien avatar João Borsani avatar Marcelo Soares - Nimbfire avatar Bernardo Camilo avatar Tayane Moura avatar avatar William Marquardt avatar Yasmin Teles avatar André Nascimento avatar avatar Victor Hugo avatar Sergio Macedo avatar Susana Bouchardet avatar Leticia Freire avatar Daniel Santana Rocha avatar Sillas Siqueira Neto avatar Alexandre Yukio Yamashita avatar Kalil Cazes avatar Ylgner Becton avatar avatar Danilo Gila de Santana avatar avatar

Forkers

ffmmjj felipe-sbatista lucas-rufino fgrutsch luanabm arquiteturajoel victormazevedo santoscristiano

ghdfs's Issues

Add support for checksum string generation

Some times it's necessary to verify if a given file is equaled to another file inside hdfs, it could happen when we are trying to verify if the file content is different from an older version.

For that, we have to implement a method to get the string checksum from a file inside hdfs and return the checksum as a String to be used when needed.

For now, this issue doesn't relate to the generation of a cross filesystem checksum.

Add suport for checksum string verification

Some times it's necessary to verify if a given file is equaled to another file inside hdfs, it could happen when we are trying to verify if the file content is different from an older version of this file.

For that, we have to implement a method to check if a given string checksum is equaled to the checksum of a file inside hdfs and return a boolean.

For now, this issue doesn't relate to the generation of a cross filesystem checksum.

Expand the support of the hdfs filesystem api

It's necessary to expand the actual hdfs filesystem support, with new overloading methods and utilities to turn the usage of the hdfs more simple and fast for commons operations.

This issue doesn't expect the creation of method utilities or overloading that extrapolate the original filesystem api, for example, read and write in the same method or write and generate the checksum.

Build ghdfs version for each Hadoop client version

It's important to Ghdfs be able to run with a wide range of Hadoop versions and delivery a consistant and common api for all ranges, and on the other hand, provide specific api implementations for each version, enabling the version to be capable of delivering the most advanced hdfs APIs and implementations from future versions.

For an example of a real issue, here is: #3

#edited

Copyright holder and license

Hi,
Could you please add LICENSE file and put the copyright holder and license inside each file?

This way there is no chance of someone stumbling across a piece of code without being exposed to the license.

Thanks!

Add support to write in hdfs from a inputstream

Given an inputstream, it's necessary to write that stream in hdfs and return a true if the operation was successful or throw an exception if it's unable to write the file.

That issue is a method overloading for the actual write method.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.