Git Product home page Git Product logo

hdfs-inotify's Introduction

hdfs-inotify

See this link for info on iNotify, particularly slide #16.

You must run this tool as the hdfs user.

$ jar hdfs-inotify-example-uber.jar <HDFS URI> <TxId>

This is a quick and dirty example. If you omit the TxId arg, like this:

$ hadoop jar hdfs-inotify.jar hdfs://hadoop-node-cluster:<port_number>

The output might be quite verbose, as you will get all TxId information.

So you might want to start with a large TxId and then work backwards if you don't get any events. If the TxId is larger than the number of tx's then you will simply get no data back)

For my test on a new HDFS I will start will a TxId of 0:

$ hadoop jar hdfs-inotify-example.jar hdfs://emr-header-1.cluster-68492:9000 0

I see an output that ends like this:

...
TxId = 351352
event type = CREATE
  path = /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2015_07_10-20_29_11
  owner = hdfs
  ctime = 1436585351213

Ctrl+C to kill the app

From that you can see the last TxId was 351352.

You can then call the app like this to get all subsequent tx's:

$ sudo -u hdfs java -jar hdfs-inotify-example-uber.jar hdfs://brooklyn.onefoursix.com:8020 351352

While that is still running, in another session, create a couple of files in HDFS, then delete one (without using -skipTrash) and delete the other with -skipTrash.

You should see a couple of CREATE , RENAME, UNLINK, APPEND, and CLOSE like this:

TxId = 351411
event type = CREATE
  path = /user/mark/data106.txt._COPYING_
  owner = mark
  ctime = 1436585999907
TxId = 351412
event type = CLOSE
TxId = 351413
event type = RENAME
  src = /user/mark/data106.txt._COPYING_
  dst = /user/mark/data106.txt
  timestamp = 1436586000137
TxId = 351420
event type = RENAME
  src = /user/mark/data106.txt
  dst = /user/mark/.Trash/Current/user/mark/data106.txt
  timestamp = 1436586013973
TxId = 351421
event type = CREATE
  path = /user/mark/data107.txt._COPYING_
  owner = mark
  ctime = 1436586067256
TxId = 351425
event type = CLOSE
TxId = 351426
event type = RENAME
  src = /user/mark/data107.txt._COPYING_
  dst = /user/mark/data107.txt
  timestamp = 1436586067489
TxId = 351427
event type = UNLINK
  path = /user/mark/data107.txt
  timestamp = 1436586074079

hdfs-inotify's People

Contributors

mabreyes avatar

Stargazers

Trần Đức Nhân avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.