Git Product home page Git Product logo

hadoop's Introduction

xuybin/hadoop

hadoop

Run Hadoop cluster with docker-compose

In China

docker rmi -f registry.cn-shenzhen.aliyuncs.com/xuybin/hadoop && curl -L -s https://raw.githubusercontent.com/xuybin/hadoop/master/docker-compose-aliyun.yml >docker-compose.yml && docker-compose up -d
docker exec -it  master /bin/bash
    /start-hadoop.sh
    jps
    exit
docker-compose ps

Outside China

docker rmi -f xuybin/hadoop &&curl -L -s https://raw.githubusercontent.com/xuybin/hadoop/master/docker-compose.yml >docker-compose.yml && docker-compose up -d
docker exec -it  master /bin/bash
    /start-hadoop.sh
    jps
    exit
docker-compose ps

Optional

docker ps
docker network ls
docker network inspect  **_hadoop
docker volume ls
docker volume inspect **_hdfs-master[slave1,slave2]

Verification Hadoop cluster

NameNode:visit http://ip:8098
ResourceManager:visit http://ip:8099

Test Hadoop cluster

mkdir src &&echo "from f1 content" >src/f1.txt && echo "from f2 content" >src/f2.txt
hadoop fs -mkdir -p target && hdfs dfs -put ./src/* target
hdfs dfs -cat target/f1.txt
hdfs dfs -cat target/f2.txt
hadoop jar /hadoop/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.7.5-sources.jar org.apache.hadoop.examples.WordCount target target2
hdfs dfs -cat target2/*

Stop Hadoop cluster

docker exec -it  master /bin/bash
    /stop-hadoop.sh
    jps
    exit
docker-compose stop

Clean Hadoop cluster

docker-compose rm -f
docker volume ls
docker volume rm **_hdfs-master[slave1,slave2]
docker network ls
docker network rm  **_hadoop

hadoop's People

Stargazers

Roman avatar

Watchers

James Cloos avatar Yongbin Xu avatar

Forkers

shiyongabc

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.