Git Product home page Git Product logo

hadoop-kerberos's Introduction

Bloomberg Big Data and NoSQL Platform

1. Configure Docker networking

Hadoop requires reverse DNS. Under docker-compose, we require an external network named "com" for hosts to resolve forward and backwards.

docker network create com

2. Download a distro of Hadoop

3. Start it up

docker volume rm hadoopkerberos_server-keytab
docker-compose up -d --force-recreate --build

4. Run HDFS commands

docker exec -it nn.example /bin/bash
kinit -kt /var/keytabs/hdfs.keytab hdfs/nn.example.com
hdfs dfs -ls /

hadoop-kerberos's People

Contributors

ifilonenko avatar siryaro avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

hadoop-kerberos's Issues

container are getting exited.

`  ✔  at 12:35:59 AM  ─╮
╰─ docker-compose up -d --build ─╯
WARN[0000] network default: network.external.name is deprecated. Please set network.name with external: true
[+] Building 2.7s (43/82) docker:desktop-linux
=> [kerberos internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [kerberos internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.05kB 0.0s
=> [dn1 internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.05kB 0.0s
=> [dn1 internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [nn internal] load metadata for docker.io/library/centos:7 2.5s
=> [data-populator internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [data-populator internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.05kB 0.0s
=> [nn internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.05kB 0.0s
=> [nn internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [kerberos auth] library/centos:pull token for registry-1.docker.io 0.0s
=> [nn internal] load build context 0.0s
=> => transferring context: 883B 0.0s
=> [nn 1/25] FROM docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 0.0s
=> [dn1 internal] load build context 0.0s
=> => transferring context: 883B 0.0s
=> [kerberos internal] load build context 0.0s
=> => transferring context: 883B 0.0s
=> [data-populator internal] load build context 0.0s
=> => transferring context: 883B 0.0s
=> [data-populator] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1e9b26ed9785355d6c3c5bf6a051de832d20072d1b1b0d9f48f8c53152bc8c35 0.0s
=> => naming to docker.io/library/hadoop-kerberos-data-populator 0.0s
=> CACHED [nn 2/25] RUN yum -y install krb5-server krb5-workstation 0.0s
=> CACHED [nn 3/25] RUN yum -y install java-1.8.0-openjdk-headless 0.0s
=> CACHED [nn 4/25] RUN yum -y install apache-commons-daemon-jsvc 0.0s
=> CACHED [nn 5/25] RUN yum install net-tools -y 0.0s
=> CACHED [nn 6/25] RUN yum install telnet telnet-server -y 0.0s
=> CACHED [nn 7/25] RUN yum -y install which 0.0s
=> CACHED [nn 8/25] RUN yum install ntp -y 0.0s
=> CACHED [nn 9/25] RUN sed -i -e 's/#//' -e 's/default_ccache_name/# default_ccache_name/' /etc/krb5.conf 0.0s
=> CACHED [nn 10/25] RUN sed -i 's/server */server 0.centos.pool.ntp.org iburst/' /etc/ntp.conf 0.0s
=> CACHED [nn 11/25] RUN ntpd 0.0s
=> CACHED [nn 12/25] RUN useradd -u 1098 hdfs 0.0s
=> CACHED [nn 13/25] ADD hadoop-2.7.1.tar.gz / 0.0s
=> CACHED [nn 14/25] RUN ln -s hadoop-2.7.1 hadoop 0.0s
=> CACHED [nn 15/25] RUN chown -R -L hdfs /hadoop 0.0s
=> CACHED [nn 16/25] COPY core-site.xml /hadoop/etc/hadoop/ 0.0s
=> CACHED [nn 17/25] COPY hdfs-site.xml /hadoop/etc/hadoop/ 0.0s
=> CACHED [nn 18/25] COPY ssl-server.xml /hadoop/etc/hadoop/ 0.0s
=> CACHED [nn 19/25] COPY yarn-site.xml /hadoop/etc/hadoop/ 0.0s
=> CACHED [nn 20/25] COPY start-namenode.sh / 0.0s
=> CACHED [nn 21/25] COPY start-datanode.sh / 0.0s
=> CACHED [nn 22/25] COPY populate-data.sh / 0.0s
=> CACHED [nn 23/25] COPY start-kdc.sh / 0.0s
=> CACHED [nn 24/25] COPY people.json / 0.0s
=> CACHED [nn 25/25] COPY people.txt / 0.0s
=> [nn] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:e5ed52142dfa540ee267c05572bed1cf7e7185f9998169daf7e776f4f5f45784 0.0s
=> => naming to docker.io/library/hadoop-kerberos-nn 0.0s
=> [dn1] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:ea0f5e0b0fa134a64c1089897326b7255f5db8f9f1c9d4ae57531e5ae6097b9d 0.0s
=> => naming to docker.io/library/hadoop-kerberos-dn1 0.0s
=> [kerberos] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:ce7c699ab644d0febacc792a73359f9b578639621f3e4a6c3f5e2a1ea1350f20 0.0s
=> => naming to docker.io/library/hadoop-kerberos-kerberos 0.0s
[+] Running 5/5
✔ Volume "hadoop-kerberos_server-keytab" Created 0.0s
✔ Container dn1.example Started 0.1s
✔ Container kerberos.example Started 0.1s
✔ Container data-populator.example Started 0.1s
✔ Container nn.example Started 4.5s

╭─    ~/Downloads/hadoop-kerberos  on   master !5 ?1   ✔  took 8s   at 12:36:09 AM  ─╮
╰─ docker ps ─╯
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
556d42092d19 hadoop-kerberos-data-populator "/populate-data.sh" 11 seconds ago Up 5 seconds data-populator.example
dcc3a7457fa0 hadoop-kerberos-kerberos "/start-kdc.sh" 11 seconds ago Up 5 seconds kerberos.example

╭─    ~/Downloads/hadoop-kerberos  on   master !5 ?1   ✔  at 12:36:15 AM  ─╮
╰─ docker ps -a ─╯
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7958e8c44497 hadoop-kerberos-dn1 "/start-datanode.sh" 14 seconds ago Exited (1) 4 seconds ago dn1.example
556d42092d19 hadoop-kerberos-data-populator "/populate-data.sh" 14 seconds ago Up 8 seconds data-populator.example
dcc3a7457fa0 hadoop-kerberos-kerberos "/start-kdc.sh" 14 seconds ago Up 8 seconds kerberos.example
c954fc7fe307 hadoop-kerberos-nn "/start-namenode.sh" 14 seconds ago Exited (1) 5 seconds ago nn.example

╭─    ~/Downloads/hadoop-kerberos  on   master !5 ?1   ✔  at 12:36:18 AM  ─╮
╰─ ─╯`

This get resolved if I comment the chown part as below but then it given permission error for key tab file..

Please help !!!
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.