sequenceiq / hadoop-docker Goto Github PK

View Code? Open in Web Editor NEW

1.2K 89.0 560.0 84 KB

Hadoop docker image

Home Page: https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/

License: Apache License 2.0

Shell 15.25% Dockerfile 84.75%

hadoop-docker's Introduction

This project isn't active. Please don't use it at all!!!

Apache Hadoop 2.7.1 Docker image

Note: this is the master branch - for a particular Hadoop version always check the related branch

A few weeks ago we released an Apache Hadoop 2.3 Docker image - this quickly become the most popular Hadoop image in the Docker registry.

Following the success of our previous Hadoop Docker images, the feedback and feature requests we received, we aligned with the Hadoop release cycle, so we have released an Apache Hadoop 2.7.1 Docker image - same as the previous version, it's available as a trusted and automated build on the official Docker registry.

FYI: All the former Hadoop releases (2.3, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2, 2.6.0) are available in the GitHub branches or our Docker Registry - check the tags.

Build the image

If you'd like to try directly from the Dockerfile you can build the image as:

docker build  -t sequenceiq/hadoop-docker:2.7.1 .

Pull the image

The image is also released as an official Docker image from Docker's automated build repository - you can always pull or refer the image when launching containers.

docker pull sequenceiq/hadoop-docker:2.7.1

Start a container

In order to use the Docker image you have just build or pulled use:

Make sure that SELinux is disabled on the host. If you are using boot2docker you don't need to do anything.

docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash

Testing

You can run one of the stock examples:

cd $HADOOP_PREFIX
# run the mapreduce
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'

# check the output
bin/hdfs dfs -cat output/*

Hadoop native libraries, build, Bintray, etc

The Hadoop build process is no easy task - requires lots of libraries and their right version, protobuf, etc and takes some time - we have simplified all these, made the build and released a 64b version of Hadoop nativelibs on this Bintray repo. Enjoy.

Automate everything

As we have mentioned previousely, a Docker file was created and released in the official Docker repository

hadoop-docker's People

Contributors

Stargazers

Watchers

Forkers

rtvt123 sayden labianchin matyix pmahent thurday gyliu513 thataustin alchen99 hadoocn uwsampa jghoman pkubuntu biharitomi closedloop soa4java bioshrek c3h3 jfenc91 flaing mkscala bossjones casshern0330 paoloantinori anthonyikeda arshak showyou reference-project kusimari luk4sschr0d3r watermen nolimitid cobjet-rd favey minimicall moerstw supermy machbio dasgoll sandersliu xuanvs lingya banitag1 josephwinston ochamber giserh obinsc liuqian1990 is-yangchen tangzhankun cnh cabichahine berngp sdsouza8440 datashed-lewis skibaa alexchantastic mackertd sroegner jinshen-cn banno ifa6 moherx onkarkadam7 minger kadamonkar chandrasekharlagadapati ming2050 dsouzasr maxlpy charlesa101 xzflin xxljlxx vleushin kidiyoor cloudconsulted asifiqbal madhavan001 topblue qdubious yerly mavencode nalingarg2 rpwr021 sneakyz pavanpavuluri heamon7 rxacevedo piaolinzhi engineerball yobibytes jgagnon1 hosenter xwzpp ericchang omusico cmosetick zhaodragon atomicjets ogarcialop

hadoop-docker's Issues

why not export /usr/local/hadoop/bin to $PATH?

Error writing file

Run

docker run -it --rm --name=hadoopserver -p 8030:8030 -p 8040:8040 -p 8042:8042 -p 8088:8088 -p 19888:19888 -p 49707:49707 -p 50010:50010 -p 50020:50020 -p 50070:50070 -p 50075:50075 -p 50090:50090 -p 9000:9000 sequenceiq/hadoop-docker:latest /etc/bootstrap.sh –d

App

public static void main(String[] args) throws IOException {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://localhost:9000");

        System.setProperty("HADOOP_USER_NAME", "root");
        System.setProperty("hadoop.home.dir", "/");

        FileSystem fileSystem = FileSystem.get(conf);

        try (FSDataOutputStream out = fileSystem.create(new Path("test.txt"), true)) {
            out.write("Test".getBytes());
        }
    }

error

org.apache.hadoop.ipc.RemoteException: File /user/root/test.txt could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3067)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:722)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)
	at org.apache.hadoop.ipc.Client.call(Client.java:1427)
	at org.apache.hadoop.ipc.Client.call(Client.java:1337)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy13.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:440)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
	at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1733)
	at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1536)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658)

what am I doing wrong?

Report of Using hadoop-docker in Docker Toolkit in Windows

Hi. Just in case anyone wants to use this image in Windows. I did the following. In the Docker Toolbox, you can run the Docker Quickstart Terminal. In the terminal, you can issue Docker commands. However, winpty is needed if you want an interactive session with your Docker container. There are some issues with how winpty passes the command line to Docker.

In order to start your image, I needed to re-tag it to avoid the slash:

docker tag sequenceiq/hadoop-docker hadoop-docker

Once that was done, I found that I could not pass the /etc/bootstrap.sh on the command line. winpty had trouble. Instead I ran this:

winpty docker run -it hadoop-docker bash

Once inside the container, I ran the bootstrap script and was able to use the hdfs command immediately after.

/etc/bootstrap.sh bash
/usr/local/hadoop/bin/hdfs dfs -ls /

Thanks!

Hadoop 2.4.1

Hadoop 2.4.1 official repository is not found. I manually checked to their repository, and it's gone. So when build from Dockerfile it will broken because there are no file to download.

Impossible to run haddop

Hello,
Just have installed hadoop-docker inside kitematik, the web interface is working, but when i try to execute this command 👍 in the bash
hadoop fs -mkdir /dblp

ive got this error;
bash : hadoop : command not found,

Please fix this, i can't do any hadoop command.

Thank you

Port 9000 (namenode IPC) not exposed

If one wants to access HDFS from another container port 9000 needs to be exposed.

Output of docker ps:

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                                                                                                                                                                                                                                          NAMES
3835390f5f2a        flume                            "start-flume"            15 minutes ago      Up 15 minutes                                                                                                                                                                                                                                                                                                                                                                                                      flume-example-a1
2ff5c8467ddc        sequenceiq/hadoop-docker:2.7.0   "/etc/bootstrap.sh -d"   4 hours ago         Up 4 hours          0.0.0.0:32828->2122/tcp, 0.0.0.0:32827->8030/tcp, 0.0.0.0:32826->8031/tcp, 0.0.0.0:32825->8032/tcp, 0.0.0.0:32824->8033/tcp, 0.0.0.0:32823->8040/tcp, 0.0.0.0:32822->8042/tcp, 0.0.0.0:32821->8088/tcp, 0.0.0.0:32820->19888/tcp, 0.0.0.0:32819->49707/tcp, 0.0.0.0:32818->50010/tcp, 0.0.0.0:32817->50020/tcp, 0.0.0.0:32816->50070/tcp, 0.0.0.0:32815->50075/tcp, 0.0.0.0:32814->50090/tcp   hadoop

Output of docker port 2ff5c8467ddc | sort -n

2122/tcp -> 0.0.0.0:32828
8030/tcp -> 0.0.0.0:32827
8031/tcp -> 0.0.0.0:32826
8032/tcp -> 0.0.0.0:32825
8033/tcp -> 0.0.0.0:32824
8040/tcp -> 0.0.0.0:32823
8042/tcp -> 0.0.0.0:32822
8088/tcp -> 0.0.0.0:32821
19888/tcp -> 0.0.0.0:32820
49707/tcp -> 0.0.0.0:32819
50010/tcp -> 0.0.0.0:32818
50020/tcp -> 0.0.0.0:32817
50070/tcp -> 0.0.0.0:32816
50075/tcp -> 0.0.0.0:32815
50090/tcp -> 0.0.0.0:32814

Contents of /usr/local/hadoop/etc/hadoop/core-site.xml in the container:

  <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://2ff5c8467ddc:9000</value>
      </property>
  </configuration>

I believe the above should be localhost or 127.0.0.1. Running netstat will confirm the port is bound to another IP address:

bash-4.1# netstat -antp | grep LISTEN | grep :9000
tcp        0      0 172.17.0.2:9000             0.0.0.0:*                   LISTEN      130/java

only start HDFS on bootstrap

It seems both HDFS and YARN are started on bootstrap. Is there a way to just launch HDFS when YARN is not needed ?

this repo not follow https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/

is it need to update?

Dockerfile should specify a particular pam tag to nail down the linux distro

I see that your docker-pam image can either be Ubuntu or CentOS. The current docker-pam Dockerfile is FROM ubuntu:14.04, but the pam image used in the hadoop-docker image that is part of your spark:1.1.1 image seems to have gotten a "latest" docker-pam at a time when it was centos-6.5.

For clarity and robustness, I suggest changing your Dockerfile to specify a tag on pam.

FWIW, assuming it works, I'd prefer having the Spark docker container use Ubuntu 14.04.

And thanks for your images - much appreciated!

install hadoop docker on multi hosts

How to use this docker image to run hadoop cluster across many physical servers ? could you give some steps to follow ?

How do I enable simple auth - SIMPLE authentication is not enabled. Available:[TOKEN]

Installed docker container, but when I try to connect to it, I am getting auth errors.
I tried to enable simple auth but the docs are not lining up to what I see installed.

Error message

SIMPLE authentication is not enabled.  Available:[TOKEN]

$ hadoop fs -fs hdfs://192.168.99.100:32846 -ls
16/02/07 15:11:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: SIMPLE authentication is not enabled.  Available:[TOKEN]

I am getting the same error from Spark

        final SparkConf sparkConfig = new SparkConf().setAppName("baseball");

        if (args.length == 0) {
            sparkConfig.setMaster("local[4]");
        }

        final JavaSparkContext sparkContext = new JavaSparkContext(sparkConfig);
        //final JavaRDD<String> lines = sparkContext.textFile("/opt/baseball/Master.csv");
        final JavaRDD<String> lines = sparkContext.textFile("hdfs://192.168.99.100:32846/opt/baseball/Master.csv");

        final JavaRDD<String> noHeader = lines.filter(line -> !line.startsWith("playerID"));

Exception in thread "main" org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1681)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

Remote Access Hadoop Instance

When i access hadoop container from my windows machine from java code.

Windows: 8.1
Hadoop Version: 2.5.1
Docker Container Port bind:
0.0.0.0:8030->8030/tcp, 0.0.0.0:8031->8031/tcpOS: Windows 8.1

CODE:
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://192.168.59.103:8030/user/zeeshan");
conf.set("mapred.job.tracker", "192.168.59.103:8031");
conf.set("hadoop.job.ugi", "hbase");
FileSystem hdfs = FileSystem.get(conf);
hdfs.createNewFile(new Path("/user/zeeshan/test"));

ERROR:
org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN]

call from sqoop to haoop : Connection refused

hello :)

we make the docker-sqoop from hadoop-docker .
And we create some link and job, that is ok.
But when we start a job,it came a error:

Call From 6b428b7ad50d/172.17.0.3 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

for more details:

  "message": "Unexpected exception",
  "cause": {
    "message": "Call From 6b428b7ad50d/172.17.0.3 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused",
    "cause": {
      "message": "Connection refused",
      "stack-trace": [{
        "file": "SocketChannelImpl.java",
        "line": -2,
        "class": "sun.nio.ch.SocketChannelImpl",
        "method": "checkConnect"
      }, {
        "file": "SocketChannelImpl.java",
        "line": 739,
        "class": "sun.nio.ch.SocketChannelImpl",
        "method": "finishConnect"
      }, {
        "file": "SocketIOWithTimeout.java",
        "line": 206,
        "class": "org.apache.hadoop.net.SocketIOWithTimeout",
        "method": "connect"
      }, {
        "file": "NetUtils.java",
        "line": 531,
        "class": "org.apache.hadoop.net.NetUtils",
        "method": "connect"
      }, {
        "file": "NetUtils.java",
        "line": 495,
        "class": "org.apache.hadoop.net.NetUtils",
        "method": "connect"
      }, {
        "file": "Client.java",
        "line": 609,
        "class": "org.apache.hadoop.ipc.Client$Connection",
        "method": "setupConnection"
      }, {
        "file": "Client.java",
        "line": 707,
        "class": "org.apache.hadoop.ipc.Client$Connection",
        "method": "setupIOstreams"
      }, {
        "file": "Client.java",
        "line": 370,
        "class": "org.apache.hadoop.ipc.Client$Connection",
        "method": "access$2800"
      }, {
        "file": "Client.java",
        "line": 1529,
        "class": "org.apache.hadoop.ipc.Client",
        "method": "getConnection"
      }, {
        "file": "Client.java",
        "line": 1446,
        "class": "org.apache.hadoop.ipc.Client",
        "method": "call"
      }, {
        "file": "Client.java",
        "line": 1407,
        "class": "org.apache.hadoop.ipc.Client",
        "method": "call"
      }, {
        "file": "ProtobufRpcEngine.java",
        "line": 229,
        "class": "org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker",
        "method": "invoke"
      }, {
        "file": null,
        "line": -1,
        "class": "com.sun.proxy.$Proxy18",
        "method": "getFileInfo"
      }, {
        "file": "ClientNamenodeProtocolTranslatorPB.java",
        "line": 771,
        "class": "org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB",
        "method": "getFileInfo"
      }, {
        "file": "NativeMethodAccessorImpl.java",
        "line": -2,
        "class": "sun.reflect.NativeMethodAccessorImpl",
        "method": "invoke0"
      }, {
        "file": "NativeMethodAccessorImpl.java",
        "line": 57,
        "class": "sun.reflect.NativeMethodAccessorImpl",
        "method": "invoke"
      }, {
        "file": "DelegatingMethodAccessorImpl.java",
        "line": 43,
        "class": "sun.reflect.DelegatingMethodAccessorImpl",
        "method": "invoke"
      }, {
        "file": "Method.java",
        "line": 606,
        "class": "java.lang.reflect.Method",
        "method": "invoke"
      }, {
        "file": "RetryInvocationHandler.java",
        "line": 187,
        "class": "org.apache.hadoop.io.retry.RetryInvocationHandler",
        "method": "invokeMethod"
      }, {
        "file": "RetryInvocationHandler.java",
        "line": 102,
        "class": "org.apache.hadoop.io.retry.RetryInvocationHandler",
        "method": "invoke"
      }, {
        "file": null,
        "line": -1,
        "class": "com.sun.proxy.$Proxy19",
        "method": "getFileInfo"
      }, {
        "file": "DFSClient.java",
        "line": 2116,
        "class": "org.apache.hadoop.hdfs.DFSClient",
        "method": "getFileInfo"
      }, {
        "file": "DistributedFileSystem.java",
        "line": 1305,
        "class": "org.apache.hadoop.hdfs.DistributedFileSystem$22",
        "method": "doCall"
      }, {
        "file": "DistributedFileSystem.java",
        "line": 1301,
        "class": "org.apache.hadoop.hdfs.DistributedFileSystem$22",
        "method": "doCall"
      }, {
        "file": "FileSystemLinkResolver.java",
        "line": 81,
        "class": "org.apache.hadoop.fs.FileSystemLinkResolver",
        "method": "resolve"
      }, {
        "file": "DistributedFileSystem.java",
        "line": 1301,
        "class": "org.apache.hadoop.hdfs.DistributedFileSystem",
        "method": "getFileStatus"
      }, {
        "file": "FileSystem.java",
        "line": 1424,
        "class": "org.apache.hadoop.fs.FileSystem",
        "method": "exists"
      }, {
        "file": "HdfsToInitializer.java",
        "line": 65,
        "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer$1",
        "method": "run"
      }, {
        "file": "HdfsToInitializer.java",
        "line": 60,
        "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer$1",
        "method": "run"
      }, {
        "file": "AccessController.java",
        "line": -2,
        "class": "java.security.AccessController",
        "method": "doPrivileged"
      }, {
        "file": "Subject.java",
        "line": 415,
        "class": "javax.security.auth.Subject",
        "method": "doAs"
      }, {
        "file": "UserGroupInformation.java",
        "line": 1657,
        "class": "org.apache.hadoop.security.UserGroupInformation",
        "method": "doAs"
      }, {
        "file": "HdfsToInitializer.java",
        "line": 60,
        "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer",
        "method": "initialize"
      }, {
        "file": "HdfsToInitializer.java",
        "line": 37,
        "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer",
        "method": "initialize"
      }, {
        "file": "JobManager.java",
        "line": 520,
        "class": "org.apache.sqoop.driver.JobManager$1",
        "method": "call"
      }, {
        "file": "JobManager.java",
        "line": 517,
        "class": "org.apache.sqoop.driver.JobManager$1",
        "method": "call"
      }, {
        "file": "ClassUtils.java",
        "line": 281,
        "class": "org.apache.sqoop.utils.ClassUtils",
        "method": "executeWithClassLoader"
      }, {
        "file": "JobManager.java",
        "line": 516,
        "class": "org.apache.sqoop.driver.JobManager",
        "method": "initializeConnector"
      }, {
        "file": "JobManager.java",
        "line": 424,
        "class": "org.apache.sqoop.driver.JobManager",
        "method": "createJobRequest"
      }, {
        "file": "JobManager.java",
        "line": 317,
        "class": "org.apache.sqoop.driver.JobManager",
        "method": "start"
      }, {
        "file": "JobRequestHandler.java",
        "line": 353,
        "class": "org.apache.sqoop.handler.JobRequestHandler",
        "method": "startJob"
      }, {
        "file": "JobRequestHandler.java",
        "line": 114,
        "class": "org.apache.sqoop.handler.JobRequestHandler",
        "method": "handleEvent"
      }, {
        "file": "JobServlet.java",
        "line": 84,
        "class": "org.apache.sqoop.server.v1.JobServlet",
        "method": "handlePutRequest"
      }, {
        "file": "SqoopProtocolServlet.java",
        "line": 81,
        "class": "org.apache.sqoop.server.SqoopProtocolServlet",
        "method": "doPut"
      }, {
        "file": "HttpServlet.java",
        "line": 710,
        "class": "javax.servlet.http.HttpServlet",
        "method": "service"
      }, {
        "file": "HttpServlet.java",
        "line": 790,
        "class": "javax.servlet.http.HttpServlet",
        "method": "service"
      }, {
        "file": "ServletHolder.java",
        "line": 808,
        "class": "org.eclipse.jetty.servlet.ServletHolder",
        "method": "handle"
      }, {
        "file": "ServletHandler.java",
        "line": 1669,
        "class": "org.eclipse.jetty.servlet.ServletHandler$CachedChain",
        "method": "doFilter"
      }, {
        "file": "AuthenticationFilter.java",
        "line": 595,
        "class": "org.apache.hadoop.security.authentication.server.AuthenticationFilter",
        "method": "doFilter"
      }, {
        "file": "DelegationTokenAuthenticationFilter.java",
        "line": 291,
        "class": "org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter",
        "method": "doFilter"
      }, {
        "file": "AuthenticationFilter.java",
        "line": 554,
        "class": "org.apache.hadoop.security.authentication.server.AuthenticationFilter",
        "method": "doFilter"
      }, {
        "file": "ServletHandler.java",
        "line": 1652,
        "class": "org.eclipse.jetty.servlet.ServletHandler$CachedChain",
        "method": "doFilter"
      }, {
        "file": "ServletHandler.java",
        "line": 585,
        "class": "org.eclipse.jetty.servlet.ServletHandler",
        "method": "doHandle"
      }, {
        "file": "ContextHandler.java",
        "line": 1127,
        "class": "org.eclipse.jetty.server.handler.ContextHandler",
        "method": "doHandle"
      }, {
        "file": "ServletHandler.java",
        "line": 515,
        "class": "org.eclipse.jetty.servlet.ServletHandler",
        "method": "doScope"
      }, {
        "file": "ContextHandler.java",
        "line": 1061,
        "class": "org.eclipse.jetty.server.handler.ContextHandler",
        "method": "doScope"
      }, {
        "file": "ScopedHandler.java",
        "line": 141,
        "class": "org.eclipse.jetty.server.handler.ScopedHandler",
        "method": "handle"
      }, {
        "file": "HandlerWrapper.java",
        "line": 97,
        "class": "org.eclipse.jetty.server.handler.HandlerWrapper",
        "method": "handle"
      }, {
        "file": "Server.java",
        "line": 499,
        "class": "org.eclipse.jetty.server.Server",
        "method": "handle"
      }, {
        "file": "HttpChannel.java",
        "line": 310,
        "class": "org.eclipse.jetty.server.HttpChannel",
        "method": "handle"
      }, {
        "file": "HttpConnection.java",
        "line": 257,
        "class": "org.eclipse.jetty.server.HttpConnection",
        "method": "onFillable"
      }, {
        "file": "AbstractConnection.java",
        "line": 540,
        "class": "org.eclipse.jetty.io.AbstractConnection$2",
        "method": "run"
      }, {
        "file": "ThreadPoolExecutor.java",
        "line": 1145,
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "method": "runWorker"
      }, {
        "file": "ThreadPoolExecutor.java",
        "line": 615,
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "method": "run"
      }, {"file": "Thread.java", "line": 745, "class": "java.lang.Thread", "method": "run"}],
      "class": "java.net.ConnectException"
    },
    "stack-trace": [{
      "file": "NativeConstructorAccessorImpl.java",
      "line": -2,
      "class": "sun.reflect.NativeConstructorAccessorImpl",
      "method": "newInstance0"
    }, {
      "file": "NativeConstructorAccessorImpl.java",
      "line": 57,
      "class": "sun.reflect.NativeConstructorAccessorImpl",
      "method": "newInstance"
    }, {
      "file": "DelegatingConstructorAccessorImpl.java",
      "line": 45,
      "class": "sun.reflect.DelegatingConstructorAccessorImpl",
      "method": "newInstance"
    }, {
      "file": "Constructor.java",
      "line": 526,
      "class": "java.lang.reflect.Constructor",
      "method": "newInstance"
    }, {
      "file": "NetUtils.java",
      "line": 792,
      "class": "org.apache.hadoop.net.NetUtils",
      "method": "wrapWithMessage"
    }, {
      "file": "NetUtils.java",
      "line": 732,
      "class": "org.apache.hadoop.net.NetUtils",
      "method": "wrapException"
    }, {
      "file": "Client.java",
      "line": 1480,
      "class": "org.apache.hadoop.ipc.Client",
      "method": "call"
    }, {
      "file": "Client.java",
      "line": 1407,
      "class": "org.apache.hadoop.ipc.Client",
      "method": "call"
    }, {
      "file": "ProtobufRpcEngine.java",
      "line": 229,
      "class": "org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker",
      "method": "invoke"
    }, {
      "file": null,
      "line": -1,
      "class": "com.sun.proxy.$Proxy18",
      "method": "getFileInfo"
    }, {
      "file": "ClientNamenodeProtocolTranslatorPB.java",
      "line": 771,
      "class": "org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB",
      "method": "getFileInfo"
    }, {
      "file": "NativeMethodAccessorImpl.java",
      "line": -2,
      "class": "sun.reflect.NativeMethodAccessorImpl",
      "method": "invoke0"
    }, {
      "file": "NativeMethodAccessorImpl.java",
      "line": 57,
      "class": "sun.reflect.NativeMethodAccessorImpl",
      "method": "invoke"
    }, {
      "file": "DelegatingMethodAccessorImpl.java",
      "line": 43,
      "class": "sun.reflect.DelegatingMethodAccessorImpl",
      "method": "invoke"
    }, {
      "file": "Method.java",
      "line": 606,
      "class": "java.lang.reflect.Method",
      "method": "invoke"
    }, {
      "file": "RetryInvocationHandler.java",
      "line": 187,
      "class": "org.apache.hadoop.io.retry.RetryInvocationHandler",
      "method": "invokeMethod"
    }, {
      "file": "RetryInvocationHandler.java",
      "line": 102,
      "class": "org.apache.hadoop.io.retry.RetryInvocationHandler",
      "method": "invoke"
    }, {
      "file": null,
      "line": -1,
      "class": "com.sun.proxy.$Proxy19",
      "method": "getFileInfo"
    }, {
      "file": "DFSClient.java",
      "line": 2116,
      "class": "org.apache.hadoop.hdfs.DFSClient",
      "method": "getFileInfo"
    }, {
      "file": "DistributedFileSystem.java",
      "line": 1305,
      "class": "org.apache.hadoop.hdfs.DistributedFileSystem$22",
      "method": "doCall"
    }, {
      "file": "DistributedFileSystem.java",
      "line": 1301,
      "class": "org.apache.hadoop.hdfs.DistributedFileSystem$22",
      "method": "doCall"
    }, {
      "file": "FileSystemLinkResolver.java",
      "line": 81,
      "class": "org.apache.hadoop.fs.FileSystemLinkResolver",
      "method": "resolve"
    }, {
      "file": "DistributedFileSystem.java",
      "line": 1301,
      "class": "org.apache.hadoop.hdfs.DistributedFileSystem",
      "method": "getFileStatus"
    }, {
      "file": "FileSystem.java",
      "line": 1424,
      "class": "org.apache.hadoop.fs.FileSystem",
      "method": "exists"
    }, {
      "file": "HdfsToInitializer.java",
      "line": 65,
      "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer$1",
      "method": "run"
    }, {
      "file": "HdfsToInitializer.java",
      "line": 60,
      "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer$1",
      "method": "run"
    }, {
      "file": "AccessController.java",
      "line": -2,
      "class": "java.security.AccessController",
      "method": "doPrivileged"
    }, {
      "file": "Subject.java",
      "line": 415,
      "class": "javax.security.auth.Subject",
      "method": "doAs"
    }, {
      "file": "UserGroupInformation.java",
      "line": 1657,
      "class": "org.apache.hadoop.security.UserGroupInformation",
      "method": "doAs"
    }, {
      "file": "HdfsToInitializer.java",
      "line": 60,
      "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer",
      "method": "initialize"
    }, {
      "file": "HdfsToInitializer.java",
      "line": 37,
      "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer",
      "method": "initialize"
    }, {
      "file": "JobManager.java",
      "line": 520,
      "class": "org.apache.sqoop.driver.JobManager$1",
      "method": "call"
    }, {
      "file": "JobManager.java",
      "line": 517,
      "class": "org.apache.sqoop.driver.JobManager$1",
      "method": "call"
    }, {
      "file": "ClassUtils.java",
      "line": 281,
      "class": "org.apache.sqoop.utils.ClassUtils",
      "method": "executeWithClassLoader"
    }, {
      "file": "JobManager.java",
      "line": 516,
      "class": "org.apache.sqoop.driver.JobManager",
      "method": "initializeConnector"
    }, {
      "file": "JobManager.java",
      "line": 424,
      "class": "org.apache.sqoop.driver.JobManager",
      "method": "createJobRequest"
    }, {
      "file": "JobManager.java",
      "line": 317,
      "class": "org.apache.sqoop.driver.JobManager",
      "method": "start"
    }, {
      "file": "JobRequestHandler.java",
      "line": 353,
      "class": "org.apache.sqoop.handler.JobRequestHandler",
      "method": "startJob"
    }, {
      "file": "JobRequestHandler.java",
      "line": 114,
      "class": "org.apache.sqoop.handler.JobRequestHandler",
      "method": "handleEvent"
    }, {
      "file": "JobServlet.java",
      "line": 84,
      "class": "org.apache.sqoop.server.v1.JobServlet",
      "method": "handlePutRequest"
    }, {
      "file": "SqoopProtocolServlet.java",
      "line": 81,
      "class": "org.apache.sqoop.server.SqoopProtocolServlet",
      "method": "doPut"
    }, {
      "file": "HttpServlet.java",
      "line": 710,
      "class": "javax.servlet.http.HttpServlet",
      "method": "service"
    }, {
      "file": "HttpServlet.java",
      "line": 790,
      "class": "javax.servlet.http.HttpServlet",
      "method": "service"
    }, {
      "file": "ServletHolder.java",
      "line": 808,
      "class": "org.eclipse.jetty.servlet.ServletHolder",
      "method": "handle"
    }, {
      "file": "ServletHandler.java",
      "line": 1669,
      "class": "org.eclipse.jetty.servlet.ServletHandler$CachedChain",
      "method": "doFilter"
    }, {
      "file": "AuthenticationFilter.java",
      "line": 595,
      "class": "org.apache.hadoop.security.authentication.server.AuthenticationFilter",
      "method": "doFilter"
    }, {
      "file": "DelegationTokenAuthenticationFilter.java",
      "line": 291,
      "class": "org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter",
      "method": "doFilter"
    }, {
      "file": "AuthenticationFilter.java",
      "line": 554,
      "class": "org.apache.hadoop.security.authentication.server.AuthenticationFilter",
      "method": "doFilter"
    }, {
      "file": "ServletHandler.java",
      "line": 1652,
      "class": "org.eclipse.jetty.servlet.ServletHandler$CachedChain",
      "method": "doFilter"
    }, {
      "file": "ServletHandler.java",
      "line": 585,
      "class": "org.eclipse.jetty.servlet.ServletHandler",
      "method": "doHandle"
    }, {
      "file": "ContextHandler.java",
      "line": 1127,
      "class": "org.eclipse.jetty.server.handler.ContextHandler",
      "method": "doHandle"
    }, {
      "file": "ServletHandler.java",
      "line": 515,
      "class": "org.eclipse.jetty.servlet.ServletHandler",
      "method": "doScope"
    }, {
      "file": "ContextHandler.java",
      "line": 1061,
      "class": "org.eclipse.jetty.server.handler.ContextHandler",
      "method": "doScope"
    }, {
      "file": "ScopedHandler.java",
      "line": 141,
      "class": "org.eclipse.jetty.server.handler.ScopedHandler",
      "method": "handle"
    }, {
      "file": "HandlerWrapper.java",
      "line": 97,
      "class": "org.eclipse.jetty.server.handler.HandlerWrapper",
      "method": "handle"
    }, {
      "file": "Server.java",
      "line": 499,
      "class": "org.eclipse.jetty.server.Server",
      "method": "handle"
    }, {
      "file": "HttpChannel.java",
      "line": 310,
      "class": "org.eclipse.jetty.server.HttpChannel",
      "method": "handle"
    }, {
      "file": "HttpConnection.java",
      "line": 257,
      "class": "org.eclipse.jetty.server.HttpConnection",
      "method": "onFillable"
    }, {
      "file": "AbstractConnection.java",
      "line": 540,
      "class": "org.eclipse.jetty.io.AbstractConnection$2",
      "method": "run"
    }, {
      "file": "ThreadPoolExecutor.java",
      "line": 1145,
      "class": "java.util.concurrent.ThreadPoolExecutor",
      "method": "runWorker"
    }, {
      "file": "ThreadPoolExecutor.java",
      "line": 615,
      "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
      "method": "run"
    }, {"file": "Thread.java", "line": 745, "class": "java.lang.Thread", "method": "run"}],
    "class": "java.net.ConnectException"
  },
  "stack-trace": [{
    "file": "HdfsToInitializer.java",
    "line": 85,
    "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer",
    "method": "initialize"
  }, {
    "file": "HdfsToInitializer.java",
    "line": 37,
    "class": "org.apache.sqoop.connector.hdfs.HdfsToInitializer",
    "method": "initialize"
  }, {
    "file": "JobManager.java",
    "line": 520,
    "class": "org.apache.sqoop.driver.JobManager$1",
    "method": "call"
  }, {
    "file": "JobManager.java",
    "line": 517,
    "class": "org.apache.sqoop.driver.JobManager$1",
    "method": "call"
  }, {
    "file": "ClassUtils.java",
    "line": 281,
    "class": "org.apache.sqoop.utils.ClassUtils",
    "method": "executeWithClassLoader"
  }, {
    "file": "JobManager.java",
    "line": 516,
    "class": "org.apache.sqoop.driver.JobManager",
    "method": "initializeConnector"
  }, {
    "file": "JobManager.java",
    "line": 424,
    "class": "org.apache.sqoop.driver.JobManager",
    "method": "createJobRequest"
  }, {
    "file": "JobManager.java",
    "line": 317,
    "class": "org.apache.sqoop.driver.JobManager",
    "method": "start"
  }, {
    "file": "JobRequestHandler.java",
    "line": 353,
    "class": "org.apache.sqoop.handler.JobRequestHandler",
    "method": "startJob"
  }, {
    "file": "JobRequestHandler.java",
    "line": 114,
    "class": "org.apache.sqoop.handler.JobRequestHandler",
    "method": "handleEvent"
  }, {
    "file": "JobServlet.java",
    "line": 84,
    "class": "org.apache.sqoop.server.v1.JobServlet",
    "method": "handlePutRequest"
  }, {
    "file": "SqoopProtocolServlet.java",
    "line": 81,
    "class": "org.apache.sqoop.server.SqoopProtocolServlet",
    "method": "doPut"
  }, {
    "file": "HttpServlet.java",
    "line": 710,
    "class": "javax.servlet.http.HttpServlet",
    "method": "service"
  }, {
    "file": "HttpServlet.java",
    "line": 790,
    "class": "javax.servlet.http.HttpServlet",
    "method": "service"
  }, {
    "file": "ServletHolder.java",
    "line": 808,
    "class": "org.eclipse.jetty.servlet.ServletHolder",
    "method": "handle"
  }, {
    "file": "ServletHandler.java",
    "line": 1669,
    "class": "org.eclipse.jetty.servlet.ServletHandler$CachedChain",
    "method": "doFilter"
  }, {
    "file": "AuthenticationFilter.java",
    "line": 595,
    "class": "org.apache.hadoop.security.authentication.server.AuthenticationFilter",
    "method": "doFilter"
  }, {
    "file": "DelegationTokenAuthenticationFilter.java",
    "line": 291,
    "class": "org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter",
    "method": "doFilter"
  }, {
    "file": "AuthenticationFilter.java",
    "line": 554,
    "class": "org.apache.hadoop.security.authentication.server.AuthenticationFilter",
    "method": "doFilter"
  }, {
    "file": "ServletHandler.java",
    "line": 1652,
    "class": "org.eclipse.jetty.servlet.ServletHandler$CachedChain",
    "method": "doFilter"
  }, {
    "file": "ServletHandler.java",
    "line": 585,
    "class": "org.eclipse.jetty.servlet.ServletHandler",
    "method": "doHandle"
  }, {
    "file": "ContextHandler.java",
    "line": 1127,
    "class": "org.eclipse.jetty.server.handler.ContextHandler",
    "method": "doHandle"
  }, {
    "file": "ServletHandler.java",
    "line": 515,
    "class": "org.eclipse.jetty.servlet.ServletHandler",
    "method": "doScope"
  }, {
    "file": "ContextHandler.java",
    "line": 1061,
    "class": "org.eclipse.jetty.server.handler.ContextHandler",
    "method": "doScope"
  }, {
    "file": "ScopedHandler.java",
    "line": 141,
    "class": "org.eclipse.jetty.server.handler.ScopedHandler",
    "method": "handle"
  }, {
    "file": "HandlerWrapper.java",
    "line": 97,
    "class": "org.eclipse.jetty.server.handler.HandlerWrapper",
    "method": "handle"
  }, {
    "file": "Server.java",
    "line": 499,
    "class": "org.eclipse.jetty.server.Server",
    "method": "handle"
  }, {
    "file": "HttpChannel.java",
    "line": 310,
    "class": "org.eclipse.jetty.server.HttpChannel",
    "method": "handle"
  }, {
    "file": "HttpConnection.java",
    "line": 257,
    "class": "org.eclipse.jetty.server.HttpConnection",
    "method": "onFillable"
  }, {
    "file": "AbstractConnection.java",
    "line": 540,
    "class": "org.eclipse.jetty.io.AbstractConnection$2",
    "method": "run"
  }, {
    "file": "ThreadPoolExecutor.java",
    "line": 1145,
    "class": "java.util.concurrent.ThreadPoolExecutor",
    "method": "runWorker"
  }, {
    "file": "ThreadPoolExecutor.java",
    "line": 615,
    "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
    "method": "run"
  }, {"file": "Thread.java", "line": 745, "class": "java.lang.Thread", "method": "run"}],
  "error-code": "GENERIC_HDFS_CONNECTOR_0007",
  "class": "org.apache.sqoop.common.SqoopException",
  "error-code-message": "Invalid input/output directory"
}

what we write in hdfs link uri config is : hdfs://localhost:9000
what we write in job outputDirectory config is : hdfs://localhost:9000/output/data or hdfs://localhost:9000/yaoutput

so , how can we deal with the problem ？

thank you very much.

Docker pull leads to not authorized error

When I run

sudo docker pull sequenceiq/hadoop-docker:2.7.1

It downloads 5 hashes and then I get

unauthorized: authentication required

and then the pull finishes.

This person can reproduce this issue.

bootstrap daemon mode

Hi,

What's the reason for this? Can this be removed?

if [[ $1 == "-d" ]]; then
  while true; do sleep 1000; done
fi

Can't write to hdfs

The user "root" doesn't have write access to the hdfs directory "/topics".

I start up the docker image:
docker run -d -p 9000:9000 sequenceiq/hadoop-docker:2.7.1

then run kafka connector-hdfs and get this error:

[2016-09-12 17:19:28,039] INFO Couldn't start HdfsSinkConnector: (io.confluent.connect.hdfs.HdfsSinkTask:72)
org.apache.kafka.connect.errors.ConnectException: org.apache.hadoop.security.AccessControlException: Permission denied: user=fclvappi005561, access=WRITE, inode="/topics":root:supergroup:drwxr-xr-x

Hadoop sample does not work

Tried both sequnceiq/hadoop-docker and hadoop-docker ubuntu on both EC2 Ubuntu and EC2 Linux. They install fine but failed to run hadoop sample code.
error message is
15/04/15 17:33:44 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).

did not finish the job and there's not output.

Change default value of dfs.encrypt.data.transfer.cipher.suites to AES?

The Hadoop official document recommends to set dfs.encrypt.data.transfer.cipher.suites to AES.
"AES offers the greatest cryptographic strength and the best performance."
Setting it to AES/CTR/NoPadding activates AES encryption. By default, this is unspecified, so AES is not used.

I wonder if the default need to be changed for security and performance. Thanks.

Error starting MRAppMaster java.lang.UnsatisfiedLinkError:

I getting this error when submit job with yarn. Does anybody get the same error like me?

INFO (org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition:1509) - Adding job token for job_1484716821384_0001 to jobTokenSecretManager
ERROR (org.apache.hadoop.mapreduce.v2.app.MRAppMaster:1409) - Error starting MRAppMaster
java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(IILjava/nio/ByteBuffer;ILjava/nio/ByteBuffer;IILjava/lang/String;JZ)V

docker run --net=host

All services are not started

Seems that the JDK link is unavailable

hi,

I am trying to build this image from scratch. I found the JDK link was unavailable now.

Please double check.

Here is the error I got:

Step 12 : RUN rpm -i jdk-7u71-linux-x64.rpm
 ---> Running in 4a06a703e69f
error: open of <html> failed: No such file or directory
error: open of <head> failed: No such file or directory
error: open of <title>Error failed: No such file or directory
error: open of Page failed: No such file or directory
error: open of 404</title> failed: No such file or directory
error: open of <base failed: No such file or directory
error: open of href=http://download.oracle.com/index.html failed: No such file or directory
error: open of /> failed: No such file or directory
error: open of <META failed: No such file or directory
error: open of NAME=ROBOTS failed: No such file or directory
error: open of CONTENT=NOINDEX, NOFOLLOW> failed: No such file or directory
error: open of <link failed: No such file or directory
error: open of rel=stylesheet failed: No such file or directory
error: open of type=text/css failed: No such file or directory
error: open of href=/errors/us/assets/hp-styles.css failed: No such file or directory
error: open of /> failed: No such file or directory
error: open of <link failed: No such file or directory
error: open of rel=stylesheet failed: No such file or directory
error: open of type=text/css failed: No such file or directory
error: open of href=/errors/us/assets/master-master.css failed: No such file or directory
error: open of /> failed: No such file or directory
error: open of <body failed: No such file or directory
error: open of style=margin: 0px failed: No such file or directory
error: open of bgcolor= failed: No such file or directory
error: open of <div failed: No such file or directory
error: open of id=banner> failed: No such file or directory
error: open of <table failed: No such file or directory
error: open of width=100% failed: No such file or directory
error: open of border=0 failed: No such file or directory
error: open of cellspacing=0 failed: No such file or directory
error: open of cellpadding=0> failed: No such file or directory
error: open of <tr> failed: No such file or directory
error: open of <td failed: No such file or directory
error: open of rowspan=2 failed: No such file or directory
error: open of valign=middle failed: No such file or directory
error: open of nowrap><a failed: No such file or directory
error: open of href=http://www.oracle.com><img failed: No such file or directory
error: open of src=/errors/us/assets/oralogo-small.gif failed: No such file or directory
error: open of width=154 failed: No such file or directory
error: open of height=19 failed: No such file or directory
error: open of hspace=10 failed: No such file or directory
error: open of vspace=25 failed: No such file or directory
error: open of border=0 failed: No such file or directory
error: open of ></a></td> failed: No such file or directory
error: open of <td failed: No such file or directory
error: open of align=right failed: No such file or directory
error: open of valign=top failed: No such file or directory
error: open of width=70% failed: No such file or directory
error: open of nowrap failed: No such file or directory
error: open of class=padMid><div failed: No such file or directory
error: open of id=bannerMid> failed: No such file or directory
error: open of </div></td> failed: No such file or directory
error: open of <td failed: No such file or directory
error: open of width=30% failed: No such file or directory
error: open of align=left failed: No such file or directory
error: open of valign=bottom failed: No such file or directory
error: open of nowrap></td> failed: No such file or directory
error: open of </tr> failed: No such file or directory
error: open of <tr> failed: No such file or directory
error: open of <td failed: No such file or directory
error: open of align=right failed: No such file or directory
error: open of valign=bottom failed: No such file or directory
error: open of nowrap failed: No such file or directory
error: open of class=padMid></td> failed: No such file or directory
error: open of <td failed: No such file or directory
error: open of width=30% failed: No such file or directory
error: open of align=left failed: No such file or directory
error: open of valign=bottom failed: No such file or directory
error: open of nowrap></td> failed: No such file or directory
error: open of </tr> failed: No such file or directory
error: open of </table> failed: No such file or directory
error: open of </div> failed: No such file or directory
error: open of <!-- failed: No such file or directory
error: open of Header failed: No such file or directory
error: open of END failed: No such file or directory
error: open of //--> failed: No such file or directory
error: open of <table failed: No such file or directory
error: open of BORDER=0 failed: No such file or directory
error: open of CELLPADDING=0 failed: No such file or directory
error: open of CELLSPACING=0 failed: No such file or directory
error: open of WIDTH=100%> failed: No such file or directory
error: open of <tr> failed: No such file or directory
error: open of <TD failed: No such file or directory
error: open of WIDTH=10><IMG failed: No such file or directory
error: open of SRC=/errors/us/assets/spacer.gif failed: No such file or directory
error: open of BORDER=0 failed: No such file or directory
error: open of ALT= failed: No such file or directory
error: open of WIDTH=10></TD> failed: No such file or directory
error: open of <td><table failed: No such file or directory
error: open of width=100% failed: No such file or directory
error: open of border=0 failed: No such file or directory
error: open of cellspacing=0 failed: No such file or directory
error: open of cellpadding=0> failed: No such file or directory
error: open of <tr> failed: No such file or directory
error: open of <td><IMG failed: No such file or directory
error: open of SRC=/errors/us/assets/spacer.gif failed: No such file or directory
error: open of BORDER=0 failed: No such file or directory
error: open of ALT= failed: No such file or directory
error: open of height=15></td> failed: No such file or directory
error: open of </tr> failed: No such file or directory
error: open of <tr> failed: No such file or directory
error: open of <td><h1>Were sorry, the page you requested was not found.</h1></td> </tr> <tr> <td><IMG SRC="/errors/us/assets/spacer.gif" BORDER=0 ALT="" height="5"></td> </tr> <tr> <td>We have recorded this error (404) to help us fix the problem.<br> You may wish to try again using one of the tools below. </td> </tr> <tr> <td><IMG SRC="/errors/us/assets/spacer.gif" BORDER=0 ALT="" height="5"></td> </tr> </table> <table width="100%" border="0" align="center" cellpadding="2" cellspacing="2"> <tr valign="middle"> <td align="right" valign="top" width="2%"><img src="/errors/us/assets/004894.gif" width="7" height="7" vspace="3"></td> <td><a href="javascript:history.back(-1)">Back to Previous Page</a></td> </tr> <tr valign="middle"> <td align="right" valign="top"><img src="/errors/us/assets/004894.gif" width="7" height="7" vspace="3"></td> <td><a href="http://www.oracle.com/us/sitemaps/index.html </tr> <tr valign="middle"> <td align="right" valign="top"><img src="/errors/us/assets/004894.gif" width="7" height="7" vspace="3"></td> <td><a href="http://www.oracle.com/technetwork/indexes/products/index.html">Product Index</a></td> </tr> <tr valign="middle"> <td align="right" valign="top"><img src="/errors/us/assets/004894.gif" width="7" height="7" vspace="3"></td> <td><a href="http://www.oracle.com/technetwork/indexes/downloads/index.html">Software Download Index</a></td> </tr> <tr valign="middle"> <td align="right" valign="top"><img src="/errors/us/assets/004894.gif" width="7" height="7" vspace="3"></td> <td>To <strong>search</strong> for your page, try our Search function.</td> </tr> </table> <table width="100%" border="0" cellspacing="0" cellpadding="0"> <tr valign="top"> <td width="99%"><form name=searchForm onSubmit="javascript: return isNotNull(this.keyword.value)" action="http://search.oracle.com/search/search" method=get target="_top"> <table cellspacing="0" cellpadding="0" border="0"> <tr> <td width="23">&nbsp;&nbsp;</td> <td align="left"><input type="hidden" name="default" value="true"/> <input class="textA" name="keyword" size="35" maxlength="2048" value="search site" onClick="javascript: document.searchForm.keyword.value = " title="Search"/> <input type="hidden" name="start" value="1"/> </td> <td align="left">&nbsp;&nbsp;</td> <td align="left"><input type="hidden" name="nodeid" value=""/> <input type="hidden" name="fid" value=""/> <input type="hidden" name="showSimilarDoc" value="true"/> <select name="group" onChange="javascript: setFormValues(document.searchForm,document.searchForm.group.value)" class="textA" size=1> <option value="All">All</option> <option value="Oracle Technology Network" selected="selected">Technology Network</option> <option value="Oracle PartnerNetwork (Public)">PartnerNetwork (Public Only)</option> <!-- <option value="Documentation">Documentation</option> --> <option value="Video and Multimedia">Video and Multimedia</option> </select> </td> <td align="left">&nbsp;</td> <td align="left"><a onClick="javascript: return isNotNull(document.searchForm.keyword.value)" href="javascript:document.searchForm.submit()"> <img src="/errors/us/assets/search-icon-search-icon.gif" width="21" height="20" vspace="1" hspace="1" border="0" /></a></td> <td align="left">&nbsp;&nbsp;</td> <td align="left"><a href="http://search.oracle.com/search/search" class="suggestedLink">Refine Search</a></td> </tr> </table> </form></td> </tr> </table> <br> <div id="footer"> <table width="100%"  border="0" cellspacing="0" cellpadding="0"> <tr> <td width="50%" class="legalese" align="right" valign="top">&nbsp;</td> <td width="50%" class="legalese" align="left" valign="top" >&nbsp;</td> </tr> <tr> <td width="50%" class="legalese" align="left" valign="top">&nbsp;</td> <td width="50%" class="legalese" align="right" valign="top"><a class=legalese href="http://www.oracle.com/rss/index.html" target="_top">RSS <img src="/errors/us/assets/feed-icon-14x14.png" width="14" height="14" border="0"></A><span class=navlink> | </span> <a class=legalese href="http://www.oracle.com/html/copyright.html" target="_top">Legal Notices and Terms for Use</a><span class=navlink> | </span> <a class=legalese href="http://www.oracle.com/html/privacy.html" target="_top">Privacy Statement</a><br /> </td> </tr> <tr> <td width="50%" class="legalese" align="right" valign="top">&nbsp;</td> <td width="50%" class="legalese" align="left" valign="top" ></td> </tr> </table> </div> </td> <TD WIDTH="10"><IMG SRC="/errors/us/assets/spacer.gif" BORDER=0 ALT="" WIDTH="10"></TD> </tr> </table> </body> </html>  failed: File name too long

Nothing done when run bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'

I run bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+' to test hadoop works well. the order is running and accepted but get stuck and nothing happens.

Perhpas this is the problem:

Thanks in advance...

Yum Checksum Issue When Updating `libselinux`

I'm using this project to establish a test container before we rebuild it using our own foundation images. During a docker build of master I was unable to get past updating libselinux.

Rpmdb checksum is invalid: dCDPT(pkg checksums): libselinux-utils.x86_64 0:2.0.94-7.el6 - u

The command '/bin/sh -c yum update -y libselinux' returned a non-zero code: 1

YARN memory default vritual memory settings limits

If you would like to write custom YARN application, i.e. via Spring Boot, using this manual (https://spring.io/guides/gs/yarn-basic/) you will face couple of errors:

One issue about guide.
Make sure you have corresponding versions of "spring-yarn-boot", "spring-boot-starter-parent" and "spring-boot-maven-plugin" in pom-files. Simpliest solution go to maven central and use latest versions.
Otherwise you may face "cannot instantiate 'yarnClient' bean" error, which tell you almost nothing why it failed
Another about this docker image.
Make sure you set limitless virtual memory for YARN.
For example, you may set in yarn-site.xml this properties:

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>Whether virtual memory limits will be enforced for containers</description>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
    <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
  </property>

Cause you use it in Docker, you have to change it somehow after hadoop cluster initiation, or before, mapping volumes with "-v" option
Another solution is to use fork of this docker image, which has this settings by default, i.e. - https://github.com/sfedyakov/hadoop-271-cluster

hadoop-docker:2.5.0 image not running in CentOS 7 64bit guest in Windows 8.1 virtualbox host

Hello,

I am bit new to hadoop installation and setup, hence tried to use your image in my new CentOS 7 64 bit machine, but facing the problem.

Did the following steps, in CentOS 7 64bit guest in Windows 8.1 virtualbox host. It is failing in mapreduce step#13. In step#11 $jps shows that all hadoop components are also not started properly.

In virtualbox, created a CentOS7Server with Server GUI
SSH terminal, logged in as root user
$Update system $su -c 'yum update'
diabled SELinux
$sudo yum install docker
$sudo service docker start
$sudo chkconfig docker on
$docker pull sequenceiq/hadoop-docker:2.5.0
$docker images
$sudo docker run -i -t sequenceiq/hadoop-docker:2.5.0 /etc/bootstrap.sh -bash

11 $jps
bash-4.1# jps
293 ResourceManager
555 Jps

cd $HADOOP_PREFIX
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar grep input output 'dfs[a-z.]+'

14/08/23 06:06:27 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
java.net.ConnectException: Call From 29ab98bc0f2a/172.17.0.3 to 29ab98bc0f2a:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy14.delete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1726)
at org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:588)
at org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:584)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:584)
at org.apache.hadoop.examples.Grep.run(Grep.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
at org.apache.hadoop.ipc.Client.call(Client.java:1382)
... 31 more

bin/hdfs dfs -cat output/*

Github Description used in Docker Hub

This is all pretty minor, but still makes for a less than ideal read:
It seems the one making the GitHub description is not aware that it is also the one used in Docker Hub... At least it makes for a more-than-silly reading at the registry; mentioning master (This one is silly in GitHub as well, though), half the description is about building the image, and it mentions that if you want it simple, head over to the Docker Hub register... Where the user already is.

Wouldn't it make more sense to either make the GH description generic enough to not be a mess at the Docker Hub, or make specific description for the Hub.

Problems with webhdfs

So far I was not able to use webhdfs with docker version of hadoop [on Ubuntu]. Here is what I tried:

Add a text file at user/root/f.txt :

curl -i -X PUT -T f.txt "http://172.17.0.2:50070/webhdfs/v1/user/root/f.txt?op=CREATE&user.name=root&overwrite=true"

Try reading contents of the file from hdfs:

curl -i -L "http://172.17.0.2:50070/webhdfs/v1/user/root/f.txt?op=OPEN&user.name=root"

For which I get

{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File /user/root/f.txt not found."}}

I tried using 3 different python libraries for webhdfs, but none work either. All of them stop with message similar to

Max retries exceeded with url: /webhdfs/v1/example_dir/example.txt?op=CREATE&user.name=root&namenoderpcaddress=d85d3582cf58:9000&overwrite=false
Failed to establish a new connection: [Errno -2] Name or service not known

when trying to create a file or folder.
I also tried rebuilding the docker image to account for port 9000 not exposed, but that did seem to help.
Am I doing something utterly wrong? I expect this to be likely given that I am a total had00p n00b :)

Why does libhadoop.so need fixing?

If you have a minute, could you please explain these lines? Thank you.

Native libs tarball for hadoop 2.7.0 is missing

Step 34 : RUN curl -Ls http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.7.0.tar | tar -x -C /usr/local/hadoop/lib/native/
---> Running in 0f8ccfecd6ef
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Expose HDFS Port also to Guest OS

Hi,

First of all, this is awesome and just what I was looking for to run mini Hadoop locally on Mac OS X. I was trying to write a small program that accessed the HDFS on the docker container from my mac. I think the port 9000 ought to be exposed, for this to happen?

This is just a one line change in the dockerfile , so let me know if this makes sense or is useful, I can send a patch.

Hadoop components

It could be useful if this includes HBase, Hive, Flume, Oozie, Pig, Spark, Tez, and ZooKeeper installations.

Thanks,
Varma

Can this image be used to build a cluster?

There seems to be hint in the Readme as to how you would run many instances of this image to build an actual hadoop cluster? or is this not suitable for that?

when running example, "Not a valid JAR " exception occurred

the image version

sequenceiq/hadoop-docker:2.5.0

the exceptions

bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'
Not a valid JAR: /usr/local/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar

[WARN] Unable to load native-hadoop library for your platform

Even though Dockerfile contains "fix" for native library warning, I am still able to see the warning when running out of the box

_WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [2d6335dea9d7]
_
Host: 3.10.0-229.20.1.el7.x86_64 CentOS Linux release 7.2.1511 (Core)

bash-4.1# ldd /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/hadoop/lib/native/libhadoop.so.1.0.0)
linux-vdso.so.1 => (0x00007ffd058ab000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fdbce779000)
libjvm.so => not found
libc.so.6 => /lib64/libc.so.6 (0x00007fdbce3e4000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdbceba3000)

bash-4.1# file /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

WebHdfsFileSystem source version 2.7.1 mismtach ??

I'm getting the following in the 2.7.1 version
java.lang.NullPointerException: while trying to invoke the method org.apache.hadoop.fs.FileSystem$Statistics.incrementWriteOps(int) of a null object loaded from field org.apache.hadoop.hdfs.web.WebHdfsFileSystem.statistics of an object loaded from local variable 'this'
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.mkdirs(WebHdfsFileSystem.java:1046)

The point is that line 1046 according to http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-hdfs/2.7.1/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java?av=f

is "final HttpOpParam.Op op = PutOpParam.Op.REMOVEDEFAULTACL;"
This line can't throw the NPE

I'm using the following container "docker pull sequenceiq/hadoop-docker:2.7.1"

Can anyone shed a light on this source mismatch ?

Example can't connect to ResourceManager

Running the example steps from README.md, when I get to the MR example I get the following output:

14/03/19 19:31:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/03/19 19:31:49 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

The last message repeats over and over endlessly. (No files end up in HDFS output/ folder either)

My datanode process is not running, stoping and starting hdfs doesn't launch it.

sbin/stop-dfs.sh 
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: no datanode to stop
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode

Unable to install wget: Rpmdb checksum is invalid: dCDPT(pkg checksums): wget...

Trying to install wget:

RUN yum -y install wget \
    && yum clean all

Getting:

...
Dependencies Resolved

================================================================================
 Package         Arch              Version                Repository       Size
================================================================================
Installing:
 wget            x86_64            1.12-10.el6            base            484 k

Transaction Summary
================================================================================
Install       1 Package(s)

Total download size: 484 k
Installed size: 1.8 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : wget-1.12-10.el6.x86_64                                      1/1

Rpmdb checksum is invalid: dCDPT(pkg checksums): wget.x86_64 0:1.12-10.el6 - u

ERROR: Service 'lr-hadoop' failed to build: The command '/bin/sh -c yum -y install wget     && yum clean all' returned a non-zero code: 1

I also have tried many different variation of installation commands, no avail.

Port Forwarding Hadoop Job Monitor URL

Hi, I'm using hadoop inside the docker container running on debian. When I run my mapreduce job I get a url where I can monitor the progress.

Something like http://20a4f5aa3c45:8088/proxy/application_1490980395017_0013/

I would like to forward a port on my server to the url above so I can monitor the progress on my server. I've look around for information on this and could not find anything. Is this possible? If so could you provide some information on how it can be accomplished.

Thanks

Data volumes for persistence and connect to Hive

I'm new to the Hadoop stack so forgive me if I'm missing something obvious.

I had two requirements I'm trying to work out with this Docker image.

how to persist hdfs to a data volume (is hdfs running?)
how to connect another container running another part of the Hadoop stack i.e Hive.

Can anyone help?

Site unreachable with Hadoop on Docker

I try to use Hadoop with Docker Toolbox on Windows 10 Family. So I followed this setup : https://linoxide.com/cluster/setup-single-node-hadoop-cluster-docker/

Download the image --> OK
Run the container --> OK

$ docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash/
Starting sshd:                                             [  OK  ]
18/11/16 10:38:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [c13feb9767ac]
c13feb9767ac: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-c13feb9767ac.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root- datanode-c13feb9767ac.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-c13feb9767ac.out
18/11/16 10:40:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-c13feb9767ac.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-c13feb9767ac.out

Run jps command --> OK

bash-4.1# jps
119 NameNode
535 ResourceManager
200 DataNode
656 Jps
319 SecondaryNameNode
620 NodeManager

Get IP adress --> OK

bash-4.1# ifconfig
...
inet addr:172.17.0.2
...

Go to browser (Google Chrome v70) --> KO
Address used : 172.17.0.2:50070

Error :

This site is unreachable 172.17.0.2 took too long to answer.

Can you help me to fix it please? If you need more informations ask me.

Thanks.

Get error when run bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'

After building the docker images,when I run the command in the container,got error:
Not a valid JAR: /usr/local/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar And how can I solve this issue

Failed to run on mesos/marathon

I installed on Ubuntu 14.04 LTS， everything works fine. I can run the test job. But when I try to launch the docker from Marathon, it failed. Can anyone give me a hint as where to look? I can run other docker app in the same environment.

Unable to connect

$ docker -v
Docker version 1.2.0, build fa7b24f/1.2.0

docker run -i -t sequenceiq/hadoop-docker:2.5.1 /etc/bootstrap.sh -bash
/
Starting sshd:                                             [  OK  ]
Starting namenodes on [cbd10023ac94]
Starting secondary namenodes [0.0.0.0]
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-cbd10023ac94.out



bash-4.1# cd $HADOOP_PREFIX
bash-4.1# # run the mapreduce
bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'
14/10/08 11:51:18 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
java.net.ConnectException: Call From cbd10023ac94/172.17.0.5 to cbd10023ac94:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
    at org.apache.hadoop.ipc.Client.call(Client.java:1415)
    at org.apache.hadoop.ipc.Client.call(Client.java:1364)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
    at com.sun.proxy.$Proxy14.delete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy14.delete(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)
    at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1726)
    at org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:588)
    at org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:584)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:584)
    at org.apache.hadoop.examples.Grep.run(Grep.java:95)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.examples.Grep.main(Grep.java:101)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
    at org.apache.hadoop.ipc.Client.call(Client.java:1382)
    ... 31 more

Hadoop 2.7.1 Link is Unavailable

I noticed the Hadoop 2.7.1 download link is broken, the closest version is 2.7.4. I made this change to my Dockerfile:

-RUN curl -s http://www.eu.apache.org/dist/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz | tar -xz -C /usr/local/
-RUN cd /usr/local && ln -s ./hadoop-2.7.1 hadoop
+RUN curl -s http://www.eu.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz | tar -xz -C /usr/local/
+RUN cd /usr/local && ln -s ./hadoop-2.7.4 hadoop

After this change I see a Java mismatch error during step 36/58.

Error in brief:

17/10/22 19:11:21 ERROR namenode.NameNode: Failed to start namenode.
java.lang.UnsupportedClassVersionError: org/apache/hadoop/mapreduce/lib/output/SequenceFileAsBinaryOutputFormat : Unsupported major.minor version 52.0

And full output of step with error:

Step 36/58 : RUN $HADOOP_PREFIX/bin/hdfs namenode -format
---> Running in dd7b5a9c08d5
17/10/22 19:11:20 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = fb16285d8175/172.17.0.2
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.4
STARTUP_MSG: classpath = /usr/local/hadoop/etc/hadoop/:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.54.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.4-tests.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.7.4.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.4.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.4.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.4-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.4-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.4.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://[email protected]/repos/asf/hadoop.git -r cd915e1e8d9d0131462a0b7301586c175728a282; compiled by 'kshvachk' on 2017-08-01T00:29Z
STARTUP_MSG: java = 1.7.0_71
/
17/10/22 19:11:20 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/10/22 19:11:20 INFO namenode.NameNode: createNameNode [-format]
17/10/22 19:11:21 ERROR namenode.NameNode: Failed to start namenode.
java.lang.UnsupportedClassVersionError: org/apache/hadoop/mapreduce/lib/output/SequenceFileAsBinaryOutputFormat : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:54)
at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:42)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:119)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.security.Groups.(Groups.java:79)
at org.apache.hadoop.security.Groups.(Groups.java:74)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:285)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:339)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:333)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:968)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1441)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)
17/10/22 19:11:21 INFO util.ExitUtil: Exiting with status 1
17/10/22 19:11:21 INFO namenode.NameNode: SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down NameNode at fb16285d8175/172.17.0.2
************************************************************/
The command '/bin/sh -c $HADOOP_PREFIX/bin/hdfs namenode -format' returned a non-zero code: 1

cannot type chinese/code in hive cli

My table able to display chinese/unicode, but typing chinese in cli will become "???"

Also, I tried to input from file, the query return nothing if chinese is in the condition:

container is not running hadoop with localhost 8020 as default namenode url

Hi, I am running 2.6.0 image on my mac which run docker natively. so NO boot2Docker or VM is required.

I started my container with the following command

docker run -p 50070:50070 -p 8020:8020 -d --name hadoop-2.6.0 sequenceiq/hadoop-docker:2.6.0

My Java Application (Spring) is trying to connect to hdfs://localhost:8020 and is failing.
but my container is running and telnet on port 8020 works too.

I sshed into container and got the default url

./hdfs getconf -confKey fs.default.name
hdfs://cd710ece3b36:9000

which now points to container id with different port.
I think that is causing the problem connecting with my spring application.
Is this a bug or do I have to do something to fix this?

Thanks

PATH setting in Dockerfile is wrong

The PATH setting will not not have the correct java in the path -
need to replace
ENV PATH $PATH:$JAVA_HOME/bin
with
ENV PATH $JAVA_HOME/bin:$PATH

[Broken link] Bintray repo

README.md
404 shows up on trying to reach out to page https://bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64bit/2.7.1/view/files

Image published to the public Docker registry is outdated

While there were some recent commits the image was last pushed to the public Docker registry 15 months ago. I would expect sequenceiq/hadoop-docker:latest and 2.7.1 to be in sync with master.

$ docker images sequenceiq/hadoop-docker
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
sequenceiq/hadoop-docker   2.7.1               42efa33d1fa3        15 months ago       1.76 GB
sequenceiq/hadoop-docker   latest              5c3cc170c6bc        20 months ago       1.77 GB

Workaround: build locally.

docker build  -t sequenceiq/hadoop-docker https://github.com/sequenceiq/hadoop-docker.git

2.7.1 tag not found in repository docker.io/sequenceiq/hadoop-docker

Hey,
I'm trying to pull Hadoop 2.7.1 image without success (with - docker pull sequenceiq/hadoop-docker:2.7.1)
I'm using docker toolbox on macOS and already made it to pull and run 2.7.0 image. is 2.7.1 is the right tag? what am I missing?

Thanks in advance,
Tal