brockn / hdfs-nfs-proxy Goto Github PK

Link to the new repo (https://github.com/cloudera/hdfs-nfs-proxy) is not found, returns status 404, please update it, since this current repo is still being linked at here.

this is a great software. but there i surface some problem,

1, it seemed not support append feature.
2, the user UID mapping is a magical thing, can let user customize the mapping thing?
eg, i have run hdfs-nfs-proxy with hadoop user,,, but on another host, i need use root to mount the nfs export, and hadoop authorize system not let me mount because,, mount.nfs send root UID/GID to hdfs-nfs-proxy, and hdfs-nfs-proxy use system /etc/passwd /etc/group to find the user name ,,,,, forward this to hadoop , and not match with hadoop user.

hadoop version 1.0.0

Copying files to hadoop via the nfs proxy using 'cp' or 'mv' can result in write deadlocks.

I'm finding that I can fairly easily lock the NFS server by running a few parallel write operations:

cd /hdfs/mountpoint/tmp
for x in $(seq 1 5); do ( dd if=/dev/urandom of=foo$x.test bs=1048576 count=500 & ); done

The Pending Write Offsets list never changes
kill -3 provides no output

12/04/23 17:56:30 INFO rpc.RPCServer: class com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler got client 192.168.1.10:944
12/04/23 17:56:30 INFO rpc.ClientWorker: Got SecurityHandler of type com.cloudera.hadoop.hdfs.nfs.security.SecurityHandler
12/04/23 17:56:30 WARN handlers.OperationRequestHandler: 0x7ffffff0 Error for client /192.168.1.10 and OPENResponse
12/04/23 17:56:30 WARN nfs4.NFS4Handler: 0x7ffffff0 Quitting due to 1 on OPENRequest for nickb
12/04/23 17:56:49 INFO handlers.OPENHandler: 0x7ffffff0 Opened /tmp/foo2.test for write org.apache.hadoop.fs.FSDataOutputStream@50703755
12/04/23 17:56:49 INFO handlers.OPENHandler: 0x7ffffff0 Opened /tmp/foo3.test for write org.apache.hadoop.fs.FSDataOutputStream@4dcac775
12/04/23 17:56:49 INFO handlers.OPENHandler: 0x7ffffff0 Opened /tmp/foo4.test for write org.apache.hadoop.fs.FSDataOutputStream@746cd30c
12/04/23 17:56:49 INFO handlers.OPENHandler: 0x7ffffff0 Opened /tmp/foo5.test for write org.apache.hadoop.fs.FSDataOutputStream@3dc81f66
12/04/23 17:57:08 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@3dc81f66 and 34799616
12/04/23 17:57:08 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@50703755 and 38600704
12/04/23 17:57:08 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 34603008
12/04/23 17:57:08 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@4dcac775 and 49414144
12/04/23 17:57:09 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@3dc81f66 and 105979904
12/04/23 17:57:09 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@50703755 and 133173248
12/04/23 17:57:09 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 131076096
12/04/23 17:57:09 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@3dc81f66 and 106012672
12/04/23 17:57:09 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@50703755 and 133206016
12/04/23 17:57:09 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 131108864

12/04/23 17:57:18 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 34603008
12/04/23 17:57:18 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@3dc81f66 and 34799616
12/04/23 17:57:18 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@50703755 and 38600704
12/04/23 17:57:18 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@4dcac775 and 85655552
12/04/23 17:57:24 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 161632256
12/04/23 17:57:24 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@50703755 and 38600704
12/04/23 17:57:24 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@3dc81f66 and 90570752
12/04/23 17:57:24 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@4dcac775 and 103546880
12/04/23 17:57:24 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 34603008
12/04/23 17:57:24 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 161636352
12/04/23 17:57:24 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@746cd30c and 161640448
12/04/23 17:57:26 INFO nfs4.WriteOrderHandler: Sync for org.apache.hadoop.fs.FSDataOutputStream@50703755 and 163704832
12/04/23 17:57:36 INFO nfs4.WriteOrderHandler: Pending Write Offsets: [38928384, 38993920, 39059456, 39124992, 39190528, 39256064, 39321600, 39387
136, 39452672, 39518208, 39583744, 39649280, 39714816, 39780352, 39845888, 39911424, 39976960, 40042496, 40108032, 40173568, 40239104, 40304640, 4
0370176, 40435712, 40501248, 40566784, 40632320, 40697856, 40763392, 40828928, 40894464, 40960000, 41025536, 41091072, 41156608, 41222144, 4128768
0, 41353216, 41418752, 41484288, 41549824, 41615360, 41680896, 41746432, 41811968, 41877504, 41943040, 42008576, 42074112, 42139648, 42205184, 422
70720, 42336256, 42401792, 42467328, 42532864, 42598400, 42663936, 42729472, 42795008, 42860544, 42926080, 42991616, 43057152, 43122688, 43188224,
43253760, 43319296, 43384832, 43450368, 43515904, 43581440, 43646976, 43712512, 43778048, 43843584, 43909120, 43974656, 44040192, 44105728, 44171
264, 44236800, 44302336, 44367872, 44433408, 44498944, 44564480, 44630016, 44695552, 44761088, 44826624, 44892160, 44957696, 45023232, 45088768, 4
... and lots more

Followed by many of these:

12/04/23 18:00:12 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1845449483
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1828672267
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1811895051
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1795117835
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1778340619
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1761563403
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1744786187
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1728008971
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1711231755
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1694454539
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1677677323
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1660900107
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1644122891
12/04/23 18:00:13 INFO rpc.ClientWorker: 0x7fffffe6 ignoring request 1627345675

Using the proxy with older hadoops will cause many thread to be in the following state:

"pool-1-thread-7" prio=10 tid=0x000000004fd8a000 nid=0x6b0 in Object.wait() [0x0000000042598000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.ipc.Client.call(Client.java:1052)
- locked <0x00000000f5fe7120> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:203)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:193)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:184)
at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.FSInfo.getUsed(FSInfo.java:61)
at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.SpaceUsedHandler.get(SpaceUsedHandler.java:38)
at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.SpaceUsedHandler.get(SpaceUsedHandler.java:1)
at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.Attribute.getAttrs(Attribute.java:175)
at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.GETATTRHandler.doHandle(GETATTRHandler.java:54)
at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.GETATTRHandler.doHandle(GETATTRHandler.java:1)
at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.OperationRequestHandler.handle(OperationRequestHandler.java:57)
at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler$1.run(NFS4Handler.java:148)
at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler$1.run(NFS4Handler.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler.process(NFS4Handler.java:136)
at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler.process(NFS4Handler.java:1)
at com.cloudera.hadoop.hdfs.nfs.rpc.ClientWorker$ClientTask.run(ClientWorker.java:379)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I see both log4j and slf4j depends in the pom.xml : any preference for one over the other?
I can make updates and standardise on one (suggest log4j). Thoughts?

I have the same issue like #5 2)

I can confirm this with the latest git version and Hadoop 1.0.4 (not CDH, used -Phadoop1) on Ubuntu 12.04:

The mount fails with:

 mount.nfs4: Remote I/O error

some Details from the error log:

12/11/30 00:32:17 INFO state.HDFSState: Writing temp files to [/tmp/hdfs-nfs-proxy/tmp]
12/11/30 00:32:17 INFO rpc.RPCServer: class com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler created server ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=2051] on 2051
12/11/30 00:32:51 INFO rpc.RPCServer: class com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler got client localhost:862
12/11/30 00:32:51 ERROR nfs.NetUtils: Unable to find the domain the server is running on. Please report.
canonicalHostName = localhost
hostname = localhost
isLoopback = true
hostAdddress = 127.0.0.1
12/11/30 00:32:51 ERROR nfs.NetUtils: Unable to find the domain the server is running on. Please report.
canonicalHostName = localhost
hostname = localhost
isLoopback = true
hostAdddress = 127.0.0.1
12/11/30 00:32:51 WARN handlers.OperationRequestHandler: 0x7ffffffa Error for client localhost/127.0.0.1 and GETATTRResponse
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.FSInfo.getObject(FSInfo.java:83)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.FSInfo.getUsed(FSInfo.java:129)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.SpaceUsedHandler.get(SpaceUsedHandler.java:37)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.SpaceUsedHandler.get(SpaceUsedHandler.java:31)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.Attribute.getAttrs(Attribute.java:122)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.GETATTRHandler.doHandle(GETATTRHandler.java:60)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.GETATTRHandler.doHandle(GETATTRHandler.java:41)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.OperationRequestHandler.handle(OperationRequestHandler.java:81)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture.doMakeProgress(NFS4AsyncFuture.java:78)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture$1.run(NFS4AsyncFuture.java:98)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture$1.run(NFS4AsyncFuture.java:95)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture.makeProgress(NFS4AsyncFuture.java:95)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.AsyncTaskExecutor$TaskRunnable.run(AsyncTaskExecutor.java:103)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.FSInfo.getObject(FSInfo.java:81)
    ... 21 more
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Access denied for user root. Superuser privilege is required
    at org.apache.hadoop.hdfs.server.namenode.PermissionChecker.checkSuperuserPrivilege(PermissionChecker.java:69)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkSuperuserPrivilege(FSNamesystem.java:5198)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:4063)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:853)
    at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy1.getStats(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy1.getStats(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.totalRawUsed(DFSClient.java:1081)
    ... 26 more
12/11/30 00:32:51 WARN nfs4.NFS4AsyncFuture: 0x7ffffffa Quitting due to 10006 on GETATTRRequest for root
12/11/30 00:32:51 INFO rpc.ClientInputHandler: 0x7ffffffa fe757077 Writing CompoundResponse to localhost:862
12/11/30 00:32:51 INFO rpc.ClientInputHandler: 0x7ffffffa Shutdown worker for client localhost:862
12/11/30 00:32:51 INFO rpc.RPCServer: class com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler got client <host omitted>:906
12/11/30 00:32:51 WARN handlers.OperationRequestHandler: 0x7ffffff5 Error for client <hostname omitted> and GETATTRResponse
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.FSInfo.getObject(FSInfo.java:83)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.FSInfo.getUsed(FSInfo.java:129)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.SpaceUsedHandler.get(SpaceUsedHandler.java:37)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.SpaceUsedHandler.get(SpaceUsedHandler.java:31)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.Attribute.getAttrs(Attribute.java:122)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.GETATTRHandler.doHandle(GETATTRHandler.java:60)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.GETATTRHandler.doHandle(GETATTRHandler.java:41)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.handlers.OperationRequestHandler.handle(OperationRequestHandler.java:81)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture.doMakeProgress(NFS4AsyncFuture.java:78)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture$1.run(NFS4AsyncFuture.java:98)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture$1.run(NFS4AsyncFuture.java:95)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4AsyncFuture.makeProgress(NFS4AsyncFuture.java:95)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.AsyncTaskExecutor$TaskRunnable.run(AsyncTaskExecutor.java:103)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.cloudera.hadoop.hdfs.nfs.nfs4.attrs.FSInfo.getObject(FSInfo.java:81)
    ... 21 more
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Access denied for user root. Superuser privilege is required
    at org.apache.hadoop.hdfs.server.namenode.PermissionChecker.checkSuperuserPrivilege(PermissionChecker.java:69)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkSuperuserPrivilege(FSNamesystem.java:5198)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:4063)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:853)
    at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy1.getStats(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy1.getStats(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.totalRawUsed(DFSClient.java:1081)
    ... 26 more
12/11/30 00:32:51 WARN nfs4.NFS4AsyncFuture: 0x7ffffff5 Quitting due to 10006 on GETATTRRequest for root
12/11/30 00:32:51 INFO rpc.ClientInputHandler: 0x7ffffff5 234bb914 Writing CompoundResponse to <host omitted>:906
12/11/30 00:32:51 INFO rpc.ClientInputHandler: 0x7ffffff5 Shutdown worker for client <hostname omitted>:906

I've tested it locally with user hadoop and my local user. I've also tested it between different hosts.

core-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/tmp</value>
  </property>
  <property>
    <name>fs.checkpoint.dir</name>
    <value>/mnt/storage1/hadoop/tmp/dfs/namesecondary,/mnt/storage2/hadoop/tmp/dfs/namesecondary,/mnt/storage3/hadoop/tmp/dfs/namesecondary</value>
  </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://<host omitted>:8020</value>
  </property>
  <property>
    <name>fs.trash.interval</name>
    <value>1440</value>
  </property>
</configuration>

hdfs-nfs-site.xml:

$ cat hdfs-nfs-site.xml 
<?xml version="1.0" encoding="UTF-8"?>

<configuration>
  <property>
    <name>hdfs.nfs.data.dir</name>
    <value>/tmp/hdfs-nfs-proxy/data</value>
    <!-- Persistent location for nfs proxy to write important data -->
  </property>
  <property>
    <name>hdfs.nfs.temp.dirs</name>
    <value>/tmp/hdfs-nfs-proxy/tmp</value>
    <!-- CSV of temp directories to write pending writes -->
  </property>
  <property>
    <name>hdfs.nfs.security.flavor</name>
    <value>unix</value>
    <!-- Security flavor, if set to kerberos (krb5p) additional properties need to be configured -->
  </property>
  <property>
    <name>hdfs.nfs.security.allowed.hosts</name>
    <value>
      localhost.localdomain rw
      * ro
    </value>
    <!--
        This is a new line seperated list of any of the following:
        1) * - which exports filesystem to all hosts
        2) hostnames or ip addresses
        3) CIDR such as 192.168.0.0/255.255.252.0 or 192.168.0.0/22
        4) Java Regex match host or ip such as: 192.168.0.[0-9]+ or [a-z]+.b.com

        followed by either ro for read-only access or rw for read write access
    -->
  </property>
</configuration>

I don't have time to dig further into the code, any hints how to resolve this are welcome. I don't have much experience with either NFSv4 nor Hadoop HDFS.

12/08/16 06:40:31 INFO rpc.RPCServer: class com.cloudera.hadoop.hdfs.nfs.nfs4.NFS4Handler got client pc-hype2windev:50692
12/08/16 06:40:31 INFO rpc.ClientWorker: Got SecurityHandler of type com.cloudera.hadoop.hdfs.nfs.security.SecurityHandler
Exception in thread "RPCServer-pc-hype2windev50692" java.lang.OutOfMemoryError: Java heap space

From
C:\Users\user>mount -o fileaccess=7 \192.168.25.51\ X:

On hadoop 100 jar.

per the RFC

Servers which provide volatile filehandles that may expire while open
(i.e., if FH4_VOL_MIGRATION or FH4_VOL_RENAME is set or if
FH4_VOLATILE_ANY is set and FH4_NOEXPIRE_WITH_OPEN not set), should
deny a RENAME or REMOVE that would affect an OPEN file of any of the
components leading to the OPEN file. In addition, the server should
deny all RENAME or REMOVE requests during the grace period upon
server restart.

On a 1000Mbit network, I timed a copy of a 1.5GB file to local disk at 45 minutes. The main reason I can find for this is that NFS read/write requests are limited to 64Kbytes, which translates to much slower Hadoop requests of 64Kbytes.

It may be a large performance win if the nfs-proxy answered all read-requests from a local disk based cache. Have the cache populated by nfs-proxy upon receiving a read-request whereby it grabs 64MBytes or 128MBytes from HDFS in one operation, then answers NFS requests from the data stored on local disk. I think this could work well enough even if initially restricted to files opened read-only.