I made some tests with the AsyncClient. Unfortunately there are several problems. I don't have the feeling that this is really ready for production :(
So far I only tested a client.get with a RecordListener.
At first I figured out that all the exceptions in the methods onSuccess and onFailure of a RecordListener will we swallowed. When you want to change it, you should print the exception in
com.aerospike.client.async.SelectorManager line 100.
I set an expiration of -1 but the record shows me an expiration of 0 for the same value. This is a bit inconsistent.
In my tests I closed the client before the RecordListener could return any results of a client.get() call. With aerospike-client-3.0.35.jar this will lead to a deadlock in 100 % of all cases. The java program won't terminate.
With the latest source code a deadlock only occurs in some cases. But this must be fixed. Your client must always shut down gracefully! In my test cases I never used a local Aerospike server. This means there was always a high network latency.
Hopefully this stack will help you:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.75-b04 mixed mode):
"RMI TCP Connection(2)-" daemon prio=6 tid=0x000000000a70f000 nid=0x1770 runnable [0x000000000ba7e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <0x00000000eb333e68> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- <0x00000000eb042fe8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"JMX server connection timeout 15" daemon prio=6 tid=0x000000000a5c0800 nid=0x16c8 in Object.wait() [0x000000000b96f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000eb18d518> (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
- locked <0x00000000eb18d518> (a [I)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" daemon prio=6 tid=0x00000000095d0000 nid=0x7f4 waiting on condition [0x000000000b73e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eaf9a980> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Connection(1)-" daemon prio=6 tid=0x00000000095cf800 nid=0x1648 runnable [0x000000000b54e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <0x00000000eb0e47b0> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- <0x00000000eb03fb00> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Accept-0" daemon prio=6 tid=0x00000000098c8000 nid=0x1618 runnable [0x000000000b2ce000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0x00000000eaf9e748> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"selector0" daemon prio=6 tid=0x000000000989d800 nid=0xc10 waiting for monitor entry [0x000000000a0af000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.WindowsSelectorImpl.putEventOps(Unknown Source)
- waiting to lock <0x00000000eab790d8> (a java.lang.Object)
at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(Unknown Source)
at sun.nio.ch.SelectionKeyImpl.nioInterestOps(Unknown Source)
at sun.nio.ch.SelectionKeyImpl.interestOps(Unknown Source)
at sun.nio.ch.SelectorImpl.register(Unknown Source)
at java.nio.channels.spi.AbstractSelectableChannel.register(Unknown Source)
- locked <0x00000000eacf6c00> (a java.lang.Object)
- locked <0x00000000eacf6c10> (a java.lang.Object)
at com.aerospike.client.async.AsyncConnection.register(AsyncConnection.java:86)
at com.aerospike.client.async.SelectorManager.registerCommands(SelectorManager.java:129)
at com.aerospike.client.async.SelectorManager.runCommands(SelectorManager.java:85)
at com.aerospike.client.async.SelectorManager.run(SelectorManager.java:69)
Locked ownable synchronizers:
- None
"Service Thread" daemon prio=6 tid=0x0000000007df0000 nid=0x106c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" daemon prio=10 tid=0x0000000007ded000 nid=0xa04 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" daemon prio=10 tid=0x0000000007dde800 nid=0x1488 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Attach Listener" daemon prio=10 tid=0x0000000007ddd000 nid=0x1624 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x0000000007ddb800 nid=0x10e8 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=8 tid=0x0000000007d89800 nid=0x7d8 in Object.wait() [0x0000000008c3f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000eaa84858> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x00000000eaa84858> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x0000000001d2c000 nid=0x1540 in Object.wait() [0x000000000890f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000eaa84470> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
- locked <0x00000000eaa84470> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"main" prio=6 tid=0x0000000001d5d800 nid=0x1420 waiting for monitor entry [0x000000000208f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(Unknown Source)
- waiting to lock <0x00000000eacf6c00> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.implClose(Unknown Source)
- locked <0x00000000eab790d8> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.implCloseSelector(Unknown Source)
- locked <0x00000000eab78e58> (a sun.nio.ch.Util$2)
- locked <0x00000000eab78df0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eab78bd8> (a sun.nio.ch.WindowsSelectorImpl)
at java.nio.channels.spi.AbstractSelector.close(Unknown Source)
at com.aerospike.client.async.SelectorManager.close(SelectorManager.java:201)
at com.aerospike.client.async.SelectorManagers.close(SelectorManagers.java:65)
at com.aerospike.client.async.AsyncCluster.close(AsyncCluster.java:86)
at com.aerospike.client.AerospikeClient.close(AerospikeClient.java:233)
at Test.main(Test.java:130)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x0000000007d82000 nid=0x1734 runnable
"GC task thread#0 (ParallelGC)" prio=6 tid=0x0000000001c64000 nid=0x1144 runnable
"GC task thread#1 (ParallelGC)" prio=6 tid=0x0000000001c66000 nid=0x15d0 runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000007df9800 nid=0x15d8 waiting on condition
JNI global references: 167
Found one Java-level deadlock:
"selector0":
waiting to lock monitor 0x0000000007d896d8 (object 0x00000000eab790d8, a java.lang.Object),
which is held by "main"
"main":
waiting to lock monitor 0x0000000007d88028 (object 0x00000000eacf6c00, a java.lang.Object),
which is held by "selector0"
Java stack information for the threads listed above:
"selector0":
at sun.nio.ch.WindowsSelectorImpl.putEventOps(Unknown Source)
- waiting to lock <0x00000000eab790d8> (a java.lang.Object)
at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(Unknown Source)
at sun.nio.ch.SelectionKeyImpl.nioInterestOps(Unknown Source)
at sun.nio.ch.SelectionKeyImpl.interestOps(Unknown Source)
at sun.nio.ch.SelectorImpl.register(Unknown Source)
at java.nio.channels.spi.AbstractSelectableChannel.register(Unknown Source)
- locked <0x00000000eacf6c00> (a java.lang.Object)
- locked <0x00000000eacf6c10> (a java.lang.Object)
at com.aerospike.client.async.AsyncConnection.register(AsyncConnection.java:86)
at com.aerospike.client.async.SelectorManager.registerCommands(SelectorManager.java:129)
at com.aerospike.client.async.SelectorManager.runCommands(SelectorManager.java:85)
at com.aerospike.client.async.SelectorManager.run(SelectorManager.java:69)
"main":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(Unknown Source)
- waiting to lock <0x00000000eacf6c00> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.implClose(Unknown Source)
- locked <0x00000000eab790d8> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.implCloseSelector(Unknown Source)
- locked <0x00000000eab78e58> (a sun.nio.ch.Util$2)
- locked <0x00000000eab78df0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eab78bd8> (a sun.nio.ch.WindowsSelectorImpl)
at java.nio.channels.spi.AbstractSelector.close(Unknown Source)
at com.aerospike.client.async.SelectorManager.close(SelectorManager.java:201)
at com.aerospike.client.async.SelectorManagers.close(SelectorManagers.java:65)
at com.aerospike.client.async.AsyncCluster.close(AsyncCluster.java:86)
at com.aerospike.client.AerospikeClient.close(AerospikeClient.java:233)
at Test.main(Test.java:130)
Found 1 deadlock.