netflix / ndbench Goto Github PK
View Code? Open in Web Editor NEWNetflix Data Store Benchmark
License: Apache License 2.0
Netflix Data Store Benchmark
License: Apache License 2.0
Implement CLI argument lists for each of the following plugins:
It will a good starting point to have docker image for NdBench so that it is easy to get it up and running in any cloud environment
hello
I have limited exposure in data store systems and cloud.
I am planning on evaluating ndbench for a particular usecase. i need help setting ndbench up and get going.
I have following questions to begin with
1. What is minimum count of serverboxes i need to start ? do you have minimum HW configuration requirements?
2. What SW/stack framework do i need to set up? i.e. specific hadoop or cassandra version or other framework?
3. how to install ndbench on my setup, integrate with framework (hadoop/cassandra/etc) and run the benchmark?
4. what kind if score or result ndbech generate and how to interpret it ?
thanks
Ankit
Depends on #116
README.md says "Details about the features can be found in the Wiki" but that URL simply redirects back to https://github.com/Netflix/ndbench/ and doesn't display a Wiki. Clicking the Wiki link in Github (immediately below the Netflix/ndbench line in the Github UI) has the same effect.
currently this uses a System.out.println
For some odd reason the throughput for the backfill function is much lower then for writes. I'm using the same concurrency.
Test was conducted with the following setup:
application.properties has following:
ndbench.config.numKeys=250000000
ndbench.config.writeRateLimit=400000
ndbench.config.readRateLimit=500000
ndbench.config.numValues=50000
ndbench.config.dataSize=1024
ndbench.config.numWriters=1000
ndbench.config.numReaders=1000
ndbench.config.backfillStartKey=1
ndbench.config.numBackfill=1000
ndbench.config.cass.host=172.31.31.228
ndbench.config.cass.keyspace=ndbench
ndbench.config.cass.cfname=scylla
ndbench.config.cass.cluster=NDBench
ndbench.config.cass.colsPerRow=6
ndbench.config.cass.writeConsistencyLevel=ONE
ndbench.config.cass.readConsistencyLevel=ONE
ndbench.config.cass.connections=32
I noticed that the backfill function is very verbose printing each key that is inserted to stdout or (in this case) to the catalina.out file while the writeSingle function just prints the summary with WPS. I wonder if this is the reason why the backfill is able to reach just 40k wps thorughput while writes are over 100k wps.
"Details about the features can be found in the Wiki"
The "Wiki" text, when clicked, doesn't do anything but reload the page.
When shutting down the ndbench application we sometimes get a stack trace that looks like this:
Nov 27, 2017 3:10:18 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Nov 27, 2017 3:10:18 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-8080"]
Nov 27, 2017 3:10:18 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-nio-8080"]
This hangs things up when doing things like integration tests that bring up the .war in the context of integration tests via gretty plugin.
It would be cleaner if we figured out how to gracefully get this thread to stop.
as a work around I am going to have a configurable option to force stop all threads (I discussed with Ionnais). This will take effect in existing contextListener: InjectedWebListener.
Are you sure that's correct?
ubuntu@ip-172-31-28-231:~$ curl http://169.254.169.254/latest/meta10-data/local-hostname
> <?xml version="1.0" encoding="iso-8859-1"?>
> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
> <head>
> <title>404 - Not Found</title>
> </head>
> <body>
> <h1>404 - Not Found</h1>
> </body>
> </html>
ubuntu@ip-172-31-28-231:~$ curl http://169.254.169.254/latest/meta-data/local-hostname
ip-172-31-28-231.us-west-2.compute.internal
Implement the following using Batch*Item APIs:
so that code diffs look the same in the side and the browser
When using batch statements, providing CL at individual statement level is not being respected, hence it needs to be provided in the batch statement.
How will the instance booted by the autoscale group automatically join the cluster? The AWSLocalClusterDiscovery seems to only get the host machine's IP. And If we don't create an AMI with ndbench already installed then what is the point of having an ASG?
Have I misunderstood something ? Please clarify.
Thanks
Hi,
I tried a lot over the web to find a good getting started tutorial for ndbench to benchmark Cassandra but failed to find one. None of the posts available online walks us through testing cassandra in cloud environment. If you guys can guide me to any information on it, that would be of a great help.
If there is any configuration file which I need to use to give cluster details with credentials please point me to it.
Thanks in advance.
While attempting to push and run this app in a CF environment, we got an error stating that
ndbench-web-0.4.0-SNAPSHOT.war didn't exist and once checked we noted that ndbench-web-0.3.11-SNAPSHOT.war had been downloaded. It appears that the 0.4.0 release is not available even though it is referred to in the manifests.yml.
Hello, is there any support (or future plans) for rate limit (throttling) in ndbench?
We need to add the ES plugin in the OSS.
I'm stressing out Cassandra 2.1.9 using Netflix NDBench running on AWS.
Hardware: m4.4xlarge - 64GB of RAM and 16 vCPU
There are 3 boxes with this hardware. 1) NDbench 2) Cass node A 2) Cass node B.
I can't get higher RPS than ~2.5k for R/w.
Given this hardware should I be able to get better RPS with single NDBench machine?
Stress Test Information
Cheers,
Diego Pacheco
Hi,
I'm trying to compile ndbench and am getting the following error:
$ ./gradlew build
> Configure project :
Inferred project: ndbench, version: 0.3.4-SNAPSHOT
Publication nebula not found in project :.
Publication named 'nebula' does not exist for project ':' in task ':artifactoryPublish'.
> Task :ndbench-core:compileJava
error: Bad service configuration file, or exception thrown while constructing Processor object: javax.annotation.processing.Processor: Provider org.antlr.v4.runtime.misc.RuleDependencyProcessor not found
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':ndbench-core:compileJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 0s
8 actionable tasks: 1 executed, 7 up-to-date
If it makes any difference, this is on Mac OS and java 1.8.0_131.
Thanks for any help!
spelling fix
Looking at how latency is reported e.g.:
https://github.com/Netflix/ndbench/blob/master/ndbench-core/src/main/java/com/netflix/ndbench/core/operations/ReadOperation.java#L43
Long startTime = System.nanoTime();
String value = client.readSingle(key);
monitor.recordReadLatency((System.nanoTime() - startTime)/1000);
The above implies the operation is synchronous and that the harness has no notion of schedule (we don't know when we intend to start measurement). This generally leads to coordinate omission. The fix is usually to add some notion of schedule mapping each operation to it's intended start time.
This is a common issue in load generators, see following blog posts(which I wrote) on the issue, it's impact and resolution in YCSB and Cassandra Stress:
http://psy-lob-saw.blogspot.com/2015/03/fixing-ycsb-coordinated-omission.html
http://psy-lob-saw.blogspot.com/2016/07/fixing-co-in-cstress.html
takes a long time to build because gradle will download all versions.
ndbench builds successfully when it is a top level git repo but it fails when it is submoduled into another git repo. It appear's that the nebula.netflixoss
plugin looks for a .git
directory at the root of the project.
Not sure what if anything you all would like to do about it. It was confusing for us at first but we got it to build successfully by setting -Pgit.root=<root-dir>
.
Setting -Pgit.root
isn't bad for us, but it took some time to understand the cause of the failure, and others might experience this pain. It would be nice if the build script wasn't so tightly coupled to git or if the plugin handled submodules gracefully. If there was an option to override the git integration that would also be a fine solution.
$ ./gradlew build --stacktrace
Git repository not found at <my-repo>/<some>/<path>/ndbench -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
FAILURE: Build failed with an exception.
* Where:
Build file '<my-repo>/<some>/<path>/ndbench/build.gradle' line: 11
* What went wrong:
An exception occurred applying plugin request [id: 'nebula.netflixoss', version: '3.6.0']
> Failed to apply plugin [id 'nebula.netflixoss']
> Cannot set property 'defaultVersionStrategy' on null object
* Try:
Run with --info or --debug option to get more log output.
* Exception is:
org.gradle.api.plugins.InvalidPluginException: An exception occurred applying plugin request [id: 'nebula.netflixoss', version: '3.6.0']
at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.applyPlugin(DefaultPluginRequestApplicator.java:224)
at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.applyPlugins(DefaultPluginRequestApplicator.java:171)
at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:156)
at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:38)
at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:25)
at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55)
at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:573)
at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:125)
at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:42)
at org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:38)
at org.gradle.initialization.DefaultGradleLauncher$2.run(DefaultGradleLauncher.java:151)
at org.gradle.internal.Factories$1.create(Factories.java:22)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:53)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:148)
at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:33)
at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:112)
at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:106)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:63)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:106)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:92)
at org.gradle.launcher.exec.GradleBuildController.run(GradleBuildController.java:66)
at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:41)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:79)
at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:51)
at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:59)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:47)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
at org.gradle.util.Swapper.swap(Swapper.java:38)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:72)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.HintGCAfterBuild.execute(HintGCAfterBuild.java:44)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:293)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
Caused by: org.gradle.api.internal.plugins.PluginApplicationException: Failed to apply plugin [id 'nebula.netflixoss']
at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:155)
at org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:112)
at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator$5.run(DefaultPluginRequestApplicator.java:174)
at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.applyPlugin(DefaultPluginRequestApplicator.java:212)
... 57 more
Caused by: java.lang.NullPointerException: Cannot set property 'defaultVersionStrategy' on null object
at nebula.plugin.netflixossproject.NetflixOssProjectPlugin$_apply_closure1.doCall(NetflixOssProjectPlugin.groovy:55)
at nebula.plugin.netflixossproject.NetflixOssProjectPlugin.apply(NetflixOssProjectPlugin.groovy:54)
at nebula.plugin.netflixossproject.NetflixOssProjectPlugin.apply(NetflixOssProjectPlugin.groovy)
at org.gradle.api.internal.plugins.ImperativeOnlyPluginApplicator.applyImperative(ImperativeOnlyPluginApplicator.java:35)
at org.gradle.api.internal.plugins.RuleBasedPluginApplicator.applyImperative(RuleBasedPluginApplicator.java:43)
at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:139)
... 60 more
BUILD FAILED
Total time: 0.835 secs
We currently test NDBench by deploying in a compute instance. However, we rarely test if it works with appRun
. The latter is the recommended way for our OSS users. Since we wrap the ndbench-web
with our internal tomcat configuration we probably need to add some unit testing to make sure that appRun works and at least some REST calls are successful.
New GuiceModule that an environment is loaded based on properties has an issue with AWS deployments #32
Please let me know if more information is needed.
On a CentOS 7 machine with java 8. I'm getting the following error after running:
./gradlew clean build
> Task :ndbench-dyno-plugins:compileJava
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedis.java:62: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedis$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedis.java:65: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedis$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedis.java:64: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisExtFunc.java:170: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisExtFunc$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisExtFunc.java:173: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisExtFunc$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisExtFunc.java:172: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisGetSetPipeline.java:77: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisGetSetPipeline$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisGetSetPipeline.java:80: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisGetSetPipeline$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisGetSetPipeline.java:79: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisHashPipeline.java:64: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisHashPipeline$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisHashPipeline.java:67: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisHashPipeline$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisHashPipeline.java:66: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalDynomiteProxyPlugin.java:68: error: <anonymous com.netflix.ndbench.plugin.local.dynomite.proxy.LocalDynomiteProxyPlugin$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalDynomiteProxyPlugin.java:71: error: getHosts() in <anonymous com.netflix.ndbench.plugin.local.dynomite.proxy.LocalDynomiteProxyPlugin$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalDynomiteProxyPlugin.java:70: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalHttpEndpointBasedTokenMapSupplier.java:23: error: no suitable constructor found for HttpEndpointBasedTokenMapSupplier(String,int,int)
super("http://{hostname}:8081/REST/v1/admin/cluster_describe", 8081, dynomitePort);
^
constructor HttpEndpointBasedTokenMapSupplier.HttpEndpointBasedTokenMapSupplier() is not applicable
(actual and formal argument lists differ in length)
constructor HttpEndpointBasedTokenMapSupplier.HttpEndpointBasedTokenMapSupplier(int) is not applicable
(actual and formal argument lists differ in length)
constructor HttpEndpointBasedTokenMapSupplier.HttpEndpointBasedTokenMapSupplier(String,int) is not applicable
(actual and formal argument lists differ in length)
16 errors
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':ndbench-dyno-plugins:compileJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
=================================================== Second try ===================================================
> Task :ndbench-dyno-plugins:compileJava FAILED
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedis.java:62: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedis$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedis.java:65: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedis$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedis.java:64: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisExtFunc.java:170: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisExtFunc$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisExtFunc.java:173: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisExtFunc$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisExtFunc.java:172: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisGetSetPipeline.java:77: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisGetSetPipeline$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisGetSetPipeline.java:80: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisGetSetPipeline$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisGetSetPipeline.java:79: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisHashPipeline.java:64: error: <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisHashPipeline$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisHashPipeline.java:67: error: getHosts() in <anonymous com.netflix.ndbench.plugin.dyno.DynoJedisHashPipeline$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/dyno/DynoJedisHashPipeline.java:66: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalDynomiteProxyPlugin.java:68: error: <anonymous com.netflix.ndbench.plugin.local.dynomite.proxy.LocalDynomiteProxyPlugin$1> is not abstract and does not override abstract method getHosts() in HostSupplier
HostSupplier hSupplier = new HostSupplier() {
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalDynomiteProxyPlugin.java:71: error: getHosts() in <anonymous com.netflix.ndbench.plugin.local.dynomite.proxy.LocalDynomiteProxyPlugin$1> cannot implement getHosts() in HostSupplier
public Collection<Host> getHosts() {
^
return type Collection<Host> is not compatible with List<Host>
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalDynomiteProxyPlugin.java:70: error: method does not override or implement a method from a supertype
@Override
^
~/ndbench/ndbench-dyno-plugins/src/main/java/com/netflix/ndbench/plugin/local/dynomite/proxy/LocalHttpEndpointBasedTokenMapSupplier.java:23: error: no suitable constructor found for HttpEndpointBasedTokenMapSupplier(String,int,int)
super("http://{hostname}:8081/REST/v1/admin/cluster_describe", 8081, dynomitePort);
^
constructor HttpEndpointBasedTokenMapSupplier.HttpEndpointBasedTokenMapSupplier() is not applicable
(actual and formal argument lists differ in length)
constructor HttpEndpointBasedTokenMapSupplier.HttpEndpointBasedTokenMapSupplier(int) is not applicable
(actual and formal argument lists differ in length)
constructor HttpEndpointBasedTokenMapSupplier.HttpEndpointBasedTokenMapSupplier(String,int) is not applicable
(actual and formal argument lists differ in length)
16 errors
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':ndbench-dyno-plugins:compileJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
so that tables can adjust their own provisioning. This should make the auto-tune feature of ndbench work for DynamoDB.
I built the latest commit on the master branch with ./gradlew build
and then started the app with DISCOVERY_ENV=AWS ./gradlew appRun
.
on GET http://localhost:8080/REST?ndbench/driver/init/DynamoDBKeyValue
I get an error:
{"isSuccess":false,"message":"Client initialization failed!"}
Looking at the tomcat application log, I find the stack trace.
10:11:39,131 ERROR NdBenchResource:208 - Error initializing the client - DynamoDBKeyValue
java.lang.RuntimeException: Client not found: DynamoDBKeyValue
at com.netflix.ndbench.core.NdBenchClientFactory.getClient(NdBenchClientFactory.java:42)
at com.netflix.ndbench.core.resources.NdBenchResource.init(NdBenchResource.java:203)
at java.lang.Thread.run(Thread.java:748)
Isn't guice supposed to be doing class path scanning to find all implementations of NdBenchClient
?
I am thinking about adding writeBulk and readBulk operations to ndbench. My thought is to add the bulkSize parameter in startWrites and startReads and default it to 1, which falls back to writeSingle and readSingle. Any comments?
@ipapapa Looks like right after the comming #130 I see my local ndbench breaks due to Eureka dependency. While I resolved some of it using
eureka.validateInstanceId=false
with that the app runs. However I see these exceptions all the time
13:12:18,230 ERROR DiscoveryClient:972 - DiscoveryClient_UNKNOWN/10.101.11.44 - was unable to refresh its cache! status = There is no known eureka server; cluster server list is empty com.netflix.discovery.shared.transport.TransportException: There is no known eureka server; cluster server list is empty at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:107) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.getApplications(EurekaHttpClientDecorator.java:134) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$6.execute(EurekaHttpClientDecorator.java:137) at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.getApplications(EurekaHttpClientDecorator.java:134) at com.netflix.discovery.DiscoveryClient.getAndStoreFullRegistry(DiscoveryClient.java:1051) at com.netflix.discovery.DiscoveryClient.fetchRegistry(DiscoveryClient.java:965) at com.netflix.discovery.DiscoveryClient.refreshRegistry(DiscoveryClient.java:1471) at com.netflix.discovery.DiscoveryClient$CacheRefreshThread.run(DiscoveryClient.java:1438) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 13:12:18,232 ERROR DiscoveryClient:860 - DiscoveryClient_UNKNOWN/10.101.11.44 - was unable to send heartbeat! com.netflix.discovery.shared.transport.TransportException: There is no known eureka server; cluster server list is empty at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:107) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92) at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89) at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:846) at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1399) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Is there any way to disable Eureka dependency or suppress these with an eureka config?
different microservices hava different data amount ,data type and data size, is it necessary to consider these differences ?
style issue / java8
Hello, I am having a problem building ndbench on a machine that has never built it. The gradlew build is throwing errors. Perhaps this is a documentation error or a configuration error on my part.
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
○ → git clone [email protected]:Netflix/ndbench.git
remote: Counting objects: 2936, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 2936 (delta 3), reused 1 (delta 1), pack-reused 2931
Receiving objects: 100% (2936/2936), 1.15 MiB | 1.90 MiB/s, done.
Resolving deltas: 100% (809/809), done.
○ → cd ndbench
○ → ls
LICENSE buildViaTravis.sh gradlew ndbench-api ndbench-dyno-plugins ndbench-janusgraph-plugins settings.gradle
OSSMETADATA codequality gradlew.bat ndbench-cass-plugins ndbench-es-plugins ndbench-proxy.js
README.md dynomite.yml img ndbench-core ndbench-evcache-plugins ndbench-sample-plugins
○ → ./gradlew clean build
> Configure project :
Inferred project: ndbench, version: 0.4.0-SNAPSHOT
Download http://dax-sdk.s3-website-us-west-2.amazonaws.com/java/DaxJavaClient-latest.jar
Publication nebula not found in project :.
Publication named 'nebula' does not exist for project ':' in task ':artifactoryPublish'.
> Task :ndbench-core:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
FAILURE: Build failed with an exception.
* What went wrong:
Could not resolve all files for configuration ':ndbench-dynamodb-plugins:compileClasspath'.
> Could not find :DaxJavaClient-latest:.
Searched in the following locations:
file:/Users/pivotal/workspace/ndbench/ndbench-dynamodb-plugins/build/DaxJavaClient-latest.jar
Required by:
project :ndbench-dynamodb-plugins
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 6s
24 actionable tasks: 23 executed, 1 up-to-date
Instead just trying a build....
○ → ./gradlew build
> Configure project :
Inferred project: ndbench, version: 0.4.0-SNAPSHOT
Download http://dax-sdk.s3-website-us-west-2.amazonaws.com/java/DaxJavaClient-latest.jar
Publication nebula not found in project :.
Publication named 'nebula' does not exist for project ':' in task ':artifactoryPublish'.
> Task :ndbench-evcache-plugins:compileJava FAILED
/<home>/ndbench/ndbench-evcache-plugins/src/main/java/com/netflix/ndbench/plugin/evcache/EVCacheTestPlugin.java:36: error: EVCacheTestPlugin is not abstract and does not override abstract method writeBulk(List<String>) in NdBenchClient
public class EVCacheTestPlugin implements NdBenchClient {
^
1 error
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':ndbench-evcache-plugins:compileJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 9s
21 actionable tasks: 10 executed, 11 up-to-date
Need the following configuration parameters additionally:
Hi There,
Recently I am getting this error when I want to run app or build it again
error: Bad service configuration file, or exception thrown while constructing Processor object: javax.annotation.processing.Processor: Provider org.antlr.v4.runtime.misc.RuleDependencyProcessor not found
:ndbench-core:compileJava FAILED
FAILURE: Build failed with an exception.
What went wrong:
Execution failed for task ':ndbench-core:compileJava'.
Compilation failed; see the compiler error output for details.
Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
This is with --info
All input files are considered out-of-date for incremental task ':ndbench-core:compileJava'.
Compiling with JDK Java compiler API.
error: Bad service configuration file, or exception thrown while constructing Processor object: javax.annotation.processing.Processor: Provider org.antlr.v4.runtime.misc.RuleDependencyProcessor not found
:ndbench-core:compileJava FAILED
:ndbench-core:compileJava (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.242 secs.
FAILURE: Build failed with an exception.
Compilation failed; see the compiler error output for details.
BUILD FAILED
Total time: 2.416 secs
Stopped 0 compiler daemon(s).
Received result Failure[value=org.gradle.initialization.ReportedException: org.gradle.internal.exceptions.LocationAwareException: Execution failed for task ':ndbench-core:compileJava'.] from daemon DaemonInfo{pid=8831, address=[b3e93fbb-894a-445b-ac91-649371f0dc9b port:52240, addresses:[/0:0:0:0:0:0:0:1, /127.0.0.1]], state=Idle, lastBusy=1507564159773, context=DefaultDaemonContext[uid=fd424181-cd01-40e5-aaa5-ba4e09a1734d,javaHome=/opt/bitnami/java,daemonRegistryDir=/root/.gradle/daemon,pid=8831,idleTimeout=10800000,daemonOpts=-XX:MaxPermSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xmx1024m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} (build should be done).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.