addthis / hydra Goto Github PK

Hi,

I've noticed that the hydra project has appeared for several years and is still under active developing. Currently, similar tasks as unified batch and stream processing could also be delivered using some popular streaming engine as Flink, for example, Uber uses AthenaX to facilitate developers with SQL on Flink to process massive data from both batch and stream data. How does hydra compared with such platform? Thanks~

when build it by mvn package use -P bdbje , hydra-local.sh generated but run it
/hydra-uber/bin/local-stack.sh start - start ZooKeeper
./hydra-uber/bin/local-stack.sh start - start spawn, querymaster etc.
./hydra-uber/bin/local-stack.sh seed - add some sample data

not been generated who can finish it

http://pastebin.com/1tq7HsU3

Hi, I am trying to get started with Hydra, I am running Hydra in local configuration for testing. Where can I get sample job configurations and sample data? Where should the data be placed if I have a csv file? I managed to configure a job, however not able to provide input data. Please help. Your sample job with automatic data inject runs correctly.

Exception in thread "SourceReader" java.lang.ExceptionInInitializerError
at com.addthis.hydra.task.run.TaskFeeder.run(TaskFeeder.java:220)
Caused by: java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
at java.lang.Runtime.addShutdownHook(Runtime.java:211)
at com.google.common.util.concurrent.MoreExecutors$Application.addShutdownHook(MoreExecutors.java:223)
at com.google.common.util.concurrent.MoreExecutors$Application.addDelayedShutdownHook(MoreExecutors.java:195)
at com.google.common.util.concurrent.MoreExecutors$Application.getExitingScheduledExecutorService(MoreExecutors.java:187)
at com.google.common.util.concurrent.MoreExecutors$Application.getExitingScheduledExecutorService(MoreExecutors.java:219)
at com.google.common.util.concurrent.MoreExecutors.getExitingScheduledExecutorService(MoreExecutors.java:169)
at com.addthis.muxy.MuxFileDirectoryCacheInstance.(MuxFileDirectoryCacheInstance.java:62)
at com.addthis.muxy.MuxFileDirectoryCacheInstance.(MuxFileDirectoryCacheInstance.java:84)
at com.addthis.muxy.MuxFileDirectoryCacheInstance.(MuxFileDirectoryCacheInstance.java:41)
at com.addthis.muxy.MuxFileDirectoryCacheInstance$Builder.build(MuxFileDirectoryCacheInstance.java:312)
at com.addthis.muxy.MuxFileDirectoryCache.(MuxFileDirectoryCache.java:32)
... 1 more

site I noticed you guys use FASTLY as your CDN, but I don’t get why you don’t use for example CLOUDFLARE which would provide huge benefits:

1. Cloudflare bandwith is FREE
   a. Cloudflare doesn’t charge for bandwith, it’s free (could be they charge you guys since you are a huge company but for sure would be cheaper than fastly)
   b. This should save your company thousands of dollars
2. More POP/datacenters locations to deliver the assets:
   a. Cloudflare: https://www.cloudflare.com/network-map
   i. And this are the ones they plan for 2015 which are insane: https://twitter.com/davidmytton/status/543126344377585664
   b. Fastly: http://www.fastly.com/network/
3. Has many other good features but I think just the 2 above are more than enough to take into account

While trying to build hydra package I got following error:

Failed to execute goal on project hydra-essentials: Could not resolve dependencies for project com.addthis.hydra:hydra-essentials:jar:5.5.13: Could not find artifact com.addthis:basis:jar:3.0.4 in central (http://repo.maven.apache.org/maven2) -> [Help 1]

Please help in resolving. Tried older version 3.4 also. Same error with -P bdbje option.

sett http://pastebin.com/GA72n2qu

Hi,

User Guide and Reference links are not loading.

i've started and stopped a virgin local stack a few times then and after letting it sit idle for 15-20 minutes, randomly did a du -sh on the hydra run directory. surprisingly it came it at close to 500MB. i drilled down and found a huge chunk of zookeeper logs in hydra/etc/zookeeper/version-2/ ... it seems they are all /spawn/queue updates. not clear why that would be given the fact that the cluster was completely idle and lacked jobs, tasks, hosts, etc. not sure how to really debug this or if i want to (i don't).

hydra's user-agent is Mozilla/4.0 (Hydra),usually blocked by waf.
can we custom the user-agent?

Hi,
Where can I get the exact syntax/format of all the sections and parameters in the job configuration file and the queries. The reference document provides all the parameters, but only a few examples are provided, not the exhaustive syntax. Same is the case for queries.