Comments (5)
Jeff,
I was able to make secor push files to HDFS just by making the following changes:
- Add hadoop-client dependency to pom.xml
- Use secor.s3.filesystem=hdfs
- Use secor.s3.bucket=namenode-host:8020/dir_path
from secor.
This is very legitimate use case. I think adding the support for HDFS ingestion is not hard at all.
In Uploader.java, currently this is hard-coded as:
String s3Prefix = "s3n://" + mConfig.getS3Bucket() + "/" + mConfig.getS3Path();
If you make the destination filesystem a configurable parameter, that probably would just work because underlying we are using HDFS's FileSystem.moveFromLocalFile method to do the S3 file move anyway.
from secor.
@SEJeff out of interest, any reason you can't just use one of the other tools like linkedin's Camus (or the newer Gobblin: https://github.com/linkedin/gobblin)?
Not saying it's an invalid request for Secor to support that as an option -- there is more to like about Secor than just lack of Hadoop dependency for sure but it seems like there are already viable alternatives if you have HDFS target.
See "Hadoop Integration" here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
from secor.
Yeah Camus and Gobblin have a lot more requirements and aren't as straightforward as secor. Also, Camus expects you to write your decoding logic and whatnot in Java. Not that it isn't entirely doable, but isn't preferable for my use case. I'm honestly unimpressed with the confluent tools for this.
Of the various kafka mirroring tools, Secor, to me, is one of the better ones from a design standpoint. I would like to be able to use it in an environment that doesn't use any cloud services whatsoever. We do have a large-ish hdfs install and I was hoping we could use that.
If you fundamentally disagree, feel free to close this issue. It would be really nice to use Secor vs other solutions however.
from secor.
Jeff,
If you want to explore, you can definitely add a HDFSUploader for secor.
We use Hadoop's FileSystem object to work with sequence file uploading, it
should be quite straightforward to have the local sequence file uploading
to HDFS.
On Mon, Nov 23, 2015 at 10:24 AM, Jeff Schroeder [email protected]
wrote:
Yeah Camus and Gobblin have a lot more requirements and aren't as
straightforward as secor. Also, Camus expects you to write your decoding
logic and whatnot in Java. Not that it isn't entirely doable, but isn't
preferable for my use case. I'm honestly unimpressed with the confluent
tools for this.Of the various kafka mirroring tools, Secor, to me, is one of the better
ones from a design standpoint. I would like to be able to use it in an
environment that doesn't use any cloud services whatsoever. We do have a
large-ish hdfs install and I was hoping we could use that.If you fundamentally disagree, feel free to close this issue. It would be
really nice to use Secor vs other solutions however.—
Reply to this email directly or view it on GitHub
#129 (comment).
from secor.
Related Issues (20)
- Blocking Issue when running on K8s on GCP HOT 1
- manually assign avro schema to topic HOT 1
- Crashing exception after some time HOT 1
- Cannot parse timestamp with timezone HOT 1
- Racing condition with parquet writer HOT 3
- Cannot make secor work with AWS instance profile HOT 8
- Ignore parsing exceptions HOT 2
- Expose JVM metrics when using Micrometer HOT 3
- Secor uploads (different) files with the same "name" into different "days". HOT 2
- Kuberntes OOM Error. Very High memory usage > 15gb HOT 2
- Is there any alternative to AvroParquetWriter so that we can avoid using IndexedRecord? HOT 1
- ParquetWriter write method is taking too long to write single avro message HOT 1
- A Simple Study about Configuration Java Types
- PartitionFinalizer fails when there is no data produced in the topic eventhough there are other topics with data HOT 3
- Add headers while calling schema registry for decoding messages HOT 1
- Duplicate message when partition revoking or rebalancing HOT 1
- secor-monitor container not getting started. HOT 2
- Tune file size create by Secor
- Secor with zookeerper less Kafka HOT 1
- Secor filling up local disk space with intermittent data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from secor.