jpbirdy / kafka-hdfs-source-connector Goto Github PK

while reading in CSV, it holds the same string format and will not allow it to be converted. This is the only free hdfs2/hdfs3 connector I have found so building out its functionality seems fesaible.

name=test-hdfs
connector.class=hdfs.HDFSSourceConnector
hadoop.conf.dir=/etc/hadoop/conf
hadoop.home=
hdfs.url=hdfs://nameservice1
hdfs.authentication.kerberos=true
connect.hdfs.principal=principal
connect.hdfs.keytab=keytab
hdfs.namenode.principal=principal
kerberos.ticket.renew.period.ms=3600000
file=
file.path=/user/directorypath
topic.prefix=test-hdfs-source

i am using the above config to connect hdfs and push data from directory to kafka

from the logs i could see the connection has been established with hadoop as below

[2018-08-01 08:47:56,013] INFO Created connector test-hdfs (org.apache.kafka.connect.cli.ConnectStandalone:99)
[2018-08-01 08:47:56,013] INFO Hadoop configuration directory /etc/hadoop/conf (hdfs.HDFSSourceTask:79)
[2018-08-01 08:47:56,272] INFO Hadoop namenode principal: hdfs/[email protected] (hdfs.HDFSSourceTask:299)
[2018-08-01 08:47:56,273] INFO Hadoop namenode principal: hdfs/[email protected] (hdfs.HDFSSourceTask:299)
[2018-08-01 08:47:56,609] INFO Login successful for user principal using keytab file principal.keytab (org.apache.hadoop.security.UserGroupInformation:966)
[2018-08-01 08:47:56,655] INFO Login successful for user principalusing keytab file principal.keytab (org.apache.hadoop.security.UserGroupInformation:966)
[2018-08-01 08:47:56,656] INFO Login as: principal (hdfs.HDFSSourceTask:304)
[2018-08-01 08:47:56,656] INFO Login as: principal (hdfs.HDFSSourceTask:304)
[2018-08-01 08:47:56,657] INFO Starting the Kerberos ticket renew thread with period 3600000ms. (hdfs.HDFSSourceTask:334)
[2018-08-01 08:47:56,657] INFO Starting the Kerberos ticket renew thread with period 3600000ms. (hdfs.HDFSSourceTask:334)

also i can see offset loading information as below

[2018-08-01 08:47:58,043] INFO loading offset /user/directory/e24f6ce5225ebcd6-31762fb100000004_2144770316_data.0. (hdfs.HDFSSourceTask:258)
[2018-08-01 08:47:58,073] INFO loading offset /user/directory/e24f6ce5225ebcd6-31762fb100000004_1165567777_data.0. (hdfs.HDFSSourceTask:258)

but i when i run console consumer for the topic i could hardly receive any message.

can you please help me solving this?

Thanks in advance