Hi,

Unable to Push the data, may be the problem in invoking the OnSchedule task

Images shows about the error

Images shows about configuration

Please look into this and let me know where the things went wrong

Add Get Processor to read data from Bigquery and use it into NiFi flows

Problem

Actually the put processor take one file at a time and write it to BigQuery. This can lead the performance.

Solution

Take a batch of files (configurable?) from nifi session and insert in BigQuery table as a batch.

"mvn clean install" command fails because of inconsistent version of the nifi-bigquery-processors dependency in the pom file of the nifi-bigquery-nar module. The part below causes an error and compilation fails.

To fix the issue just remove the "-SNAPSHOT" part from the version.

A common usage pattern for streaming data into BigQuery is to split a logical table into many smaller tables to create smaller sets of data.

To use a template table via the PutBigQuery, add table suffix composition inside the Processor configuration options.

Hi
I am consuming JSON from KAFKA with ConsumeKafka_0_11 JSON TOPIC and than transfer it to PutBigquery, I create table and have permissions but I do not see any record in the table.

At NiFi Data Provenance I see only DROP TYPE

Any Idea how to solve it ?

Thank you

BigQuery ensures data consistency on streaming load using an insertId for each inserted row.

Add configuration field for Put Processor in order to extract the unique id from incoming data. We can use JSON Path to extract the id from incoming data.