Git Product home page Git Product logo

influxdb-lineprotocol's Introduction

NAME

InfluxDB::LineProtocol - Write and read InfluxDB LineProtocol

VERSION

version 1.015

SYNOPSIS

use InfluxDB::LineProtocol qw(data2line line2data);

# convert some Perl data into InfluxDB LineProtocol
my $influx_line = data2line('measurement', 42);
my $influx_line = data2line('measurement', { cost => 42 });
my $influx_line = data2line('measurement', 42, { tag => 'foo'} );

# convert InfluxDB Line back into Perl
my ($measurement, $values, $tags, $timestamp) =
  line2data("metric,location=eu,server=srv1 value=42 1437072299900001000");

DESCRIPTION

InfluxDB is a rather new time series database. Since version 0.9 they use their LineProtocol to write time series data into the database. This module allows you to generate such a line from a datastructure, handling all the annoying escaping and sorting for you. You can also use it to parse a line (maybe you want to add some tags to a line written by another app).

Please read the InfluxDB docs so you understand how metrics, values and tags work.

InfluxDB::LineProtocol will always try to implement the most current version of the InfluxDB line protocol, while allowing you to also get the old behaviour. Currently we support 0.9.3 and newer per default, and 0.9.2 if you ask nicely.

FUNCTIONS

data2line

data2line($metric, $single_value);
data2line($metric, $values_hashref);
data2line($metric, $value, $tags_hashref);
data2line($metric, $value, $nanoseconds);
data2line($metric, $value, $tags_hashref, $nanoseconds);

data2line takes various parameters and converts them to an InfluxDB Line.

metric has to be valid InfluxDB measurement name. Required.

value can be either a scalar, which will be turned into "value=$value"; or a hashref, if you want to write several values (or a value with another name than "value"). Required.

tags_hashref is an optional hashref of tag-names and tag-values.

nanoseconds is an optional integer representing nanoseconds since the epoch. If you do not pass it, InfluxDB::LineProtocol will use Time::HiRes to get the current timestamp.

line2data

my ($metric, $value_hashref, $tags_hashref, $timestamp) = line2data( $line );

line2data parses an InfluxDB line and always returns 4 values.

tags_hashref is undef if there are no tags!

PRECISION

InfluxDB support different timestamp precisions:

Nanosecond (ns, the default), microseconds (us), milliseconds (ms), seconds (s), minutes (m) and hours (h). If you do not want to generate lines using nanoseconds (which might be a good idea, because InfluxDB uses less space and has better performance if you choose a smaller precision), you can specify the wanted precision on load of InfluxDB::LineProtocol:

use InfluxDB::LineProtocol->import(qw(data2line precision=ms));

Please note that yo have to tell InfluxDB the precision when posting lines to /write!

LOADING LEGACY PROTOCOL VERSIONS

To use an old version of the line protocol, specify the version you want when loading InfluxDB::LineProtocol:

use InfluxDB::LineProtocol qw(v0.9.2 data2line);

You will get a version of data2line that conforms to the 0.9.2 version of the line protocol.

Currently supported version are:

  • 0.9.3 and newer

    default, no need to specify anything

  • 0.9.2

    load via v0.9.2

TODO

SEE ALSO

  • InfluxDB provides access to the old 0.8 API. It also allows searching etc.
  • AnyEvent::InfluxDB - An asynchronous library for InfluxDB time-series database. Does not implement escaping etc, so if you want to use AnyEvent::InfluxDB to send data to InfluxDB you can use InfluxDB::LineProtocol to convert your measurement data structure before sending it via AnyEvent::InfluxDB.

THANKS

Thanks to

  • validad.com for funding the development of this code.
  • Jose Luis Martinez for implementing negative & exponential number support and pointing out the change in the line protocol in 0.9.3.
  • mvgrimes for fixing a bug when nanosecond timestamps cause some Perls to render the timestamp in scientific notation.
  • Adrian Popa for fixing a bug when handling large scientific notation data.
  • zachary-bull for adding code to escape = in tag keys.

AUTHOR

Thomas Klausner [email protected]

COPYRIGHT AND LICENSE

This software is copyright (c) 2016 - 2022 by Thomas Klausner.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

POD ERRORS

Hey! The above document had some coding errors, which are explained below:

  • Around line 173:

    L<> starts or ends with whitespace

influxdb-lineprotocol's People

Contributors

domm avatar gregoa avatar guillemj avatar klaus avatar mad-ady avatar manwar avatar mariopaumann avatar mvgrimes avatar pplu avatar zachary-bull avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

influxdb-lineprotocol's Issues

InfluxDB::LineProtocol bug when handling large scientific notation data

Hello, I hope you are doing well.

I'm using your InfluxDB::LineProtocol version 1.011 to bring InfluxDB support to Smokeping (not sure if you're familiar with it: oetiker/SmokePing#201). Thanks for the module!

During trials I've noticed I'm not getting some data into influxdb and on closer inspection, the problem comes from the way the module encodes floats written in scientific notation. Data which is smaller than 1 is encoded correctly, while data over 1 is not and is converted to string.
Here is the test code:

use InfluxDB::LineProtocol qw(data2line line2data);
 
# convert some Perl data into InfluxDB LineProtocol
my $littlething = sprintf('%e', 0.22);
my $influx_line = data2line('measurement', $littlething, { tag => 'foo'} );
print $influx_line."\n";

my $bigthing = sprintf('%e', 1.22);
$influx_line = data2line('measurement', $bigthing, { tag => 'foo'} );
print $influx_line."\n";

This is its standard output (see the quotes on the larger value):

measurement,tag=foo value=2.200000e-01 1589285174118556000
measurement,tag=foo value="1.220000e+00" 1589285174118593000

I've tracked down the problem  to line 75 (_format_value) which doesn't consider + a valid symbol. I've corrected it to this and I confirm that it works correctly:    elsif ( $v =~ /^-?\d+(?:\.\d+)?(?:e(?:-|\+)?\d+)?$/ ) {

measurement,tag=foo value=2.200000e-01 1589285174118556000
measurement,tag=foo value=1.220000e+00 1589285174118593000

FAIL Installing InfluxDB::LineProtocol failed.

Hello

I tried to update InfluxDB::LineProtocol to version 1.003 without success.

Here you will see my install log output:

cpanm (App::cpanminus) 1.7027 on perl 5.020002 built for MSWin32-x64-multi-thread
Work directory is D:\strawberry-perl-5.20.2.1-64bit-PDL\data/.cpanm/work/1438531954.5160
You have make D:\strawberry-perl-5.20.2.1-64bit-PDL\c\bin\dmake.exe
You have LWP 6.13
Falling back to Archive::Tar 2.04
Searching InfluxDB::LineProtocol on cpanmetadb ...
--> Working on InfluxDB::LineProtocol
Fetching http://www.cpan.org/authors/id/D/DO/DOMM/InfluxDB-LineProtocol-1.003.tar.gz
-> OK
Unpacking InfluxDB-LineProtocol-1.003.tar.gz
Entering InfluxDB-LineProtocol-1.003
Checking configure dependencies from META.json
Checking if you have Module::Build 0.28 ... Yes (0.4211)
Configuring InfluxDB-LineProtocol-1.003
Running Build.PL
Created MYMETA.yml and MYMETA.json
Creating new 'Build' script for 'InfluxDB-LineProtocol' version '1.003'
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have warnings 0 ... Yes (1.23)
Checking if you have Time::HiRes 0 ... Yes (1.9726)
Checking if you have Test::Most 0 ... Yes (0.34)
Checking if you have Carp 0 ... Yes (1.3301)
Checking if you have Exporter 0 ... Yes (5.71)
Checking if you have Module::Build 0.28 ... Yes (0.4211)
Checking if you have lib 0 ... Yes (0.63)
Checking if you have strict 0 ... Yes (1.08)
Checking if you have Test::More 0 ... Yes (1.001014)
Building and testing InfluxDB-LineProtocol-1.003
Building InfluxDB-LineProtocol
t/00-load.t ... ok

Failed test 'data2line case 16'

at t/10_basic.t line 190.

got: 'metric value=7.51696501241595e-005 1437072205500681000'

expected: 'metric value=7.51696501241595e-05 1437072205500681000'

Failed test 'line2data case 16'

at t/10_basic.t line 192.

Compared $data->[1]{"value"}

got : '7.51696501241595e-05'

expect : '7.51696501241595e-005'

Looks like you failed 2 tests of 48.

t/10_basic.t ..
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/48 subtests
(1 TODO test unexpectedly succeeded)

Test Summary Report

t/10_basic.t (Wstat: 512 Tests: 48 Failed: 2)
Failed tests: 33-34
TODO passed: 37
Non-zero exit status: 2
Files=2, Tests=49, 1 wallclock secs ( 0.09 usr + 0.05 sys = 0.14 CPU)
Result: FAIL
Failed 1/2 test programs. 2/49 subtests failed.
-> FAIL Installing InfluxDB::LineProtocol failed. See D:\strawberry-perl-5.20.2.1-64bit-PDL\data.cpanm\work\1438531954.5160\build.log for details. Retry with --force to force install it.

Don't hesitate to ask if you want me to test something specific.

Regards

sdeseille

How to force float data type for values

Hi, I am wondering how to force a float data type for certain values. I have a function that returns "0" at the very beginning of the measurement and some float values (e.g. "0.1", "0.2", ...) during the subsequent measurements.

So during the first measurement Influxdb is initialized with an integer data type, the subsequent inserts of float values fail.

My current workaround is to add a final ".0" before calling data2line:

$val .= ".0" unless ($val =~ /\./)

Is there a more elegant way?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.