The influxv2tovm from frli4797

influxv2tovm's Introduction

Script to import data from InfluxDB >=2.0 to VictoriaMetrics and is highly inspired and developed from https://github.com/jonppe/influx_to_victoriametrics/ by https://github.com/jonppe

This include some support for batching larger data sets rather than trying to read it all in one request.

https://github.com/VictoriaMetrics/vmctl provides similar features for InfluxDB 1.X.

Every unique timeseries is queried one by one and exported to VictoriaMetrics. For large databases, this might cause too large chunks and some kind of further split would be needed.

Usage

./influx_export.py -h
usage: influxv2tovm.py [-h] [--INFLUXDB_V2_ORG INFLUXDB_V2_ORG] [--INFLUXDB_V2_URL INFLUXDB_V2_URL] [--INFLUXDB_V2_TOKEN INFLUXDB_V2_TOKEN]
                        [--INFLUXDB_V2_SSL_CA_CERT INFLUXDB_V2_SSL_CA_CERT] [--INFLUXDB_V2_TIMEOUT INFLUXDB_V2_TIMEOUT] [--INFLUXDB_V2_VERIFY_SSL INFLUXDB_V2_VERIFY_SSL]
                        [--vm-addr VM_ADDR]
                        bucket

Script for exporting InfluxDB data into victoria metrics instance. InfluxDB settings can be defined on command line
or as environment variables (or in .env file  if python-dotenv is installed). 

InfluxDB related args described in https://github.com/influxdata/influxdb-client-python#via-environment-properties

positional arguments:
  bucket                InfluxDB source bucket, which is also added to the tag "db" in VM

optional arguments:
  -h, --help            show this help message and exit
  --INFLUXDB_V2_ORG INFLUXDB_V2_ORG, -o INFLUXDB_V2_ORG
                        InfluxDB organization
  --INFLUXDB_V2_URL INFLUXDB_V2_URL, -u INFLUXDB_V2_URL
                        InfluxDB Server URL, e.g., http://localhost:8086
  --INFLUXDB_V2_TOKEN INFLUXDB_V2_TOKEN, -t INFLUXDB_V2_TOKEN
                        InfluxDB access token.
  --INFLUXDB_V2_SSL_CA_CERT INFLUXDB_V2_SSL_CA_CERT, -S INFLUXDB_V2_SSL_CA_CERT
                        Server SSL Cert
  --INFLUXDB_V2_TIMEOUT INFLUXDB_V2_TIMEOUT, -T INFLUXDB_V2_TIMEOUT
                        InfluxDB timeout
  --INFLUXDB_V2_VERIFY_SSL INFLUXDB_V2_VERIFY_SSL, -V INFLUXDB_V2_VERIFY_SSL
                        Verify SSL CERT.
  --vm-addr VM_ADDR, -a VM_ADDR
                        VictoriaMetrics server
  --dry-run, -n         Dry run, don't write changes to VM
  --pivot, -P           Pivot entity_id to be measurement, this is specific to Homeassistant metrics

Author: Fredrik J-L
Thanks to: Johannes Aalto

SPDX-License-Identifier: Apache-2.0

influxv2tovm's People

Contributors

Watchers

influxv2tovm's Issues

TypeError: string indices must be integers, not 'str'

Hello,

trying to migrate influxdb 2.x data to VM, I got the following error messages: `

Dry run True Pivot False Finding unique time series. Traceback (most recent call last): File "/homeassistant/pyscript/influxv2tovm.py", line 343, in <module> main(vars(parser.parse_args())) File "/homeassistant/pyscript/influxv2tovm.py", line 268, in main migrator.migrate() File "/homeassistant/pyscript/influxv2tovm.py", line 95, in migrate measurements_and_fields = self.__find_all_measurements() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/homeassistant/pyscript/influxv2tovm.py", line 188, in __find_all_measurements measurements_and_fields.update(df[self.__measurement_key].unique()) ~~^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: string indices must be integers, not 'str' Exception ignored in: <function InfluxMigrator.__del__ at 0x7f38143d1580> Traceback (most recent call last): File "/homeassistant/pyscript/influxv2tovm.py", line 78, in __del__ self.__progress_file.close() ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'InfluxMigrator' object has no attribute '_InfluxMigrator__progress_file

I tried to migrate a special bucket with filtered Data (time frame) and dropped tag values.

When I try to migrate an acual active filled bucket without any filtering or shaping, the dry run works and I get the following end-notice:

Exception ignored in: <function InfluxMigrator.__del__ at 0x7f9c8b5f9580>ading. Total: 524.3 kB (1/1) Traceback (most recent call last): File "/homeassistant/pyscript/influxv2tovm.py", line 78, in __del__ self.__progress_file.close() ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'InfluxMigrator' object has no attribute '_InfluxMigrator__progress_file' All done

Thanks for any help on this

Greets

Text encoding issues migrating Home Assistant data

Thank you for the this fork and the work to add the logic for HomeAssistant migrations. It got me 80% of the way there but I still experienced significant issues while migrating my Influx2 HA dataset.

My issues were related to text encoding and your version of the code not correctly complying to the Influx Line Protocol Standard, especially the section about escaping and quoting.

I have modified the code to correctly escape the names and values of keys and the value columns is now quoted for strings with escaped ", /r, /n as these would break the line protocol. I have a pretty diverse dataset and managed to migrate over 4GB without any problems.

I will fork your repo and upload mine with a devcontainer definition which should make it a lot easier for anyone to set this up who is not familiar with Python

Recommend Projects

frli4797 / influxv2tovm Goto Github PK

influxv2tovm's Introduction

Usage

influxv2tovm's People

Contributors

Watchers

influxv2tovm's Issues

TypeError: string indices must be integers, not 'str'

Text encoding issues migrating Home Assistant data

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent