Comments (4)
Technically the openHistorian, being based on SnapDB, can use any compression algorithm.
The default one is designed to do a good job with a variety of streaming time-series data sources, we casually refer to this as Time-Series Special Compression (or TSSC) - but there are variations of this, e.g., the one used in STTP. Most of the compression algorithms were developed by @StevenChisholm.
For the the current implementation in openHistorian, you can find the code here, check the Encode
and Decode
methods:
from openhistorian.
Where can I find the specific information of the TSSC algorithm or how can I learn this algorithm. Is there any literature which you recommend about this algorithm?
from openhistorian.
openHistorian can archive any time-series type data in a streaming fashion, so the goal is to apply general purpose streaming compression rules to specific data elements, longitudinally, and based on the nature of the data, produce good compression ratios with minimal CPU impact.
The TSSC algorithm used by openHistorian is more simple than the one used by STTP in the number of measurement states that is maintained. When the IEEE 2664 standard is released, it will contain a section and an appendix describing TSSC in greater detail.
In general, TSSC takes each of the elements of a time-series measurement and handles each data type with separate compression algorithms, creating parallel compression streams for each data element in the measurement. The nature of the data element being compressed then infers the necessary compression algorithm tuning to produce the best results.
For archived data, timestamps will be near each other, normally varying by no more than a few seconds. For the 64-bit timestamps, this means the data variation may only occur in the bottom 16 of the total 64 bits of the timestamp. With the bulk of the bits repeating invariably, the total bit set needs to be only archived once or on substantial change, then only the changing bits need to be archived.
Additionally, if the timestamps vary less, the algorithm can automatically adjust and archive even fewer bits. This same type of pattern works well for identification numbers, which are finite in number, and state flags, which vary little. Data values, however, need special attention.
Data value elements for a given measurement can be rather random, however, many values change slowly over time, from measurement to measurement. For example, a measured frequency value tends to change only incrementally over several data measurements; in fact, other frequencies in the same subscription may only differ by just a few bits. With this knowledge a few extra opcodes can be maintained that represent the unvarying bits of many types of measurements. Now only the changed bit values need to be encoded into the archive so that the stream can be reinflated without loss upon reception.
from openhistorian.
Thank you very much!I got it.
from openhistorian.
Related Issues (20)
- API to query measurement details HOT 1
- Data missing on export data from openhistorian webmanager HOT 4
- ATMEGA2560 Modbus openHistorian display to Grafana HOT 8
- Full Documentation for OpenHistorian HOT 3
- Is there any mechanism to store Historian Data directly into any RDBMS like Oracle/SQL Server instead of Files HOT 3
- Update openHistorian Grafana Instance HOT 1
- Multiple indexes
- Service unexpected stop HOT 23
- Grafana and OH User Synchronization requires log-on HOT 2
- Trend/Export Measurements Timestamps HOT 1
- Adding New Device (hardware PMU) HOT 10
- Fix build tools to update version in SDK project files
- Reinstall under different service credentials does not overwrite HTTP reservations HOT 2
- How to disable openHistorian debug host automated start HOT 4
- How to get openPDC data in openHistorian. HOT 2
- Error in Github Desktop when pulling Source/Libraries/PythonAPIModule' HOT 1
- How to connect openHistorian with other datasources
- Compression algorithm about the OpenHistoian HOT 2
- COMTRADE Export with fields that contain commas
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openhistorian.