Git Product home page Git Product logo

quantbox.data's Introduction

QuantBox.Data数据存储格式

##介绍 自定义的二进制行情数据存储格式,支持兼容Bar和Tick数据,支持无限深度行情,支持除权除息信息。将行情使用此格式编码后再用7z或zip压缩后再存储。
行情格式名暂定Protobuf Data Zero(.pd0),Protobuf表示所用的核心库,0表示了此种编码的特点。

项目文件介绍

  1. QuantBox.Data.Serializer,编解码库
  2. Test,测试用例
  3. DataInspector,数据文件查看、编辑、转换工具
  4. QuantBox行情数据存储方案,文件格式设计思路
  5. Data,行情示例文件
  6. SelfDecode,.pd0/.7z子解压程序.解压当前目录以及子目录下的数据,到csv文件.默认解压到当前目录的SelfDecodeData下面

在自己的项目中使用

引用QuantBox.Data.Serializer库即可,参考DataInspector中的使用方法,再转换成自己系统中的数据格式
Test中有提供CSV转数据的一档行情与五档行情示例,用户可以基于此进行修改。

行情数据示例

20141225号中金所IF+TF五档行情.7z,使用了7z进行了压缩,可以用7z或WinRAR进行解压。 可以用DataInspector打开20141225.pd0文件进行查看。

编码效率

示例提供的是:
A)20141225号中金所IF+TF五档行情。一共是164266条.
B)20150120号白银ag1506全档行情。一共是32798条.
先将原始数据编码成pd0格式,然后用7z中的PPMd极限压缩。

V1版

A)编码后为4.95 MB,平均约31.6字节每条,再压缩后是1.83 MB,平均11.7字节每条。
B)编码后为10.4 MB,平均334.6字节每条,再压缩后是0.99 MB,平均31.8字节每条

V2版

A)编码后为5.46 MB,平均约34.8字节每条,再压缩后是1.85 MB,平均11.8字节每条。
B)编码后为700 KB,平均21.9字节每条,再压缩后是245 KB,平均7.67字节每条

quantbox.data's People

Contributors

wukan1986 avatar hthubert avatar fouvy avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.