Git Product home page Git Product logo

iextrading4j-hist's Introduction

IEXTrading4j HIST: Open source IEX Trading TOPS and DEEP library

Build Status Maven Central codecov Quality Gate

Table of Contents

Quick Start

Java SE 8 or greater is required to use IEXTrading4j HIST library.

<dependency>
	<groupId>pl.zankowski</groupId>
	<artifactId>iextrading4j-hist-all</artifactId>
	<version>1.4.0</version>
</dependency>

Library is up to:

  • TOPS 1.6 version 1.66 - October 19, 2021
  • TOPS 1.5 version 1.56 - May 9, 2017
  • DEEP version 1.08 - October 19, 2021

Description

IEX Trading allows users to receive their market data completly for free. Their data can be accessed in real-time during market session or can be downloaded as recorded sessions in PCAP data format. Market Data is transported in binary format and can be read according to specification shared on their site.

This library allows you to transform binary packets into human readable Market Data events in Java.

More on this topic can found here: IEX Trading Market Data

IEX Trading

IEX A Fair, Simple, Transparent Exchange.

IEX is a stock exchange based in the United States. Started by Brad Katsuyama, it opened for trading on October 25, 2013. The company’s offices are located at 4 World Trade Center in New York City. The matching engine is located across the Hudson River in Weehawken, New Jersey, and the initial point of presence is located in a data center in Secaucus, New Jersey. IEX's main innovation is a 38-mile coil of optical fiber placed in front of its trading engine. This 350 microsecond delay adds a round-trip delay of 0.0007 seconds and is designed to negate the certain speed advantages utilized by some high-frequency traders.

IEX was created in response to questionable trading practices that had become widely used across traditional public Wall Street exchanges as well as dark pools and other alternative trading systems. The IEX exchange aims to attract investors by promising to "play fair" by operating in a transparent and straightforward manner, while also helping to level the playing field for traders. Strategies to achieve those goals include:

  • Publishing the matching rules used in the exchanges's computerized order matching engine.
  • Offering a limited number of simple and familiar order types.
  • Charging fixed fees on most orders (or a flat percentage rate on small orders).
  • Ensuring market pricing data arrives at external points of presence simultaneously.
  • Slightly delaying market pricing data to all customers (no colocation).
  • Refusing to pay for order flow.

Check out their beautiful site: IEX Trading

DEEP

DEEP is used to receive real-time depth of book quotations direct from IEX. The depth of book quotations received via DEEP provide an aggregated size of resting displayed orders at a price and side, and do not indicate the size or number of individual orders at any price level. Non-displayed orders and non-displayed portions of reserve orders are not represented in DEEP.

DEEP also provides last trade price and size information. Trades resulting from either displayed or non-displayed orders matching on IEX will be reported. Routed executions will not be reported.

TOPS

TOPS is used to receive real-time top of book quotations direct from IEX. The quotations received via TOPS provide an aggregated size and do not indicate the size or number of individual orders at the best bid or ask. Non-displayed orders and non-displayed portions of reserve orders are not represented in TOPS.

TOPS also provides last trade price and size information. Trades resulting from either displayed or non-displayed orders matching on IEX will be reported. Routed executions will not be reported.

Sample

To run samples it is required to have pcap library (Npcap recommended) installed on computer. Make sure to select checkbox "Install with WinPcap API-compatible mode" if you're using Windows. You will also need to import org.pcap4j:

    <dependency>
      <groupId>org.pcap4j</groupId>
      <artifactId>pcap4j-core</artifactId>
      <version>1.7.5</version>
    </dependency>
    <dependency>
      <groupId>org.pcap4j</groupId>
      <artifactId>pcap4j-packetfactory-static</artifactId>
      <version>1.7.5</version>
    </dependency>

TOPS

private void readTOPSsample() throws PcapNativeException, InterruptedException, NotOpenException {
    PcapHandle handle = Pcaps.openOffline("path_to_pcap", PcapHandle.TimestampPrecision.NANO);

    handle.loop(-1, new PacketListener() {
        @Override
        public void gotPacket(Packet packet) {
            byte[] data = packet.getPayload().getPayload().getPayload().getRawData();
            IEXSegment block = IEXTOPSMessageBlock.createIEXSegment(data);
            System.out.println(block);
        }
    });

    handle.close();
}

DEEP

private void readDEEPsample() throws PcapNativeException, InterruptedException, NotOpenException {
    PcapHandle handle = Pcaps.openOffline("path_to_pcap", PcapHandle.TimestampPrecision.NANO);

    handle.loop(-1, new PacketListener() {
        @Override
        public void gotPacket(Packet packet) {
            byte[] data = packet.getPayload().getPayload().getPayload().getRawData();
            IEXSegment block = IEXDEEPMessageBlock.createIEXSegment(data);
            System.out.println(block);
        }
    });

    handle.close();
}

Roadmap

  • Code coverage
  • Cleanup

License

Code and documentation released under the Apache License, Version 2.0

Data provided for free by IEX.

IEX Trading API Exhibit A: Exhibit A

Donations

To support this repository: bc1qv6cz6t302qvamxlhyr3m7mdwrnc7anqc35wp3t

iextrading4j-hist's People

Contributors

vslee avatar wojciechzankowski avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

iextrading4j-hist's Issues

IEX TOPS Message has to contain 42 bytes

Hi Team,

pl.zankowski.iextrading4j.hist.tops.IEXTOPSMessageBlock is throwing exception

Exception

[main] ERROR org.pcap4j.core.PcapHandle - The executor has thrown an exception.
java.lang.IllegalArgumentException: IEX TOPS Message has to contain 42 bytes

I observed that this error coming due to below condition
if (bytes.length != 42) {
throw new IllegalArgumentException("IEX TOPS Message has to contain 42 bytes");
}

May I know why this condition is added ?
Please advise me how to tackle such exception

thanks,
Vikas

ERROR [org.pcap4j.core.PcapHandle] :: java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi All,

https://api.iextrading.com/1.0/hist?date=20171117

During parsing PCAP data using open source pcap4j library, i am getting Error "java.lang.OutOfMemoryError: GC overhead limit exceeded".
I have attached complete Error Stack trace here as attachment.

My System configuration details :
RAM 8 GB
HardDisk 500 GB

There is no more process running in my system which consumes my hardware resources. but still i am getting same error.

I also restarted system , thinking that it might be intermittent issue but it keeps coming.

Kindly help me on this.

thanks
GCError.txt

packet.getPayload() returns null

import org.pcap4j.core.*;
import org.pcap4j.packet.Packet;
import pl.zankowski.iextrading4j.hist.api.message.IEXSegment;
import pl.zankowski.iextrading4j.hist.deep.IEXDEEPMessageBlock;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;

public class IexDeepReader {
public static File unGzip(File infile, boolean deleteGzipfileOnSuccess) throws IOException {
GZIPInputStream gin = new GZIPInputStream(new FileInputStream(infile));
FileOutputStream fos = null;
try {
File folder = new File("temp");
if (!folder.isDirectory())
folder.mkdirs();

        File outFile = new File(folder, infile.getName().replaceAll("\\.gz$", ""));
        if (outFile.isFile())//already exists
            return outFile;

        fos = new FileOutputStream(outFile);
        byte[] buf = new byte[100000];
        int len;
        while ((len = gin.read(buf)) > 0) {
            fos.write(buf, 0, len);
        }

        fos.close();
        if (deleteGzipfileOnSuccess) {
            infile.delete();
        }
        return outFile;
    } finally {
        if (gin != null) {
            gin.close();
        }
        if (fos != null) {
            fos.close();
        }
    }
}

private void readDEEPsample() throws PcapNativeException, InterruptedException, NotOpenException, IOException {
    String filename = "data%2Ffeeds%2F20180628%2F20180628_IEXTP1_DEEP1.0.pcap.gz";
    PcapHandle handle = Pcaps.openOffline(unGzip(new File(filename), false).getPath(),
                                          PcapHandle.TimestampPrecision.NANO);

    handle.loop(-1, new PacketListener() {
        @Override
        public void gotPacket(Packet packet) {
            byte[] data = packet.getPayload().getPayload().getPayload().getRawData();
            IEXSegment block = IEXDEEPMessageBlock.createIEXSegment(data);
            System.out.println(block);
        }
    });

    handle.close();
}

public static void main(String[] args) {
    IexDeepReader reader = new IexDeepReader();
    try {
        reader.readDEEPsample();
    } catch (PcapNativeException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (NotOpenException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

}

Try to run tops sample

Hi, I tried to run your tops sample but I only get this output:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

I have installed npcap latest version and selected checkbox "Install with WinPcap API-compatible mode"

Can you support me?

Thanks a lot.

DeepSample program is throwing error org.pcap4j.core.PcapNativeException

data%2Ffeeds%2F20171104%2F20171104_IEXTP1_DEEP1.0.pcap.gz
Hi All,

Thanks for your support. I am looking for one small guidance/help.

I was trying to read Sample(PCAP format) file using below libraries and observed that it throws exception like below.
I was using api readDEEPsample() of DeepSample.java
iextrading4j-hist/iextrading4j-hist-samples/src/main/java/pl/zankowski/iextrading4j/hist/samples/DEEPSample.java

Exception:

org.pcap4j.core.PcapNativeException: pcap_open_offline_with_tstamp_precision is not supported by the pcap library installed in this environment.
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:234)

Libraries used :

iextrading4j-hist-api-1.0.0.jar
iextrading4j-hist-deep-1.0.0.jar
iextrading4j-hist-tops-1.0.0.jar
pcap4j-core-1.7.2.jar
pcap4j-packetfactory-static-1.7.2.jar

Sample pcap file :
data%2Ffeeds%2F20171104%2F20171104_IEXTP1_DEEP1.0.pcap.gz file which contains pcap file

Additional Info

Pcaps.libVersion WinPcap version 4.1.3 (packet.dll version 4.1.0.2980), based on libpcap version 1.0 branch 1_0_rel0b (20091008)

May I request you please to help me on this.

thanks
Vikas

org.pcap4j.core.PcapNativeException: path_to_pcap: No such file or directory

Hi,
Thanks for the project. I try to get it run.

I just installed it and after some problems with the Npcap I got the output:
13:49:17.765 [main] INFO org.pcap4j.core.NativeMappings - Pcap4J successfully loaded a native pcap library: Npcap version 0.99-r7, based on libpcap version 1.8.1
Exception in thread "main" org.pcap4j.core.PcapNativeException: path_to_pcap: No such file or directory
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:241)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.readTOPSsample(TOPSSample.java:21)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.main(TOPSSample.java:17)

Then I configured the parameter to "C:\Program Files (x86)\Nmap" and then I got
Exception in thread "main" org.pcap4j.core.PcapNativeException: C:\Program Files (x86)\Nmap: Permission denied
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:241)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.readTOPSsample(TOPSSample.java:20)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.main(TOPSSample.java:16)

I set the Nmap directory to allow access, but this I get this exception.

Thanks for help

Issue : Some of Deeps/Tops History data is failed to read

Hi All,

Thanks for your support.

During parsing Deeps/Tops History data , There are mixed of a reading response. Some of data has been parsed and some of them got failed.
[](I have attached complete response of execution of Tops pcap file
TopsResponse.docx)

. I have highlighted error red color.

I am referring to DeepSample/TopsSample code from https://github.com/WojciechZankowski/iextrading4j-hist

I have tried this tops data(in gz format) attached here and got an issue.
data%2Ffeeds%2F20171104%2F20171104_IEXTP1_TOPS1.6.pcap.gz

May I request you please to help me on this. Please guide me how to handle such error.

Do i need to write our own wrapper class on open source library to tackle such error ?

thanks,
Vikas

Cannot parse IEXSaleConditionFlag

Describe the bug
Cannot parse IEXSaleConditionFlag.

To Reproduce
Steps to reproduce the behavior:

  1. Try to parse TOPS data from 2018-06-22 using library
  2. There is lots of erros saying that IEXSaleConditionFlag cannot be parse: -112

Expected behavior
Should parse whole file without erros

Additional context
TOPS data 2018-06-22

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.