Describe the bug
Cannot parse IEXSaleConditionFlag.

To Reproduce
Steps to reproduce the behavior:

1. Try to parse TOPS data from 2018-06-22 using library
2. There is lots of erros saying that IEXSaleConditionFlag cannot be parse: -112

Expected behavior
Should parse whole file without erros

Additional context
TOPS data 2018-06-22

Hi Team,

pl.zankowski.iextrading4j.hist.tops.IEXTOPSMessageBlock is throwing exception

Exception

[main] ERROR org.pcap4j.core.PcapHandle - The executor has thrown an exception.
java.lang.IllegalArgumentException: IEX TOPS Message has to contain 42 bytes

I observed that this error coming due to below condition
if (bytes.length != 42) {
throw new IllegalArgumentException("IEX TOPS Message has to contain 42 bytes");
}

May I know why this condition is added ?
Please advise me how to tackle such exception

thanks,
Vikas

Hi,
Thanks for the project. I try to get it run.

I just installed it and after some problems with the Npcap I got the output:
13:49:17.765 [main] INFO org.pcap4j.core.NativeMappings - Pcap4J successfully loaded a native pcap library: Npcap version 0.99-r7, based on libpcap version 1.8.1
Exception in thread "main" org.pcap4j.core.PcapNativeException: path_to_pcap: No such file or directory
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:241)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.readTOPSsample(TOPSSample.java:21)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.main(TOPSSample.java:17)

Then I configured the parameter to "C:\Program Files (x86)\Nmap" and then I got
Exception in thread "main" org.pcap4j.core.PcapNativeException: C:\Program Files (x86)\Nmap: Permission denied
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:241)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.readTOPSsample(TOPSSample.java:20)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.main(TOPSSample.java:16)

I set the Nmap directory to allow access, but this I get this exception.

Thanks for help

data%2Ffeeds%2F20171104%2F20171104_IEXTP1_DEEP1.0.pcap.gz
Hi All,

Thanks for your support. I am looking for one small guidance/help.

I was trying to read Sample(PCAP format) file using below libraries and observed that it throws exception like below.
I was using api readDEEPsample() of DeepSample.java
iextrading4j-hist/iextrading4j-hist-samples/src/main/java/pl/zankowski/iextrading4j/hist/samples/DEEPSample.java

Exception:

org.pcap4j.core.PcapNativeException: pcap_open_offline_with_tstamp_precision is not supported by the pcap library installed in this environment.
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:234)

Libraries used :

iextrading4j-hist-api-1.0.0.jar
iextrading4j-hist-deep-1.0.0.jar
iextrading4j-hist-tops-1.0.0.jar
pcap4j-core-1.7.2.jar
pcap4j-packetfactory-static-1.7.2.jar

Sample pcap file :
data%2Ffeeds%2F20171104%2F20171104_IEXTP1_DEEP1.0.pcap.gz file which contains pcap file

Additional Info

Pcaps.libVersion WinPcap version 4.1.3 (packet.dll version 4.1.0.2980), based on libpcap version 1.0 branch 1_0_rel0b (20091008)

May I request you please to help me on this.

thanks
Vikas

Hi All,

Thanks for your support.

During parsing Deeps/Tops History data , There are mixed of a reading response. Some of data has been parsed and some of them got failed.
[](I have attached complete response of execution of Tops pcap file
TopsResponse.docx)
. I have highlighted error red color.

I am referring to DeepSample/TopsSample code from https://github.com/WojciechZankowski/iextrading4j-hist

I have tried this tops data(in gz format) attached here and got an issue.
data%2Ffeeds%2F20171104%2F20171104_IEXTP1_TOPS1.6.pcap.gz

May I request you please to help me on this. Please guide me how to handle such error.

Do i need to write our own wrapper class on open source library to tackle such error ?

thanks,
Vikas

Hi All,

https://api.iextrading.com/1.0/hist?date=20171117

During parsing PCAP data using open source pcap4j library, i am getting Error "java.lang.OutOfMemoryError: GC overhead limit exceeded".
I have attached complete Error Stack trace here as attachment.

My System configuration details :
RAM 8 GB
HardDisk 500 GB

There is no more process running in my system which consumes my hardware resources. but still i am getting same error.

I also restarted system , thinking that it might be intermittent issue but it keeps coming.

Kindly help me on this.

thanks
GCError.txt

Hi, I tried to run your tops sample but I only get this output:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

I have installed npcap latest version and selected checkbox "Install with WinPcap API-compatible mode"

Can you support me?

Thanks a lot.

import org.pcap4j.core.*;
import org.pcap4j.packet.Packet;
import pl.zankowski.iextrading4j.hist.api.message.IEXSegment;
import pl.zankowski.iextrading4j.hist.deep.IEXDEEPMessageBlock;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;

public class IexDeepReader {
public static File unGzip(File infile, boolean deleteGzipfileOnSuccess) throws IOException {
GZIPInputStream gin = new GZIPInputStream(new FileInputStream(infile));
FileOutputStream fos = null;
try {
File folder = new File("temp");
if (!folder.isDirectory())
folder.mkdirs();

        File outFile = new File(folder, infile.getName().replaceAll("\\.gz$", ""));
        if (outFile.isFile())//already exists
            return outFile;

        fos = new FileOutputStream(outFile);
        byte[] buf = new byte[100000];
        int len;
        while ((len = gin.read(buf)) > 0) {
            fos.write(buf, 0, len);
        }

        fos.close();
        if (deleteGzipfileOnSuccess) {
            infile.delete();
        }
        return outFile;
    } finally {
        if (gin != null) {
            gin.close();
        }
        if (fos != null) {
            fos.close();
        }
    }
}

private void readDEEPsample() throws PcapNativeException, InterruptedException, NotOpenException, IOException {
    String filename = "data%2Ffeeds%2F20180628%2F20180628_IEXTP1_DEEP1.0.pcap.gz";
    PcapHandle handle = Pcaps.openOffline(unGzip(new File(filename), false).getPath(),
                                          PcapHandle.TimestampPrecision.NANO);

    handle.loop(-1, new PacketListener() {
        @Override
        public void gotPacket(Packet packet) {
            byte[] data = packet.getPayload().getPayload().getPayload().getRawData();
            IEXSegment block = IEXDEEPMessageBlock.createIEXSegment(data);
            System.out.println(block);
        }
    });

    handle.close();
}

public static void main(String[] args) {
    IexDeepReader reader = new IexDeepReader();
    try {
        reader.readDEEPsample();
    } catch (PcapNativeException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (NotOpenException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

}