Comments (8)
Sure, the PyPCAPKit has an interface called pcapkit.foundation.traceflow.TraceFlow
. It will follow each TCP stream and either save each stream into a target file or store them as an attribute of the instance. You can simply specify trace=True
when calling the pcapkit.foundation.extraction.Extrator
constructor. For more information, pls refer to the document.
from pypcapkit.
Please kindly provide the source PCAP file and corresponding JSON output. Supposedly, this should be an issue within the JSON serialiser of DictDumper. I shall look into it ASAP.
Thanks for your cooperation and support to the PyPCAPKit project.
from pypcapkit.
any of these should work, doesn't really matter, you're looking for literal, unescaped hex in the payload, which should be pretty easy to make up
from pypcapkit.
meanwhile, being unsatisfied that dictdumper
absolutely requires a file to write output to (plus your custom implementation is not readily exposed
PyPCAPKit/pcapkit/foundation/extraction.py
Line 667 in 2e21f0a
json
object hook / serializer class:
import json
import datetime
import ipaddress
class json_serialize(json.JSONEncoder):
def encode(self,o):
if isinstance(o,bytes):
return json.encoder.py_encode_basestring(o.hex())
elif isinstance(o,datetime.datetime):
return json.encoder.py_encode_basestring(o.isoformat())
elif isinstance(o,(ipaddress.IPv4Address,ipaddress.IPv6Address)):
return json.encoder.py_encode_basestring(str(o))
elif isinstance(o,dict):
xs = []
for k,v in o.items():
xs.append(":".join([json.encoder.py_encode_basestring(k),self.encode(v)]))
xs = ",".join(xs)
return "{%s}"%xs
elif isinstance(o,list):
"[%s]"%(",".join(self.encode(v) for v in o))
return super().encode(o)
USAGE: print(json.dumps(frame.info.info2dict(),ensure_ascii=False,sort_keys=True,separators=(",",":"),cls=json_serialize))
from pypcapkit.
also, the default
engine doesn't seem to be able to deal with ARP packets (attached)
switching to dpkt
works, but perhaps is there a way to fix it for engine=default
?
I saw ...protocol...arp
is defined in this repository, but I don't know where to modify the engine=default
code
from pypcapkit.
Could you pls confirm that you're using the lastest version of DictDumper, i.e. dictdumper==0.8.4.post2
? The JSON serialiser should've been fixed in that release.
As to the initial of the DictDumper project, it was designed to be a steam output handler to the filesystem, so that we can write a structual file piece by piece, instead of dump -> load -> dump
loop (JSON itself for instance).
You can of course customise your own handler to the output stream in PyPCAPKit, as well customised serialiser based on the DictDumper project. The simpliest way, actually, is just to use the pcapkit.foundation.extraction.Extractor
in auto=False
mode as an iterator, and to handle each extracted packet with your own code.
About the extrator engine, PyPCAPKit supports several different modules, you may just provide your own selection through the CLI and/or API interfaces.
For the ARP parsing issue, please create a pull request on the pcapkit.protocols.link.arp.ARP
class should you have some idea.
Thanks for your cooperation and support to the PyPCAPKit project.
from pypcapkit.
Okay
- confirmed the version of DictDumper is what you stated
$ sudo python3 -B -u -m pip install -U dictdumper
Requirement already up-to-date: dictdumper in /usr/local/lib/python3.8/site-packages (0.8.4.post2)
- confirmed that the dumping is working properly now i.e.
jq
can parse it
$ jq -rc . out.json
{"Global Header":{"magic_number":{"data":{"type":"bytes","value":"�ò�","hex":"d4c3b2a1"},"byteorder":"little","nanosecond":false},"version_major":2,"version_minor":4,"thiszone":0,"sigfigs":0,"snaplen":262144,"network":{"enum":"LinkType","desc":"[LinkType] Link-Layer Header Type Values","name":"ETHERNET","value":1},"packet":{"type":"bytes","value":"�ò�\u0002\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0001\u0000\u0000\u0000","hex":"d4c3b2a10200040000000000000000000000040001000000"}},"Frame 1":{"frame_info":{"ts_sec":1573042247,"ts_usec":848399,"incl_len":60,"orig_len":60},"time":"2019-11-06T20:10:47.848399","number":1,"time_epoch":1573042247.848399,"len":60,"cap_len":60,"packet":{"type":"bytes","value":"\u0000\f)}\u001d�\u0000\f)\u0019�a\b\u0006\u0000\u0001\b\u0000\u0006\u0004\u0000\u0001\u0000\f)\u0019�a\n\u0014\u001e�\u0000\f)}\u001d�\n\u0014\u001e�\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","hex":"000c297d1db4000c2919dc6108060001080006040001000c2919dc610a141e83000c297d1db40a141e82000000000000000000000000000000000000"},"error":"TypeError: expected string or bytes-like object","raw":{"packet":{"type":"bytes","value":"\u0000\f)}\u001d�\u0000\f)\u0019�a\b\u0006\u0000\u0001\b\u0000\u0006\u0004\u0000\u0001\u0000\f)\u0019�a\n\u0014\u001e�\u0000\f)}\u001d�\n\u0014\u001e�\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","hex":"000c297d1db4000c2919dc6108060001080006040001000c2919dc610a141e83000c297d1db40a141e82000000000000000000000000000000000000"},"error":"TypeError: expected string or bytes-like object"},"protocols":"Raw"},"Frame 2":{"frame_info":{"ts_sec":1573042247,"ts_usec":848533,"incl_len":60,"orig_len":60},"time":"2019-11-06T20:10:47.848533","number":2,"time_epoch":1573042247.848533,"len":60,"cap_len":60,"packet":{"type":"bytes","value":"\u0000\f)\u0019�a\u0000\f)}\u001d�\b\u0006\u0000\u0001\b\u0000\u0006\u0004\u0000\u0002\u0000\f)}\u001d�\n\u0014\u001e�\u0000\f)\u0019�a\n\u0014\u001e�\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","hex":"000c2919dc61000c297d1db408060001080006040002000c297d1db40a141e82000c2919dc610a141e83000000000000000000000000000000000000"},"error":"TypeError: expected string or bytes-like object","raw":{"packet":{"type":"bytes","value":"\u0000\f)\u0019�a\u0000\f)}\u001d�\b\u0006\u0000\u0001\b\u0000\u0006\u0004\u0000\u0002\u0000\f)}\u001d�\n\u0014\u001e�\u0000\f)\u0019�a\n\u0014\u001e�\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","hex":"000c2919dc61000c297d1db408060001080006040002000c297d1db40a141e82000c2919dc610a141e83000000000000000000000000000000000000"},"error":"TypeError: expected string or bytes-like object"},"protocols":"Raw"}}
- re:ARP parsing issue, I just switched over to
dpkt
, anddpkt
is able to parse Ethernet/ARP packet successfully
By the way, is PyPCAPKit able to parse TCP streams (just like how Wireshark has the feature to "Follow TCP stream"), instead of frame-by-frame?
from pypcapkit.
Fixed in v0.15.3.
from pypcapkit.
Related Issues (20)
- Can run test_reassembly.py HOT 1
- Error running test_reassembly.py HOT 1
- Add Example to print reassembled byte stream HOT 1
- Dynamic code execution is always an issue and should never be used. HOT 2
- Import of PCAPKit crashes HOT 2
- Import crash related to #113 HOT 2
- Import Error HOT 4
- Unable to run pcapkit.extract HOT 3
- Unable to install from source HOT 1
- Invalid .pyup.yml detected
- Reassembly not working since 0.16.3 HOT 1
- Question: how to process realtime pcap stream HOT 4
- pcapkit.extraction not working HOT 9
- `follow_tcp_stream` (and flow tracing in general) doesn't work HOT 1
- UDP response messages (DNS responses) are skipped when ipv4 reassembly is set to true.
- ipv6 packet with option decodes error
- PyPCAPKit does no longer install correctly HOT 1
- Just a question HOT 5
- ipv4_reassembly HOT 2
- Defragmentaion of IPv4/IPv6 Pakets.. HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pypcapkit.