Git Product home page Git Product logo

trackme's Introduction

TrackMe - Server side http/tls tracking demo in go

TrackMe is a custom, low-level http/1 and h2 server, that responds with the fine details about the request made.

It returns the ja3, akamai h2 fingerprint, header + header order, h2 frames, and much more.

Generating the certificates and config

You first need to generate the certificate.pem and the key.pem files.

$ mkdir certs
$ openssl req -x509 -newkey rsa:4096 -keyout certs/key.pem -out certs/chain.pem -sha256 -days 365 -nodes

Then, you need to copy the example config (and maybe edit it)

$ cp config.example.json config.json
$ nano config.json
...

Running it (Docker)

$ docker build -t "trackme:Dockerfile" .
$ docker run -p 80:80 -p 443:443 "trackme:Dockerfile"

Running it (Without Docker)

You can build a binary by running go build -o TrackMe *.go

After that, just run the binary (sudo ./TrackMe)

Different fingerprints

The site returns 3 different fingerprints: the JA3, a TLS fingerprint, an HTTP/2 "akamai-fingerprint" (Only works on HTTP/2 connections) and my own custom "PeetPrint".

Custom Fingerpint ("PeetPrint")

I wanted to extend JA3, so I created my own TLS fingerprint algorithm. It's better suited for fingerprinting TLS1.3 connections, because JA3 doesn't really do that well, and has more datapoints. The designed is inspired by the http/2 fingerprint proposed by akamai.

It looks like this:

supported-tls-versions|supported-protocols|supported-groups|supported-signature-algorithms|psk-key-exchange-mode|certificate-compression-algorithms|cipher-suites|sorted-extensions

"-" is used as the seperator.

supported-tls-versions: Seperated list of supported TLS versions as sent in the supported_versions extension.

supported-protocols: Seperated list of supported HTTP versions as sent in the application_layer_protocol_negotiation extension. http/1.0 => 1.0, http/1.1 => 1.1, http/2 => 2

supported-groups: Seperated list of supported elliptic curve groups as sent in the supported_groups extension.

supported-signature-algorithms: Seperated list of supported signatue algorithms as sent in the signature_algorithms extension.

psk-key-exchange-mode The PSK key exchange mode as specified in the psk_key_exchange_modes extension. Usually 0 or 1.

certificate-compression-algorithms Seperated list of the certificate compression algorithms as sent in the compress_certificate extension.

cipher-suites: Seperated list of the supported cipher suites.

sorted-extensions: Sorted list of the supported extensions. (Sorted because of order randomization used by chrome)

All TLS GREASE values must be replaced with "GREASE".

That means, a fingerprint could look something like this:

GREASE-772-771|2-1.1|GREASE-29-23-24|1027-2052-1025-1283-2053-1281-2054-1537|1|2|GREASE-4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53|GREASE-0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-GREASE-21-41

API endpoints

The site exposes a lot of different API endpoints.

/api/all

Returns all of the collected data about an request

/api/tls

Returns only the TLS data

/api/clean

Returns only the different fingerprints (akamai-fp+ja3)

/api/request-count

Returns the total request count the database captured. Only works when connected to a database.

/api/search-ja3

Param: ?by=<ja3>

Returns the most seen other identifiers (user-agent, h2, peetprint) that were seen together with this identifier. Only works when connected to a database.

/api/search-h2

Param: ?by=<akamai-fp>

Returns the most seen other identifiers (user-agent, JA3, peetprint) that were seen together with this identifier. Only works when connected to a database.

/api/search-peetprint

Param: ?by=<peetprint>

Returns the most seen other identifiers (user-agent, h2, JA3) that were seen together with this identifier. Only works when connected to a database.

Docker

You can also run the server in a docker container using docker-compose.

# generate certs and update your config.json
docker-compose -up --build
# visit https://localhost/api/all

TLS & HTTP2 fingerprinting resources

trackme's People

Contributors

awaitresponse avatar bogdanfinn avatar hellodword avatar muslashwhy avatar saucesteals avatar venomous avatar wwhtrbbtt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

trackme's Issues

CORS support

hi I have been trying to add CORS to this project and just can't get it to work.. any idea how I could do that ?
update:
I added the following line to the connection_handler.go file in the HandleHTTP2 method:
encoder.WriteField(hpack.HeaderField{Name: "Access-Control-Allow-Origin", Value: "*"})
It solved the CORS problem but I get the following error: ERR_HTTP2_PROTOCOL_ERROR

Requester IP address in MongoDB

Logging the IP & timestamp of a request in a separate mongodb object w/ a "foreign key" like reference to the fingerprint object may be helpful.

[Bug] Unsound hack in JA4 c-part

TrackMe/ja4.go

Lines 82 to 85 in d9579a8

// VERY dirty hack: append padding, because it gets filtered out for JA3
// but we want it here. _SHOULD_ be included in ever TLS clienthello, so
// _shouldnt_ cause any issues.
extensions = append(extensions, "21")
is under an unsound assumption that every TLS clienthello will include a padding extension. I don't see any standard that requires padding extension as a mandatory component. In fact, some implementations, e.g. curl on my machine, do NOT have padding in the extensions.

eth0: No such device exists (SIOCGIFHWADDR: No such device)

When running the server on an EC2 machine, the following error occurs:
eth0: No such device exists (SIOCGIFHWADDR: No such device)

Reason is that in tcp.go file the device name is hardcoded to "eth0", whereas the default device name on an AWS EC2 machine is different (e.g. "ens5").

Suggested fix: add the device name to config.json.
(of course re-compilation with correct device name is also an option...)

Thanks!

goroutine leak in parseHTTP2

Hi
I found that the memory consumption of the app is growing with each request. My knowledge of Go is very basic, so I couldn't solve it.

	for {
		fmt.Println("NumGoroutine:", runtime.NumGoroutine())
		conn, err := listener.Accept()

I added a counter in the main loop and observed that the NumGoroutine count increases with each request without decreasing. This issue occurs only with HTTP/2 requests, so I suspect the problem lies with the "go parseHTTP2" frame-reader in the infinite "for" loop.

To address this, I tried sending a signal to close the loop, which seemed to resolve the issue initially (code below). The https://localhost/api/all endpoint now opens in curl and Chrome without increasing the goroutine count, but it doesn't open in Firefox. I think the high number of PRIORITY frames might be causing some issue in Firefox. I have tried a few other similar methods to terminate this function with goroutine, such as using channels and timeouts, but unfortunately, I'm stuck on it.

connection_handler.go

import (
	"bytes"
	"fmt"
	"log"
	"net"
	"strconv"
	"strings"
	"time"

	tls "github.com/wwhtrbbtt/utls"
	"golang.org/x/net/http2"
	"golang.org/x/net/http2/hpack"
)

const HTTP2_PREAMBLE = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"

func parseHTTP1(request []byte) Response {
	// Split the request into lines
	lines := strings.Split(string(request), "\r\n")

	// Split the first line into the method, path and http version
	firstLine := strings.Split(lines[0], " ")

	// Split the headers into an array
	var headers []string
	for _, line := range lines {
		if strings.Contains(line, ":") {
			headers = append(headers, line)
		}
	}

	if len(firstLine) != 3 {
		return Response{
			HTTPVersion: "--",
			Method:      "--",
			path:        "--",
		}
	}
	return Response{
		HTTPVersion: firstLine[2],
		path:        firstLine[1],
		Method:      firstLine[0],
		Http1: &Http1Details{
			Headers: headers,
		},
	}
}

func parseHTTP2(f *http2.Framer, c chan ParsedFrame, quit chan struct{}) {
	for {
		frame, err := f.ReadFrame()
		if err != nil {
			r := "ERROR_CLOSE"
			if strings.HasSuffix(err.Error(), "unknown certificate") {
				r = "ERROR"
			}
			// log.Println("Error reading frame", err, r)
			c <- ParsedFrame{Type: r}
			return
		}

		select {
		case <-quit:
			fmt.Println("parseHTTP2 quit")
			return
		default:
			p := ParsedFrame{}
			p.Type = frame.Header().Type.String()
			p.Stream = frame.Header().StreamID
			p.Length = frame.Header().Length
			p.Flags = GetAllFlags(frame)

			switch frame := frame.(type) {
			case *http2.SettingsFrame:
				p.Settings = []string{}
				frame.ForeachSetting(func(s http2.Setting) error {
					setting := fmt.Sprintf("%q", s)
					setting = strings.Replace(setting, "\"", "", -1)
					setting = strings.Replace(setting, "[", "", -1)
					setting = strings.Replace(setting, "]", "", -1)

					p.Settings = append(p.Settings, setting)
					return nil
				})
			case *http2.HeadersFrame:
				d := hpack.NewDecoder(4096, func(hf hpack.HeaderField) {})
				d.SetEmitEnabled(true)
				h2Headers, err := d.DecodeFull(frame.HeaderBlockFragment())
				if err != nil {
					//log.Println("Error decoding headers", err)
					return
				}

				for _, h := range h2Headers {
					h := fmt.Sprintf("%q: %q", h.Name, h.Value)
					h = strings.Trim(h, "\"")
					h = strings.Replace(h, "\": \"", ": ", -1)
					p.Headers = append(p.Headers, h)
				}
				if frame.HasPriority() {
					prio := Priority{}
					p.Priority = &prio
					// 6.2: Weight: An 8-bit weight for the stream; Add one to the value to obtain a weight between 1 and 256
					p.Priority.Weight = int(frame.Priority.Weight) + 1
					p.Priority.DependsOn = int(frame.Priority.StreamDep)
					if frame.Priority.Exclusive {
						p.Priority.Exclusive = 1
					}
				}
			case *http2.DataFrame:
				p.Payload = frame.Data()
			case *http2.WindowUpdateFrame:
				p.Increment = frame.Increment
			case *http2.PriorityFrame:

				prio := Priority{}
				p.Priority = &prio
				// 6.3: Weight: An 8-bit weight for the stream; Add one to the value to obtain a weight between 1 and 256
				p.Priority.Weight = int(frame.PriorityParam.Weight) + 1
				p.Priority.DependsOn = int(frame.PriorityParam.StreamDep)
				if frame.PriorityParam.Exclusive {
					p.Priority.Exclusive = 1
				}
			case *http2.GoAwayFrame:
				p.GoAway = &GoAway{}
				p.GoAway.LastStreamID = frame.LastStreamID
				p.GoAway.ErrCode = uint32(frame.ErrCode)
				p.GoAway.DebugData = frame.DebugData()
			}

			c <- p
		}
	}
}

func HandleTLSConnection(conn net.Conn) bool {
	// Read the first line of the request
	// We only read the first line to determine if the connection is HTTP1 or HTTP2
	// If we know that it isnt HTTP2, we can read the rest of the request and then start processing it
	// If we know that it is HTTP2, we start the HTTP2 handler

	l := len([]byte(HTTP2_PREAMBLE))
	request := make([]byte, l)

	_, err := conn.Read(request)
	if err != nil {
		//log.Println("Error reading request", err)
		if strings.HasSuffix(err.Error(), "unknown certificate") && local {
			log.Println("Local error (probably developement) - not closing conn")
			return true
		}
		return false
	}

	hs := conn.(*tls.Conn).ClientHello

	parsedClientHello := ParseClientHello(hs)
	JA3Data := CalculateJA3(parsedClientHello)
	peetfp, peetprintHash := CalculatePeetPrint(parsedClientHello, JA3Data)
	tlsDetails := TLSDetails{
		Ciphers:          JA3Data.ReadableCiphers,
		Extensions:       parsedClientHello.Extensions,
		RecordVersion:    JA3Data.Version,
		NegotiatedVesion: fmt.Sprintf("%v", conn.(*tls.Conn).ConnectionState().Version),
		JA3:              JA3Data.JA3,
		JA3Hash:          JA3Data.JA3Hash,
		PeetPrint:        peetfp,
		PeetPrintHash:    peetprintHash,
		SessionID:        parsedClientHello.SessionID,
		ClientRandom:     parsedClientHello.ClientRandom,
	}

	// Check if the first line is HTTP/2
	if string(request) == HTTP2_PREAMBLE {
		handleHTTP2(conn, tlsDetails)
	} else {
		// Read the rest of the request
		r2 := make([]byte, 1024-l)
		_, err := conn.Read(r2)
		if err != nil {
			log.Println(err)
			return true
		}
		// Append it to the first line
		request = append(request, r2...)

		// Parse and handle the request
		details := parseHTTP1(request)
		details.IP = conn.RemoteAddr().String()
		details.TLS = tlsDetails
		respondToHTTP1(conn, details)
	}
	return true
}

func respondToHTTP1(conn net.Conn, resp Response) {
	// log.Println("Request:", resp.ToJson())
	// log.Println(len(resp.ToJson()))

	res1, ctype := Router(resp.path, resp)

	res := "HTTP/1.1 200 OK\r\n"
	res += "Content-Length: " + fmt.Sprintf("%v\r\n", len(res1))
	res += "Content-Type: " + ctype + "; charset=utf-8\r\n"
	res += "Server: TrackMe\r\n"
	res += "\r\n"
	res += string(res1)
	res += "\r\n\r\n"

	_, err := conn.Write([]byte(res))
	if err != nil {
		log.Println("Error writing HTTP/1 data", err)
		return
	}
	err = conn.Close()
	if err != nil {
		log.Println("Error closing HTTP/1 connection", err)
		return
	}
}

// https://stackoverflow.com/questions/52002623/golang-tcp-server-how-to-write-http2-data
func handleHTTP2(conn net.Conn, tlsFingerprint TLSDetails) {
	// make a new framer to encode/decode frames
	fr := http2.NewFramer(conn, conn)
	c := make(chan ParsedFrame)
	var frames []ParsedFrame

	// Same settings that google uses
	err := fr.WriteSettings(
		http2.Setting{
			ID: http2.SettingInitialWindowSize, Val: 1048576,
		},
		http2.Setting{
			ID: http2.SettingMaxConcurrentStreams, Val: 100,
		},
		http2.Setting{
			ID: http2.SettingMaxHeaderListSize, Val: 65536,
		},
	)
	if err != nil {
		log.Println(err)
		return
	}

	var frame ParsedFrame
	var headerFrame ParsedFrame

	quit := make(chan struct{})
	go parseHTTP2(fr, c, quit)

	for {
		frame = <-c
		if frame.Type == "ERROR_CLOSE" {
			err = conn.Close()
			if err != nil {
				log.Println("Cant close connection", err)
			}
			return
		} else if frame.Type == "ERROR" {
			return
		}
		// log.Println(frame)
		frames = append(frames, frame)
		if frame.Type == "HEADERS" {
			headerFrame = frame
		}
		if len(frame.Flags) > 0 && frame.Flags[0] == "EndStream (0x1)" {
			quit <- struct{}{}
			break
		}
	}

	// get method, path and user-agent from the header frame
	var path string
	var method string
	var userAgent string

	for _, h := range headerFrame.Headers {
		if strings.HasPrefix(h, ":method") {
			method = strings.Split(h, ": ")[1]
		}
		if strings.HasPrefix(h, ":path") {
			path = strings.Split(h, ": ")[1]
		}
		if strings.HasPrefix(h, "user-agent") {
			userAgent = strings.Split(h, ": ")[1]
		}
	}

	resp := Response{
		IP:          conn.RemoteAddr().String(),
		HTTPVersion: "h2",
		path:        path,
		Method:      method,
		UserAgent:   userAgent,
		Http2: &Http2Details{
			SendFrames:            frames,
			AkamaiFingerprint:     GetAkamaiFingerprint(frames),
			AkamaiFingerprintHash: GetMD5Hash(GetAkamaiFingerprint(frames)),
		},
		TLS: tlsFingerprint,
	}

	res, ctype := Router(path, resp)

	// Prepare HEADERS
	hbuf := bytes.NewBuffer([]byte{})
	encoder := hpack.NewEncoder(hbuf)
	encoder.WriteField(hpack.HeaderField{Name: ":status", Value: "200"})
	encoder.WriteField(hpack.HeaderField{Name: "server", Value: "TrackMe.peet.ws"})
	encoder.WriteField(hpack.HeaderField{Name: "content-length", Value: strconv.Itoa(len(res))})
	encoder.WriteField(hpack.HeaderField{Name: "content-type", Value: ctype})

	// Write HEADERS frame
	err = fr.WriteHeaders(http2.HeadersFrameParam{StreamID: headerFrame.Stream, BlockFragment: hbuf.Bytes(), EndHeaders: true})
	if err != nil {
		log.Println("could not write headers: ", err)
		return
	}

	chunks := splitBytesIntoChunks(res, 1024)
	for _, c := range chunks {
		fr.WriteData(headerFrame.Stream, false, c)
	}
	fr.WriteData(headerFrame.Stream, true, []byte{})
	fr.WriteGoAway(headerFrame.Stream, http2.ErrCodeNo, []byte{})

	time.Sleep(time.Millisecond * 500)
	conn.Close()
}

GREASE status in MongoDB

I think it would be helpful to add a boolean for GREASE support in the MongoDB since it's not in the JA3 hash. GREASE is helpful in identifying bots or malicious requests spoofing user_agents.

EX: Reported user_agent is a current version of Safari on any OS. But, there is no grease support in the TLS negotiation. Therefore the conclusion is the user_agent is being faked.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.