Git Product home page Git Product logo

fifi's Introduction

fifi

fifi sends to a given list of url's HTTP requests, calculates on each response a signature and groups them based on the values.

Responses with the same signature may indicate similar implementation pattern, technologies and homogenious data processing.

Background

Recently, spring boot had a wide spreaded RCE vulnerability, known as Spring4Shell (CVE-2022-22965). Due to the fact that modern web application are implemented based on micro service pattern, various paths of a domain may end up on different applications/containers/CDN. To limit the attack surface system administrator, DevOps Engineers and SRE's are highly interested in limiting the available information about a service in the public.

This tool provides help to identify differences in the response headers from a given list of urls.

Installation

Ether download it from the release page or complie it by yourself:

go install github.com/NodyHub/fifi@latest

Usage and example output

[~/git/fifi]% fifi -h
usage: fifi [files]
fifi sends to a given list of url's HTTP requests, calculates on each response a signature and groups them based on the values.

Default reads from stdin

Options:
--------
[files] provide the urls in files.
  -H string
    	Host
  -X string
    	Method (default "GET")
  -a string
    	Authorization
  -c string
    	Cookie
  -diff string
    	Signature diff with json file from previous run
  -json
    	Output json
  -m int
    	Maximum retries for request (default 3)
  -r	Include HTTP response code in signature calculation
  -s	Include 'Server' response header in signature calculation
  -t int
    	Threads (default 1)
  -u string
    	User-Agent (default GoLang default)
  -v	Verbose output
  -w int
    	Wait ms between requests
  -x int
    	Timeout seconds (default 1)

github.com/NodyHub/[email protected]
[~/git/fifi]% cat ~/uber.url.lst | fifi -v -t 4 -s
2022/05/04 10:58:04 reading from stdin...
2022/05/04 10:58:04 Collected 11 different urls, starting analysis
2022/05/04 10:58:04 parsedArgs.ParallelRequests: 4
2022/05/04 10:58:04 Thread 3 starts
2022/05/04 10:58:04 Thread 2 starts
2022/05/04 10:58:04 Thread 1 starts
2022/05/04 10:58:04 Thread 0 starts
2022/05/04 10:58:04 1705792451 https://auth.uber.com/login/?next_url=https%3A%2F%2Fm.uber.com%2F&privileged_op_url=https%3A%2F%2Fm.uber.com%2F
2022/05/04 10:58:05 1705792451 https://auth.uber.com/login
2022/05/04 10:58:05 1705792451 https://auth.uber.com/login/social/?from=facebook&state=%7B%22query%22%3A%22%3Fnext_url%3Dhttps%253A%252F%252Fm.uber.com%252F%26privileged_op_url%3Dhttps%253A%252F%252Fm.uber.com%252F%26uber_client_name%3Dm2%22%2C%22csrfToken%22%3A%221650443852-01-FNOsAwdU4I8HWkiFZuimbrTHjauX146ik_Hq9h7k1Ew%22%2C%22app%22%3A%22%22%7D&response_type=token
2022/05/04 10:58:05 1705792451 https://auth.uber.com/login/?breeze_local_zone=dca11&next_url=https%3A%2F%2Fm.uber.com%2F&state=NUUybaiHU9SIaKz56QjyvtJTz5CJC25zhhyocPV9guM%3D
2022/05/04 10:58:05 1705792451 https://auth.uber.com/login/
2022/05/04 10:58:05 1705792451 https://auth.uber.com/login/session
2022/05/04 10:58:05 Thread 2 finished
2022/05/04 10:58:05 1705792451 https://auth.uber.com/login/?breeze_local_zone=dca1&state=0A-OdN1vuv_FDbpofRZqJg9maKASCY4k0kCRVEiSDGw%3D&uber_client_name=riderSignUp&uclick_id=840a8ddd-ac10-47e6-aec4-e492968acc42
2022/05/04 10:58:05 Thread 1 finished
2022/05/04 10:58:05 ERROR (0): Get "https://auth.uber.com/login/social": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022/05/04 10:58:06 ERROR (0): Get "https://auth.uber.com/login/social/?next_url=https%3A%2F%2Fm.uber.com%2F&privileged_op_url=https%3A%2F%2Fm.uber.com%2F&uber_client_name=m2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022/05/04 10:58:06 ERROR (1): Get "https://auth.uber.com/login/social": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022/05/04 10:58:07 ERROR (1): Get "https://auth.uber.com/login/social/?next_url=https%3A%2F%2Fm.uber.com%2F&privileged_op_url=https%3A%2F%2Fm.uber.com%2F&uber_client_name=m2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022/05/04 10:58:08 ERROR (2): Get "https://auth.uber.com/login/social": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022/05/04 10:58:09 ERROR (2): Get "https://auth.uber.com/login/social/?next_url=https%3A%2F%2Fm.uber.com%2F&privileged_op_url=https%3A%2F%2Fm.uber.com%2F&uber_client_name=m2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022/05/04 10:58:11 ERROR: maxRetry(3) reached, go to next url
2022/05/04 10:58:11 2898507639 https://auth.uber.com/login/social/
2022/05/04 10:58:11 1667219945 https://auth.uber.com/
2022/05/04 10:58:11 Thread 0 finished
2022/05/04 10:58:12 ERROR: maxRetry(3) reached, go to next url
2022/05/04 10:58:12 Thread 3 finished

Summary:
===================================
Headers received in every response:
===================================
 - X-Frame-Options
 - X-Xss-Protection
 - Alt-Svc
 - Date
 - Server
 - Strict-Transport-Security
 - Cache-Control
 - Vary
 - X-Content-Type-Options
 - Content-Type
 - Via
 - X-Envoy-Upstream-Service-Time
 - X-Uber-Edge
===================================

-----------------------------------
Signature: 1667219945 ; URLs: 1
Additional headers:
 - Server: ufe

Urls:
[404] https://auth.uber.com/
-----------------------------------

-----------------------------------
Signature: 1705792451 ; URLs: 7
Additional headers:
 - Content-Security-Policy
 - Etag
 - Server: ufe
 - Set-Cookie
 - Set-Cookie
 - Timing-Allow-Origin
 - X-Content-Security-Policy
 - X-Csrf-Token
 - X-Webkit-Csp

Urls:
[200] https://auth.uber.com/login
[200] https://auth.uber.com/login/
[200] https://auth.uber.com/login/?breeze_local_zone=dca1&state=0A-OdN1vuv_FDbpofRZqJg9maKASCY4k0kCRVEiSDGw%3D&uber_client_name=riderSignUp&uclick_id=840a8ddd-ac10-47e6-aec4-e492968acc42
[200] https://auth.uber.com/login/?breeze_local_zone=dca11&next_url=https%3A%2F%2Fm.uber.com%2F&state=NUUybaiHU9SIaKz56QjyvtJTz5CJC25zhhyocPV9guM%3D
[200] https://auth.uber.com/login/?next_url=https%3A%2F%2Fm.uber.com%2F&privileged_op_url=https%3A%2F%2Fm.uber.com%2F
[200] https://auth.uber.com/login/session
[200] https://auth.uber.com/login/social/?from=facebook&state=%7B%22query%22%3A%22%3Fnext_url%3Dhttps%253A%252F%252Fm.uber.com%252F%26privileged_op_url%3Dhttps%253A%252F%252Fm.uber.com%252F%26uber_client_name%3Dm2%22%2C%22csrfToken%22%3A%221650443852-01-FNOsAwdU4I8HWkiFZuimbrTHjauX146ik_Hq9h7k1Ew%22%2C%22app%22%3A%22%22%7D&response_type=token
-----------------------------------

-----------------------------------
Signature: 2898507639 ; URLs: 1
Additional headers:
 - Content-Security-Policy
 - Etag
 - Server: ufe
 - Set-Cookie
 - Set-Cookie
 - X-Content-Security-Policy
 - X-Csrf-Token
 - X-Webkit-Csp

Urls:
[404] https://auth.uber.com/login/social/
-----------------------------------

Application behaviour

graph TD
    A[User] -->|all url's| B(fifi)
    B --> C{For all url's}
    C --> D[Send HTTP request]
    D --> E[Calculate signature]
    E --> F[Store response, based on signature]
    F --> C
    C --> G[Generate output]
Loading

Similar or related projects

fifi's People

Contributors

cugu avatar nodyhub avatar

Stargazers

avatar

Watchers

avatar avatar avatar avatar avatar

fifi's Issues

Do not use the canonical form of the header's to avoid information loss

According to the documentation go does automatically generate the canonical form of the response header. That lead to information loss and, fifi might miss differences based on the case.

The raw response header should be used here:

fifi/main.go

Lines 61 to 72 in aca1bb0

for _, row := range strings.Split(string(rawResponse), "\n") {
if len(row) > 0 {
field := strings.TrimSpace(strings.Split(row, ":")[0])
for h, v := range response.Header {
if strings.EqualFold(field, h) {
idx := hdrCounter[h]
res = append(res, headerEntry{h, v[idx]})
hdrCounter[h] = idx + 1
}
}
}
}

Make fifi multithreaded

HTTP calls should be performed in a parallel manner, making the tool much quicker. Maybe a worker pool would be sufficient for this purpose.

Failing error handling

Thats a bug!

022/05/04 06:55:13 ERROR (0): Get "https://<snip>/sales-accelerator/oidc/logoutcallback": dial tcp: lookup <snip> on 1.1.1.1:53: no such host
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x634924]

goroutine 7 [running]:
main.performRequest(0xc00007ea00, {0xc000b37950, 0x48})
	/home/runner/work/fifi/fifi/main.go:210 +0x544
main.getAllSignatures.func1(0x0?, 0xc00007ea00, 0x0?, 0xc00000c0a8, 0x0)
	/home/runner/work/fifi/fifi/main.go:291 +0x192
created by main.getAllSignatures
	/home/runner/work/fifi/fifi/main.go:283 +0x37d

Preserver response header order

Golang stores currently the response header in a map[string][]string which makes it impossible to reverse the initial order of the header values.

The order of the header values may indicate even further information about differences.

Handle HTTP client timeouts

When the client timeout is reached, it throws an error and exists.
Timeout errors should be caught and properly handled.

022/04/13 14:13:42 Get "https://[REDACTED].aspx": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
exit status 1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.