projectdiscovery / wappalyzergo Goto Github PK
View Code? Open in Web Editor NEWA high performance go implementation of Wappalyzer Technology Detection Library
License: MIT License
A high performance go implementation of Wappalyzer Technology Detection Library
License: MIT License
Hi projectdiscovery team, thank you guys for the great tool. I want to report the issue that caused this error "invalid memory address or nil pointer dereference". I have debugged and figured out that the VersionRegex is Nil, the error happened in "github.com/projectdiscovery/wappalyzergo.(*VersionRegex).MatchString"
You guys can reproduce the error using below code!
package main
import (
"fmt"
"io"
"log"
"net/http"
wappalyzer "github.com/projectdiscovery/wappalyzergo"
)
func main() {
resp, err := http.DefaultClient.Get("https://samchicks.com")
if err != nil {
log.Fatal(err)
}
data, _ := io.ReadAll(resp.Body) // Ignoring error for example
wappalyzerClient, err := wappalyzer.New()
fingerprints := wappalyzerClient.Fingerprint(resp.Header, data)
fmt.Printf("%v\n", fingerprints)
// Output: map[Acquia Cloud Platform:{} Amazon EC2:{} Apache:{} Cloudflare:{} Drupal:{} PHP:{} Percona:{} React:{} Varnish:{}]
}
The repo you're pulling from for fingerprint update has barely been updated since inception. Your action runs weekly and it seems your releases do as well, however both actions are kinda pointless if there isn't any new fingerprints/code.
I'd suggest having a look at https://github.com/enthec/webappanalyzer for fingerprint updates, they've actually had new stuff since inception.
The GitHub update action points to the wrong file fingerprints_data.go
instead of fingerprints_data.json
:
fingerprints_data.go
=> fingerprints_data.json
fingerprints_data.go
with embedA third of the technologies listed in fingerprints_data.json will never be detected, as the only keys listed there are the technology name and description
examples:
"AddShoppers":{"description": "AddShoppers is the ...
"ECharts":{"description": "ECharts is an open-source...
...
And there are 884 similar technologies, which only add weight to file fingerprints_data.json for nothing, because from original file keys are missing: scriptSrc, dom
P.S.: Apologies for my bad English, I use a translator
An always up-to-date javascript injectable library version should be prepared and kept up to date. The library should be injected via headless into existing browsers' contexts and collect enriched info within the JS engine for a specific domain open in a browser tab. This is similar to the behavior of the official wappalyzer extension https://chrome.google.com/webstore/detail/wappalyzer-technology-pro
Hi all!
Pretty recently, the Wappalyzer original project has been made private (see here).
Therefore, the fingerprints fail to download from the original repository.
I saw some discussion here: https://news.ycombinator.com/item?id=37236746
And there are several repositories with the latest snapshot.
Should we consider to start maintaining a list of fingerprints inside this same repository using the original one as a baseline?
I just created a repository for myself that I would love to try to maintain with the community if needed: https://github.com/BBerastegui/wappalyzer-fingerprints
From now we can pull from there.
Cheers!
Would you be open to a PR that imports https://github.com/wappalyzer/wappalyzer/blob/master/src/categories.json on a cron, similar to how technologies are pulled weekly, to make category data available?
The current wappalyzer dataset is kind of unreadable due to size and native json intrinsic unreadability which grows linearly with the size of json. It can be a good occasion to investigate a new implementation and structure the signatures in a folder tree fashion, with a more readable format such as yaml (potentially nuclei templates covering technologies could be reused).
All the signatures must respect the following requirements:
Hi Team,
As always another major and interesting opensource project from you all 🔥
One of the first thing that I noticed that https://github.com/rverton/webanalyze downloads the fingerprints from the wappalyzer repo and save them locally and also has a update
flag which can be used to update the same.
But looks like this project is hard coding them in https://github.com/projectdiscovery/wappalyzergo/blob/master/fingerprints_data.go which is weird considering you have to update the code every time their is an update in the wappalyzer fingerprint which is quite frequent.
Also the original file https://github.com/AliasIO/wappalyzer/blob/master/src/technologies.json is of 22181
lines where as even if we beautify https://github.com/projectdiscovery/wappalyzergo/blob/master/fingerprints_data.go its of 10861
lines only. Would love to understand why is this difference, are you ignoring non useful tags from the list ?
I did see this https://github.com/projectdiscovery/wappalyzergo/blob/master/cmd/update-fingerprints/main.go But how does this works if am using this in my code like below, does it tries to update the fingerprint every time wappalyzer.New()
is being called ?
Can we manually invoke this update part using something like wappalyzer.update()
only once before reusing the same wappalyzerClient.
for _, url range urls{
resp, err := http.DefaultClient.Get(url)
if err != nil {
log.Fatal(err)
}
data, _ := ioutil.ReadAll(resp.Body) // Ignoring error for example
wappalyzerClient, err := wappalyzer.New()
fingerprints := wappalyzerClient.Fingerprint(resp.Header, data)
fmt.Printf("%v\n", fingerprints)
}
--
Regards,
@bugbaba
Add the following GitHub actions
hi, i use wappalyzergo as library,the fingerprints_data.json file is bound to the repo code. When I update the json file, I need to update the whole golang project, I think it would be nice to have a function on the Wappalyze object to define fingerprints, In this way, when used as a function library, it avoids constantly updating the entire project code in order to update the data file
// now I use some tricks to solve this problem
// the wappalyzerFileBytes define from outter resource, such as cloud object storage service
//go:linkname compileFingerprint github.com/projectdiscovery/wappalyzergo.compileFingerprint
func compileFingerprint(fingerprint *wappalyzer.Fingerprint) *wappalyzer.CompiledFingerprint
type WappalyzeWithUnsafe struct {
fingerprints unsafe.Pointer
}
func NewWappalyzerClient() *wappalyzer.Wappalyze {
EnsureWappalyzerFile()
wappalyzerClient := new(wappalyzer.Wappalyze)
wappalyzerClientCompiledFingerprints := &wappalyzer.CompiledFingerprints{
Apps: make(map[string]*wappalyzer.CompiledFingerprint),
}
var fingerprintsStruct wappalyzer.Fingerprints
json.Unmarshal([]byte(wappalyzerFileBytes), &fingerprintsStruct)
for i, fingerprint := range fingerprintsStruct.Apps {
wappalyzerClientCompiledFingerprints.Apps[i] = compileFingerprint(fingerprint)
}
wappalyzeUnsafe := (*WappalyzeWithUnsafe)(unsafe.Pointer(wappalyzerClient))
wappalyzeUnsafe.fingerprints = unsafe.Pointer(wappalyzerClientCompiledFingerprints)
return wappalyzerClient
}
Sadly, Wappalyzer has been removed from github due to some bitter feelings. The links to the signature files now result in 404
s.
However, it is still possible to pull the signatures from the Chrome extension. This gets you a crx
file, which similar to a jar
or apk
is just a glorified zip
. The signatures can be downloaded and extracted from the following URL:
51pwn@123 update-fingerprints $ go build main.go
51pwn@123 update-fingerprints $ ./main
2022/06/24 13:24:01 Read fingerprints from the server
2022/06/24 13:24:01 Starting normalizing of 3084 fingerprints...
2022/06/24 13:24:01 Got 3084 valid fingerprints
unexpected fault address 0xb01dfacedebac1e
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x105f71f]
goroutine 1 [running]:
runtime.throw({0x12c771b?, 0x1?})
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/panic.go:992 +0x71 fp=0xc0001b5628 sp=0xc0001b55f8 pc=0x1033871
runtime.sigpanic()
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/signal_unix.go:825 +0x305 fp=0xc0001b5678 sp=0xc0001b5628 pc=0x1048d25
aeshashbody()
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:1343 +0x39f fp=0xc0001b5680 sp=0xc0001b5678 pc=0x105f71f
runtime.mapiternext(0xc0000e4740)
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/map.go:934 +0x2cb fp=0xc0001b56f0 sp=0xc0001b5680 pc=0x101084b
runtime.mapiterinit(0x0?, 0x1?, 0x1b?)
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/map.go:861 +0x228 fp=0xc0001b5710 sp=0xc0001b56f0 pc=0x1010528
reflect.mapiterinit(0x2?, 0x1273fe0?, 0x100b195?)
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/map.go:1373 +0x19 fp=0xc0001b5738 sp=0xc0001b5710 pc=0x105c759
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
/Users/51pwn/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_map.go:112
github.com/json-iterator/go.(*mapEncoder).Encode(0xc0002f1050, 0xc0001a85a8, 0xc000148060)
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_map.go:257 +0x21c fp=0xc0001b57f0 sp=0xc0001b5738 pc=0x123917c
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc0002f10b0, 0x125c0ce?, 0xc000148060)
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:110 +0x56 fp=0xc0001b5868 sp=0xc0001b57f0 pc=0x1247176
github.com/json-iterator/go.(*structEncoder).Encode(0xc0002f10e0, 0x0?, 0xc000148060)
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:158 +0x765 fp=0xc0001b5950 sp=0xc0001b5868 pc=0x1247b85
github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc0001ae960?, 0x0?, 0x0?)
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_optional.go:70 +0xa4 fp=0xc0001b59a0 sp=0xc0001b5950 pc=0x123ed84
github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc00060e940, 0xc0001a85a8, 0xc0002f0120?)
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:219 +0x82 fp=0xc0001b59d8 sp=0xc0001b59a0 pc=0x1232082
github.com/json-iterator/go.(*Stream).WriteVal(0xc000148060, {0x1267fa0, 0xc0001a85a8})
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:98 +0x158 fp=0xc0001b5a48 sp=0xc0001b59d8 pc=0x1231398
github.com/json-iterator/go.(*frozenConfig).Marshal(0xc0001ae960, {0x1267fa0, 0xc0001a85a8})
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/config.go:299 +0xc9 fp=0xc0001b5ae0 sp=0xc0001b5a48 pc=0x1229ba9
github.com/json-iterator/go.Marshal(...)
/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/adapter.go:34
main.main()
/Users/51pwn/MyWork/wappalyzergo/cmd/update-fingerprints/main.go:96 +0x2ca fp=0xc0001b5f80 sp=0xc0001b5ae0 pc=0x125004a
runtime.main()
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/proc.go:250 +0x212 fp=0xc0001b5fe0 sp=0xc0001b5f80 pc=0x1035f52
runtime.goexit()
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001b5fe8 sp=0xc0001b5fe0 pc=0x1062341
goroutine 7 [IO wait]:
internal/poll.runtime_pollWait(0x25c3120, 0x72)
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000248300?, 0xc00050e000?, 0x0)
/usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000248300, {0xc00050e000, 0xa954, 0xa954})
/usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000248300, {0xc00050e000?, 0xc000076380?, 0xc00050e005?})
/usr/local/Cellar/go/1.18.2/libexec/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000010010, {0xc00050e000?, 0x243ffff?, 0x2245700?})
/usr/local/Cellar/go/1.18.2/libexec/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00016e720, {0xc00050e000?, 0x0?, 0x500?})
/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:784 +0x3d
bytes.(*Buffer).ReadFrom(0xc00010e278, {0x133e9a0, 0xc00016e720})
/usr/local/Cellar/go/1.18.2/libexec/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc00010e000, {0x25fd008?, 0xc000010010}, 0x203000?)
/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:806 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc00010e000, 0x0)
/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:613 +0x116
crypto/tls.(*Conn).readRecord(...)
/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:581
crypto/tls.(*Conn).Read(0xc00010e000, {0xc00014f000, 0x1000, 0x11e5d60?})
/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:1284 +0x16f
bufio.(*Reader).Read(0xc0001484e0, {0xc0001364a0, 0x9, 0x11f36e2?})
/usr/local/Cellar/go/1.18.2/libexec/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x133e8c0, 0xc0001484e0}, {0xc0001364a0, 0x9, 0x9}, 0x9)
/usr/local/Cellar/go/1.18.2/libexec/src/io/io.go:331 +0x9a
io.ReadFull(...)
/usr/local/Cellar/go/1.18.2/libexec/src/io/io.go:350
net/http.http2readFrameHeader({0xc0001364a0?, 0x9?, 0xc0002160f0?}, {0x133e8c0?, 0xc0001484e0?})
/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:1566 +0x6e
net/http.(*http2Framer).ReadFrame(0xc000136460)
/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:1830 +0x95
net/http.(*http2clientConnReadLoop).run(0xc0001b9f98)
/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:8815 +0x130
net/http.(*http2ClientConn).readLoop(0xc00010c180)
/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:8711 +0x6f
created by net/http.(*http2Transport).newClientConn
/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:7439 +0xa65
Hello Team,
As the technologies.json
file was growing the wappalyzer team had decide to break it into multiple files based on starting character In this commit https://github.com/AliasIO/wappalyzer/commit/2401457a994f40f23140b62e406590bdaa0476a2
We can see the new json files here https://github.com/AliasIO/wappalyzer/tree/master/src/technologies and as this project depends on technologies.json
at https://github.com/projectdiscovery/wappalyzergo/blob/master/cmd/update-fingerprints/main.go#L20 it will return 404 error and break the process.
So now we need to download all thee files and then merge them together, before using them.
--
Regards,
@bugbaba
package main
import (
"fmt"
wappalyzer "github.com/projectdiscovery/wappalyzergo"
)
func main() {
wappalyzerClient, _ := wappalyzer.New()
fingerprints := wappalyzerClient.Fingerprint(map[string][]string{
"Set-Cookie": []string{"path=/; jsessionid=111; path=/, jsessionid=111;"},
}, []byte(""))
fingerprints1 := wappalyzerClient.Fingerprint(map[string][]string{
"Set-Cookie": []string{"jsessionid=111; path=/;"},
}, []byte(""))
fmt.Println(fingerprints)
fmt.Println(fingerprints1)
}
output:
map[]
map[Java:{}]
If the value changes position, the test will fail
Sorry for my bad english, using google translate.
defines only if the fingerprint is immediately after Set-Cookie:
eg (Set-Cookie:[laravel_session=)
but if somewhere in the middle, such as (Set-Cookie:[XSRF-TOKEN=; expires=, ...; ...; path=/ laravel_session=eyJ*;)
then no longer defines
example domains:
hit4central.com
shcherbyna-advocat.com
amiguiec.xyz
hiom.in
puncbimsi.cf
suzuki.com.pl
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.