ninoseki / miteru Goto Github PK
View Code? Open in Web Editor NEWA phishing kit collector for scavengers
Home Page: https://urlscan.io/search/#task.tags%3Amiteru%20AND%20task.tags%3Aphishkit
License: MIT License
A phishing kit collector for scavengers
Home Page: https://urlscan.io/search/#task.tags%3Amiteru%20AND%20task.tags%3Aphishkit
License: MIT License
Add following extensions to grab.
Files with original extension .tar.gz
are downloaded as .gz
causing some unpackers on Linux to give an error.
Is there any reason you can think of that would cause the domain data to not be added to file downloads?
The issue started today. When a file downloads, the name is a hash value only, and not domain_filename.zip_hash.zip like it used to be. I've listed examples below, just curious as to the cause, and if you've seen this before.
steved3@steved3-lab:~$ ruby -v
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]
steved3@steved3-lab:~$ gem -v
3.2.32
Debian 10
Example (26-11-21):
hxxps://chbi.duckdns.org: it doesn't contain a phishing kit.
hxxps://chbi.duckdns.org/chase.com: it might contain a phishing kit: us-online.zip.
hxxps://chbi.duckdns.org/chase.com/us-online: �[0;91;49mit might contain a phishing kit: us-online.zip(2101KB).�[0m
Download hxxps://chbi.duckdns.org/chase.com/us-online.zip as /media/steved3/LINUX/miteru_kit_dl//0a44729c-397c-449f-a4f1-a8e80cfc7138.zip
Don't download hxxps://chbi.duckdns.org/chase.com/us-online.zip. The same hash is already recorded. (SHA256: 85dd241c7f5a286ce65a8214958ede3e09036c2e04cbc5ca97bde6167fcaf347).
Previous working example(24-11-21):
hxxp://firateducation.com/owa/Office365: It doesn't contain a phishing kit.
hxxp://firateducation.com/owa: �[0;91;49mIt might contain a phishing kit: OfficeEdu.zip(3478KB).�[0m
Download hxxp://firateducation.com/owa/OfficeEdu.zip as /media/steved3/LINUX/miteru_kit_dl//firateducation.com_OfficeEdu.zip_934e0954d57e299e16a9.zip
Hi,
Is it possible to change the target URL to a specific domain like '.xyz' and or network range ?
Cheers.
13: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/crawler.rb:34:in
block in execute'
12: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/website.rb:36:in has_kits?' 11: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/website.rb:28:in
index?'
10: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/website.rb:13:in title' 9: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/element.rb:177:in
text'
8: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/node_set.rb:276:in text' 7: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/node_set.rb:276:in
each'
6: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/node_set.rb:278:in block in text' 5: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/text.rb:24:in
text'
4: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/entity_decoder.rb:5:in try_decode' 3: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/entity_decoder.rb:14:in
decode'
2: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/html/entities.rb:2142:in decode' 1: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/entities.rb:77:in
decode'
/var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/entities.rb:77:in gsub': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
System info:
Ubuntu 18.04.2 LTS
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
What I'm doing wrong?
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
Gemfile
docker-compose.yml
redis/redis-stack 6.2.6-v10
docker/Dockerfile
ruby 3-alpine3.13
.github/workflows/gem.yml
actions/checkout v4
ruby/setup-ruby v1
.github/workflows/ruby.yml
actions/checkout v4
ruby/setup-ruby v1
postgres 16
mysql 8.0
Hi ninoseki,
May I ask if I want to add new feed source, which are the files I need to revise? Thanks for helping!
Hello, opening an issue request as mentioned on Twitter. I'm hoping you might be able to assist me.
The problem I'm having is that PhishStats is reporting an error on scan. It's happened over the last few days (error below), but prior 10 Feb 2021, things worked fine.
I get the error when I call miteru via cron (daily at 01:30am) or manually. Also, when I load the PhishStats URL directly (https://phishstats.info:2096/api/phishing?_sort=-id&size=100), things load fine on the browser and it appears to work.
I've rebooted the system. Prior to the errors starting, I made no changes to the server.
Error:
02/13/2021
Failed to load PhishStats feed (execution expired)
Loaded 11017 URLs to crawl. (crawling in 12 threads)
02/14/2021
Failed to load PhishStats feed (execution expired)
Loaded 10943 URLs to crawl. (crawling in 12 threads)
02/15/2021
Failed to load PhishStats feed (execution expired)
Loaded 20682 URLs to crawl. (crawling in 12 threads)
Command via cron:
/usr/bin/ruby2.5 /usr/local/bin/miteru --threads=12 --size=10000 --ayashige --auto-download --directory-traveling --download-to=/media/steved3/LINUX/miteru_kit_dl/
Command for manual usage:
miteru command used (manual) miteru --threads=12 --size=10000 --ayashige --auto-download --directory-traveling --download-to=/media/steved3/LINUX/miteru_kit_dl/
OS: Debian 10
steved3@steved3-lab:~$ ruby -v
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]
miteru v0.14.7
steps taken:
rebooted system
attempted manual scan to see if error persisted (it does)
attempted to load PhishStats URL via browser to confirm API is functional
Not sure if related, but after the update, the Ayashige feed stopped working.
Error:
Failed to load ayashige feed (Miteru::HTTPResponseError)
Feed source:
https://ayashige.herokuapp.com/feed
If unrelated or a non-issue, please feel free to close. Just wanted to bring it to your attention.
Since update to version 0.13.0 the feed crawling gets stuck after few URLs, no errors, I tried many options combinations with same output.
This is the latest try, manually interrupted because stuck there for hours:
$ /usr/local/bin/miteru execute --threads=12 --directory-traveling --size=4000 --auto-download --download-to=/opt/miteru/
Loaded 9610 URLs to crawl. (crawling in 12 threads)
http://103.112.226.142/Mozi.m: It doesn't contain a phishing kit.
http://103.112.226.142: It doesn't contain a phishing kit.
http://103.30.183.173: It doesn't contain a phishing kit.
http://103.30.183.173/adm/sites: It doesn't contain a phishing kit.
http://103.30.183.173/adm: It doesn't contain a phishing kit.
http://103.91.90.221: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7/multifunctional-box: It doesn't contain a phishing kit.
http://104.131.148.172: It doesn't contain a phishing kit.
http://104.248.26.90: It doesn't contain a phishing kit.
http://103.91.90.221/AdminPanel: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin/127016282754576: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin/127016282754576/ixee5102uofn: It doesn't contain a phishing kit.
http://103.30.183.173/adm/sites/zn4uqjzca: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin/127016282754576/ixee5102uofn/8yq-00923-71189530-n6iw8-ptmmjll: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7/multifunctional-box/close-4xol48ieqx-7dupxos475y8: It doesn't contain a phishing kit.
http://103.91.90.221/AdminPanel/Reporting: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7/multifunctional-box/close-4xol48ieqx-7dupxos475y8/018438913656-Upg2Is7: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin: It doesn't contain a phishing kit.
http://106.12.111.189: It doesn't contain a phishing kit.
http://108.171.179.117: It doesn't contain a phishing kit.
http://108.171.179.117/qbshelpdesk: It doesn't contain a phishing kit.
http://106.12.111.189/wr0pezn/sites/s0kgm6: It doesn't contain a phishing kit.
http://108.171.179.117/qbshelpdesk/55br0-tqr-155: It doesn't contain a phishing kit.
http://106.12.111.189/wr0pezn: It doesn't contain a phishing kit.
^CTraceback (most recent call last):
19: from /usr/local/bin/miteru:23:in `<main>'
18: from /usr/local/bin/miteru:23:in `load'
17: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/exe/miteru:8:in `<top (required)>'
16: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
15: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
14: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
13: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
12: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/lib/miteru/cli.rb:30:in `execute'
11: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/lib/miteru/crawler.rb:53:in `execute'
10: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/lib/miteru/crawler.rb:30:in `execute'
9: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:227:in `each'
8: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:275:in `map'
7: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:353:in `work_in_threads'
6: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:204:in `in_threads'
5: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:204:in `handle_interrupt'
4: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:211:in `block in in_threads'
3: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:211:in `handle_interrupt'
2: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:212:in `block (2 levels) in in_threads'
1: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:212:in `map'
/var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:212:in `value': Interrupt
System info:
Ubuntu 18.04.3 LTS
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
Sorry to open yet another issue.
But the URLScan feed isn't triggering in the script. The collection issue started on December 2, but I didn't report it as I thought it was temporary problem. Given that the problem has remained, thought it best to bring it to your attention, as the URLScan gem seems to be what is making the call.
I've checked the database, and the only feeds that seem to be working are PhishStats and Ayashige. The miteru command I use has been the same one I've used since the start.
/home/steved3/miteru execute --threads=12 --size=10000 --ayashige --auto-download --directory-traveling --download-to=/media/steved3/LINUX/miteru_kit_dl
Attempted to alter the number of responses via --size (example: --size=9000) which didn't work.
When I visit the URLScan query directly via browser, it loads 10,000 results fine.
Example: https://urlscan.io/api/v1/search/?q=task.method:automatic
Loaded 11023 URLs to crawl. (crawling in 12 threads)
Loaded 10833 URLs to crawl. (crawling in 12 threads)
Loaded 1664 URLs to crawl. (crawling in 12 threads)
Loaded 1791 URLs to crawl. (crawling in 12 threads)
Image was taken 4 hours after this scan.
OS: Debian 10
steved3@steved3-lab:~$ ruby -v
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]
steved3@steved3-lab:~$ gem -v
3.2.32
steved3@steved3-lab:~$ gem which miteru
/home/steved3/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/miteru-1.1.0/lib/miteru.rb
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.