Hi ninoseki,
Thanks for sharing the amazing phishing kit detection tool. May I ask do you have any idea among the list of phishing feed resources, which one provides the majority as well as high-quality number of phishing URLs? Which resource do you think is the most reliable and updated? Thanks

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update mysql docker tag to v8.4
• chore(deps): update redis/redis-stack docker tag to v7

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Hello, opening an issue request as mentioned on Twitter. I'm hoping you might be able to assist me.

The problem I'm having is that PhishStats is reporting an error on scan. It's happened over the last few days (error below), but prior 10 Feb 2021, things worked fine.

I get the error when I call miteru via cron (daily at 01:30am) or manually. Also, when I load the PhishStats URL directly (https://phishstats.info:2096/api/phishing?_sort=-id&size=100), things load fine on the browser and it appears to work.

I've rebooted the system. Prior to the errors starting, I made no changes to the server.

Error:
02/13/2021
Failed to load PhishStats feed (execution expired)
Loaded 11017 URLs to crawl. (crawling in 12 threads)

02/14/2021
Failed to load PhishStats feed (execution expired)
Loaded 10943 URLs to crawl. (crawling in 12 threads)

02/15/2021
Failed to load PhishStats feed (execution expired)
Loaded 20682 URLs to crawl. (crawling in 12 threads)

Command via cron:

/usr/bin/ruby2.5 /usr/local/bin/miteru --threads=12 --size=10000 --ayashige --auto-download --directory-traveling --download-to=/media/steved3/LINUX/miteru_kit_dl/

Command for manual usage:

miteru command used (manual) miteru --threads=12 --size=10000 --ayashige --auto-download --directory-traveling --download-to=/media/steved3/LINUX/miteru_kit_dl/

OS: Debian 10

steved3@steved3-lab:~$ ruby -v
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]

miteru v0.14.7

steps taken:
rebooted system
attempted manual scan to see if error persisted (it does)
attempted to load PhishStats URL via browser to confirm API is functional

Is there any reason you can think of that would cause the domain data to not be added to file downloads?

The issue started today. When a file downloads, the name is a hash value only, and not domain_filename.zip_hash.zip like it used to be. I've listed examples below, just curious as to the cause, and if you've seen this before.

steved3@steved3-lab:~$ ruby -v
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]

steved3@steved3-lab:~$ gem -v
3.2.32

Debian 10

Example (26-11-21):
hxxps://chbi.duckdns.org: it doesn't contain a phishing kit.
hxxps://chbi.duckdns.org/chase.com: it might contain a phishing kit: us-online.zip.
hxxps://chbi.duckdns.org/chase.com/us-online: �[0;91;49mit might contain a phishing kit: us-online.zip(2101KB).�[0m
Download hxxps://chbi.duckdns.org/chase.com/us-online.zip as /media/steved3/LINUX/miteru_kit_dl//0a44729c-397c-449f-a4f1-a8e80cfc7138.zip
Don't download hxxps://chbi.duckdns.org/chase.com/us-online.zip. The same hash is already recorded. (SHA256: 85dd241c7f5a286ce65a8214958ede3e09036c2e04cbc5ca97bde6167fcaf347).

Previous working example(24-11-21):
hxxp://firateducation.com/owa/Office365: It doesn't contain a phishing kit.
hxxp://firateducation.com/owa: �[0;91;49mIt might contain a phishing kit: OfficeEdu.zip(3478KB).�[0m
Download hxxp://firateducation.com/owa/OfficeEdu.zip as /media/steved3/LINUX/miteru_kit_dl//firateducation.com_OfficeEdu.zip_934e0954d57e299e16a9.zip

13: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/crawler.rb:34:inblock in execute'
12: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/website.rb:36:in has_kits?' 11: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/website.rb:28:in index?'
10: from /var/lib/gems/2.5.0/gems/miteru-0.12.2/lib/miteru/website.rb:13:in title' 9: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/element.rb:177:in text'
8: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/node_set.rb:276:in text' 7: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/node_set.rb:276:in each'
6: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/node_set.rb:278:in block in text' 5: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/text.rb:24:in text'
4: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/entity_decoder.rb:5:in try_decode' 3: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/entity_decoder.rb:14:in decode'
2: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/html/entities.rb:2142:in decode' 1: from /var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/entities.rb:77:in decode'
/var/lib/gems/2.5.0/gems/oga-2.15/lib/oga/xml/entities.rb:77:in gsub': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)

System info:
Ubuntu 18.04.2 LTS
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]

What I'm doing wrong?

Hi ninoseki,
May I ask if I want to add new feed source, which are the files I need to revise? Thanks for helping!

Not sure if related, but after the update, the Ayashige feed stopped working.

Error:
Failed to load ayashige feed (Miteru::HTTPResponseError)

Feed source:
https://ayashige.herokuapp.com/feed

If unrelated or a non-issue, please feel free to close. Just wanted to bring it to your attention.

Sorry to open yet another issue.

But the URLScan feed isn't triggering in the script. The collection issue started on December 2, but I didn't report it as I thought it was temporary problem. Given that the problem has remained, thought it best to bring it to your attention, as the URLScan gem seems to be what is making the call.

I've checked the database, and the only feeds that seem to be working are PhishStats and Ayashige. The miteru command I use has been the same one I've used since the start.

Miteru command used:

/home/steved3/miteru execute --threads=12 --size=10000 --ayashige --auto-download --directory-traveling --download-to=/media/steved3/LINUX/miteru_kit_dl

Troubleshooting:

Attempted to alter the number of responses via --size (example: --size=9000) which didn't work.

When I visit the URLScan query directly via browser, it loads 10,000 results fine.
Example: https://urlscan.io/api/v1/search/?q=task.method:automatic

Miteru logs:

November 30

Loaded 11023 URLs to crawl. (crawling in 12 threads)

December 01

Loaded 10833 URLs to crawl. (crawling in 12 threads)

December 02

Loaded 1664 URLs to crawl. (crawling in 12 threads)

December 03

Loaded 1791 URLs to crawl. (crawling in 12 threads)

Image was taken 4 hours after this scan.

Operational Details:

OS: Debian 10

steved3@steved3-lab:~$ ruby -v
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]

steved3@steved3-lab:~$ gem -v
3.2.32

steved3@steved3-lab:~$ gem which miteru
/home/steved3/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/miteru-1.1.0/lib/miteru.rb

• https://openphish.com/phishing_feeds.html
  • https://openphish.com/feed.txt
• Update Frequency: 1 hour

Hi,

Is it possible to change the target URL to a specific domain like '.xyz' and or network range ?

Cheers.

Add following extensions to grab.

• .7z
• .rar
• .tar
• .gz

Files with original extension .tar.gz are downloaded as .gz causing some unpackers on Linux to give an error.

Since update to version 0.13.0 the feed crawling gets stuck after few URLs, no errors, I tried many options combinations with same output.

This is the latest try, manually interrupted because stuck there for hours:

$ /usr/local/bin/miteru execute --threads=12 --directory-traveling --size=4000 --auto-download --download-to=/opt/miteru/

Loaded 9610 URLs to crawl. (crawling in 12 threads)
http://103.112.226.142/Mozi.m: It doesn't contain a phishing kit.
http://103.112.226.142: It doesn't contain a phishing kit.
http://103.30.183.173: It doesn't contain a phishing kit.
http://103.30.183.173/adm/sites: It doesn't contain a phishing kit.
http://103.30.183.173/adm: It doesn't contain a phishing kit.
http://103.91.90.221: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7/multifunctional-box: It doesn't contain a phishing kit.
http://104.131.148.172: It doesn't contain a phishing kit.
http://104.248.26.90: It doesn't contain a phishing kit.
http://103.91.90.221/AdminPanel: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin/127016282754576: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin/127016282754576/ixee5102uofn: It doesn't contain a phishing kit.
http://103.30.183.173/adm/sites/zn4uqjzca: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin/127016282754576/ixee5102uofn/8yq-00923-71189530-n6iw8-ptmmjll: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7/multifunctional-box/close-4xol48ieqx-7dupxos475y8: It doesn't contain a phishing kit.
http://103.91.90.221/AdminPanel/Reporting: It doesn't contain a phishing kit.
http://104.131.148.172/1kfhr7/multifunctional-box/close-4xol48ieqx-7dupxos475y8/018438913656-Upg2Is7: It doesn't contain a phishing kit.
http://104.248.26.90/wp-admin: It doesn't contain a phishing kit.
http://106.12.111.189: It doesn't contain a phishing kit.
http://108.171.179.117: It doesn't contain a phishing kit.
http://108.171.179.117/qbshelpdesk: It doesn't contain a phishing kit.
http://106.12.111.189/wr0pezn/sites/s0kgm6: It doesn't contain a phishing kit.
http://108.171.179.117/qbshelpdesk/55br0-tqr-155: It doesn't contain a phishing kit.
http://106.12.111.189/wr0pezn: It doesn't contain a phishing kit.
^CTraceback (most recent call last):
	19: from /usr/local/bin/miteru:23:in `<main>'
	18: from /usr/local/bin/miteru:23:in `load'
	17: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/exe/miteru:8:in `<top (required)>'
	16: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
	15: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
	14: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
	13: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
	12: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/lib/miteru/cli.rb:30:in `execute'
	11: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/lib/miteru/crawler.rb:53:in `execute'
	10: from /var/lib/gems/2.5.0/gems/miteru-0.13.0/lib/miteru/crawler.rb:30:in `execute'
	 9: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:227:in `each'
	 8: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:275:in `map'
	 7: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:353:in `work_in_threads'
	 6: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:204:in `in_threads'
	 5: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:204:in `handle_interrupt'
	 4: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:211:in `block in in_threads'
	 3: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:211:in `handle_interrupt'
	 2: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:212:in `block (2 levels) in in_threads'
	 1: from /var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:212:in `map'
/var/lib/gems/2.5.0/gems/parallel-1.19.1/lib/parallel.rb:212:in `value': Interrupt

System info:
Ubuntu 18.04.3 LTS
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]

Hi, this is from today, seems I still got encoding related issues...