Comments (3)
I have improved the script responsible for removing duplicates. Now the deduplication should work better.
PS C:\GitHub\Sefinek-Blocklist-Collection\scripts> node .\remove-duplicates.js
🔍 Searching for duplicates in blocklist/template directory...
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\gambling.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\test.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\useless-websites.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\blocklistproject.ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\DandelionSprout.GameConsoleAdblockList.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\jerryn70.GoodbyeAds.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\yoyo.AdsTrackersEtc.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\anime\myanimelist.net.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\anime\shinden.pl.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\discord.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\skype.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\spotify.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\steam-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\steam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\whatsapp.txt
🗑️ 3 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\0Zinc. easylist.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\0Zinc. easyprivacy.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\4skinS kywalker.Anti-Porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\abuse.urlhaus.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\adaway.hosts.txt
🗑️ 173 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\anud eepND.adservers.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\AssoEchap.stalkerware-indicators.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\bigdargon.hostsVN.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.abuse.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.drugs.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.fraud.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.gambling.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.phishing.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.piracy.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.ransomware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.redirect.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.scam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.youtube.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\cbuijs.adult-domains.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\chadmayfield.pi_blocklist_porn_all.txt
🗑️ 11 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\craiu .mobiletrackers.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\crazy-max.WindowsSpyBlocker.hosts-spy.txt
🗑️ 289 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Dand elionSprout-AntiMalwareHosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\deathbybandaid.CountryCodesLists-France.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\deathbybandaid.ParsedBlacklists-easylist-fr.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\deathbybandaid.ParsedBlacklists-easylist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\developerdan.amp-hosts-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\developerdan.dating-services-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\developerdan.hate-and-junk-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\digitalside.latestdomains.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\disconnectme.simple_ad.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\disconnectme.simple_malvertising.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Dogino .Discord-Phishing-URLs-phishing.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Dogino .Discord-Phishing-URLs-scam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\durablenapkin.scamblocklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\ente-dev.google-amp-hosts.txt
🗑️ 38 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\ente- dev.tv.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.add.2o7Net.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.add.Risk.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.add.Spam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.UncheckyAds.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.AdguardDNS.txt
🗑️ 13 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\fireb og.Admiral.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Easylist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Easyprivacy.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Prigent-Ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Prigent-Crypto.txt
🗑️ 5 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebo g.w3kbl.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\frogeye.firstparty-trackers-hosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\hole-cert.domains_hosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\hoshsadiq.adblock-nocoin-list.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\justdomains.adguarddns.txt
🗑️ 20 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\justd omains.easylist.txt
🗑️ 1154 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\jus tdomains.easyprivacy.txt
🗑️ 1 duplicate removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\kboghda dy.youtubelist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.adguard_host.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.adguard_mobile_host.txt
🗑️ 173 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Majk iIT.adservers.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.Ad_filter_list_by_Disconnect.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.easylist_host.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.easy_privacy_host.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.gambling-hosts.txt
🗑️ 10687 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Ma jkiIT.hostfile.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.NoTrack_Tracker_Blocklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.SmartTV_ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\malware-filter.urlhaus-filter-hosts-online.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\marktr on.fakenews.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\mhhakim.custom-porn-blocklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\mhhakim.porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\mitchellkrogza.Stop.Google.Analytics.Ghost.Spam-INACTIVE.txt
🗑️ 59 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\neode vpro.neodevhost.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\notracking.hostnames.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\oisd.big.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\oisd.nsfw.txt
🗑️ 11 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\phish ingArmy.phishing_army_blocklist_extended.txt
🗑️ 4 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Polish FiltersTeam.KADhosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\quidsup.notrack-malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\r-a-y.AdguardApps.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\r-a-y.AdguardMobileAds.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\r-a-y.AdguardMobileSpyware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\RPiList-Malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\RPiList-Phishing.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\RPiList.Spam-Mails.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Sinfonietta.pornography-hosts.txt
🗑️ 20 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Snota 418.Crypto-streams.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Spam404.main-blacklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\StevenBlack.fakenews-gambling-porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\StevenBlack.hosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\stopforumspam.toxic_domains_whole.txt
🗑️ 136 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\xlim it91.blacklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\zerodot1.CoinBlockerLists.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\games\all-valve-games.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\games\league-of-legends.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\games\valorant.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\blocklistproject.abuse.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\blocklistproject.fraud.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\blocklistproject.malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\main.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\phishing.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\reported-by-norton.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\web-attacks.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\booth.pm.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\gamebanana.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\omegle.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\patreon.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\pinterest.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\pixiv.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\riotgames.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\youtube-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\youtube.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\facebook.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\instagram.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\snapchat.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\tiktok.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\twitter.txt
from sefinek-blocklist-collection.
That's awesome, and what about merging lists into one? Because some apps crushing when trying to put a lot of links at a time. Like TechnitiumDNS, crushing after 15 links trying to update.
from sefinek-blocklist-collection.
@asterriya, I tried to do it, but unfortunately it's beyond my abilities. Idk how to do it. If someone has a moment to write this script, I would be grateful.
from sefinek-blocklist-collection.
Related Issues (19)
- Missing disclaimer in "Useless websites" section HOT 1
- [feature]: Add removed domains to the whitelist HOT 2
- None of generated list is work HOT 8
- My lists in PiHole are not updated. HOT 4
- Maintenance HOT 1
- https://blocklist.sefinek.net/ not working HOT 1
- Source suggestion HOT 14
- List of suggestions for sources of hosts data Gambling (Indonesia) HOT 2
- RPZ & UNBOUND FORMAT HOT 3
- Added hostsvn list and removed outdate list HOT 2
- DNS-Blocklists HOT 3
- Server responds with Forbidden 403 Error for Python users HOT 4
- Duplicates files names HOT 8
- deselect button HOT 4
- False positive HOT 4
- Blocking of LGBTQ+ helpful and friendly sites, describing as "useless" HOT 4
- change url generator HOT 1
- Tiny Correction HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sefinek-blocklist-collection.