Git Product home page Git Product logo

Comments (3)

sefinek24 avatar sefinek24 commented on August 15, 2024

I have improved the script responsible for removing duplicates. Now the deduplication should work better.

PS C:\GitHub\Sefinek-Blocklist-Collection\scripts> node .\remove-duplicates.js
🔍 Searching for duplicates in blocklist/template directory...
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\gambling.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\test.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\useless-websites.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\blocklistproject.ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\DandelionSprout.GameConsoleAdblockList.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\jerryn70.GoodbyeAds.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\ads\yoyo.AdsTrackersEtc.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\anime\myanimelist.net.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\anime\shinden.pl.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\discord.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\skype.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\spotify.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\steam-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\steam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\apps\whatsapp.txt
🗑️ 3 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\0Zinc. easylist.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\0Zinc. easyprivacy.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\4skinS kywalker.Anti-Porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\abuse.urlhaus.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\adaway.hosts.txt
🗑️ 173 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\anud eepND.adservers.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\AssoEchap.stalkerware-indicators.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\bigdargon.hostsVN.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.abuse.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.drugs.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.fraud.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.gambling.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.phishing.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.piracy.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.ransomware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.redirect.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.scam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\blocklistproject.youtube.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\cbuijs.adult-domains.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\chadmayfield.pi_blocklist_porn_all.txt
🗑️ 11 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\craiu .mobiletrackers.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\crazy-max.WindowsSpyBlocker.hosts-spy.txt
🗑️ 289 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Dand elionSprout-AntiMalwareHosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\deathbybandaid.CountryCodesLists-France.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\deathbybandaid.ParsedBlacklists-easylist-fr.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\deathbybandaid.ParsedBlacklists-easylist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\developerdan.amp-hosts-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\developerdan.dating-services-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\developerdan.hate-and-junk-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\digitalside.latestdomains.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\disconnectme.simple_ad.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\disconnectme.simple_malvertising.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Dogino .Discord-Phishing-URLs-phishing.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Dogino .Discord-Phishing-URLs-scam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\durablenapkin.scamblocklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\ente-dev.google-amp-hosts.txt
🗑️ 38 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\ente- dev.tv.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.add.2o7Net.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.add.Risk.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.add.Spam.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\FadeMind.UncheckyAds.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.AdguardDNS.txt
🗑️ 13 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\fireb og.Admiral.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Easylist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Easyprivacy.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Prigent-Ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebog.Prigent-Crypto.txt
🗑️ 5 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\firebo g.w3kbl.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\frogeye.firstparty-trackers-hosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\hole-cert.domains_hosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\hoshsadiq.adblock-nocoin-list.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\justdomains.adguarddns.txt
🗑️ 20 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\justd omains.easylist.txt
🗑️ 1154 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\jus tdomains.easyprivacy.txt
🗑️ 1 duplicate removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\kboghda dy.youtubelist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.adguard_host.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.adguard_mobile_host.txt
🗑️ 173 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Majk iIT.adservers.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.Ad_filter_list_by_Disconnect.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.easylist_host.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.easy_privacy_host.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.gambling-hosts.txt
🗑️ 10687 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Ma jkiIT.hostfile.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.NoTrack_Tracker_Blocklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\MajkiIT.SmartTV_ads.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\malware-filter.urlhaus-filter-hosts-online.txt
🗑️ 2 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\marktr on.fakenews.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\mhhakim.custom-porn-blocklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\mhhakim.porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\mitchellkrogza.Stop.Google.Analytics.Ghost.Spam-INACTIVE.txt
🗑️ 59 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\neode vpro.neodevhost.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\notracking.hostnames.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\oisd.big.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\oisd.nsfw.txt
🗑️ 11 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\phish ingArmy.phishing_army_blocklist_extended.txt
🗑️ 4 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Polish FiltersTeam.KADhosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\quidsup.notrack-malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\r-a-y.AdguardApps.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\r-a-y.AdguardMobileAds.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\r-a-y.AdguardMobileSpyware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\RPiList-Malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\RPiList-Phishing.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\RPiList.Spam-Mails.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Sinfonietta.pornography-hosts.txt
🗑️ 20 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Snota 418.Crypto-streams.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\Spam404.main-blacklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\StevenBlack.fakenews-gambling-porn.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\StevenBlack.hosts.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\stopforumspam.toxic_domains_whole.txt
🗑️ 136 duplicates removed from C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\xlim it91.blacklist.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\forks\zerodot1.CoinBlockerLists.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\games\all-valve-games.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\games\league-of-legends.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\games\valorant.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\blocklistproject.abuse.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\blocklistproject.fraud.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\blocklistproject.malware.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\main.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\phishing.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\reported-by-norton.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\malicious\web-attacks.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\booth.pm.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\gamebanana.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\omegle.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\patreon.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\pinterest.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\pixiv.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\riotgames.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\youtube-extended.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\sites\youtube.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\facebook.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\instagram.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\snapchat.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\tiktok.txt
✔️ No duplicates found in C:\GitHub\Sefinek-Blocklist-Collection\blocklist\template\social\twitter.txt

from sefinek-blocklist-collection.

LexterS999 avatar LexterS999 commented on August 15, 2024

That's awesome, and what about merging lists into one? Because some apps crushing when trying to put a lot of links at a time. Like TechnitiumDNS, crushing after 15 links trying to update.

from sefinek-blocklist-collection.

sefinek24 avatar sefinek24 commented on August 15, 2024

@asterriya, I tried to do it, but unfortunately it's beyond my abilities. Idk how to do it. If someone has a moment to write this script, I would be grateful.

Script: https://github.com/sefinek24/Sefinek-Blocklist-Collection/blob/main/scripts/create-everything-list.js

from sefinek-blocklist-collection.

Related Issues (19)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.