adguardteam / filterscompiler Goto Github PK

@Alex-302 commented on Fri Nov 02 2018

these rules are also valid

3g.baidu.com,m.baidu.com,m5.baidu.com,www.baidu.com#?#div:-abp-has(>div[id$="_1"]+div[id$="_5"])
cwer.me,cwer.ru,cwer.su,cwer.ws#?#div[id$="sidebar"] > .rounded:-abp-has(> .roundct > noindex)
dzgeek.com#?#li + .postbit:-abp-has(img[src$="/avatars/partenaire.png"])
e.mail.ru#?#div:-abp-has(> * > * > * > * > [data-id]:-abp-contains(/^Директ$/))
hideip.me#?#[style*="block"][style$="400px"]:style(width: 10px !important; height:10px !important; display: block !important; transform: scale(0) !important; position: fixed !important)
libertycity.ru#?#[class$="column"] table[width="100%"]:-abp-has(td[align="center"]:-abp-contains(Ad))
mail.ru,~octavius.mail.ru#?#div[class]:-abp-has(> div > div > div:-abp-contains(/^адрес и телефон$|^18\+$|^директ$|^mytarget$/i))
memesmix.net#?#div[id$="-menu"] [class*="module-body"]:-abp-has(> * > script[src]:not([src*="vk.com"]))
niebezpiecznik.pl#?#div[style="width:300px;float:left;"] ~ p:first-of-type:-abp-contains(/^$/)
pornolab.biz,pornolab.cc,pornolab.lib,pornolab.net#?#.forums tr[id^="f-"]:-abp-has(.topic_icon[src$="/folder_lock.gif"])
stratege.ru#?#div[class$="_box"]:-abp-has(> div[class*="title"]:-abp-contains(Реклама))
thg.ru#?#table:-abp-has(> tbody > tr > td > span:-abp-contains(/^реклама$/i))
www.drive2.ru#?#*:-abp-has(> *:-abp-contains(/^реклама$/i))
zen.yandex.by,zen.yandex.com,zen.yandex.com.tr,zen.yandex.fr,zen.yandex.kz,zen.yandex.ru,zen.yandex.ua#?#.feed__item:-abp-has(*:-abp-contains(/^реклама$/i))

All these rules contain $ symbol, maybe this i a reason.

Rule ||smrmembers-smr.smartmediarep.com/smc/smrmembers^$mp4,important is not found in Base filter for FF extension:

https://filters.adtidy.org/extension/firefox/filters/2.txt

$mp4 and $important are supported by FF extension

AG for Android have a Stealth mode, so I think that rules with $stealth modifier shouldn't be removed from these filters:
https://filters.adtidy.org/android/filters/2.txt
https://filters.adtidy.org/android/filters/2_optimized.txt

But these rules are working. Don't remove them.

@Alex-302 commented on Tue Nov 13 2018

• iOS
  $network
$empty / ,empty
$csp=
$webrtc
• Safari
  $empty / ,empty
$webrtc

For instance, for Android:

{
            "filterId": 112,
            "name": "Liste AR",
            "description": "Supplement for EasyList. Language: العربية",
            "timeAdded": "2014-06-30T10:56:55+0300",
            "homepage": "https://code.google.com/p/liste-ar-adblock/",
            "expires": 172800,
            "displayNumber": 2,
            "groupId": 7,
            "subscriptionUrl": "https://filters.adtidy.org/android/filters/112_optimized.txt",
            "version": "2.0.25.73",
            "timeUpdated": "2018-12-04T19:00:30+0300",
            "languages": [
                "ar"
            ],
            "tags": [
                1,
                9,
                10,
                38
            ]
        },

Use _optimized for Android, iOS, Safari.
Use regular versions for other platforms.

@ameshkov commented on Mon Aug 21 2017

The problem is that Samsung cannot handle ^ symbol when it means the end of URL:

For instance:
||google-analytics.com/analytics.js^ won't match https://google-analytics.com/analytics.js

To fix this issue we should add a duplicate rule with | instead of ^ for every rule matching this regexp:
^(@@)?\|\|.+?/.+?\^(\$.*)?$

For instance, this rule:

`||google-analytics.com/analytics.js^$third-party`

Should be replaced with two:

`||google-analytics.com/analytics.js^$third-party`
`||google-analytics.com/analytics.js|$third-party`

It should be done for Content Blocker filters only

@Mizzick commented on Wed Oct 18 2017

It should be done for Content Blocker filters only

But what is Content FIlter? How to define it?

@ameshkov commented on Sun Oct 22 2017

But what is Content FIlter? How to define it?

I mean rules for iOS, Safari, and for Android Content Blocker

Invalid rule: better.com#$##header-floating\ navbar { position: absolute !important; } incorrect style: #header-floating\ navbar { position: absolute !important; }
Invalid rule: bongino.com#$##site-header\ 1 > .site-header-main { height: 167.3px !important; } incorrect style: #site-header\ 1 > .site-header-main { height: 167.3px !important; }
Invalid rule: deadline.com#$#.pmc-u-background-white.\/\/.header__bar { position: relative !important; transform: translateY(0) !important; } incorrect style: .pmc-u-background-white.\/\/.header__bar { position: relative !important; transform: translateY(0) !important; }
Invalid rule: engadget.com#$#.o-sticky_header\@tp- { position: relative !important; } incorrect style: .o-sticky_header\@tp- { position: relative !important; }
Invalid rule: sports.yahoo.com#$##atomic .Mt\(headerHeight\) { margin-top: 22px !important; } incorrect style: #atomic .Mt\(headerHeight\) { margin-top: 22px !important; }
Invalid rule: texasmonthly.com#$##\#novella-header { position: relative !important; top: 0 !important; } incorrect style: #\#novella-header { position: relative !important; top: 0 !important; }
Invalid rule: theculturetrip.com#$#.header-menu\.styled__HeaderMenu-s8ctnhx-0.flipboard-remove { position: absolute !important; } incorrect style: .header-menu\.styled__HeaderMenu-s8ctnhx-0.flipboard-remove { position: absolute !important; }
Invalid rule: thedeepmag.ca#$##\#novella-header { position: absolute !important; top: 0 !important; } incorrect style: #\#novella-header { position: absolute !important; top: 0 !important; }
Invalid rule: thenation.com#$##\'navbar\' { position: relative !important; height: 130px !important; } incorrect style: #\'navbar\' { position: relative !important; height: 130px !important; }
Invalid rule: yahoo.com#$##atomic, #Navigation, #UH-0-Header, #UH-1-NavLite, #UH, #UH-0-SportsUH-0-Header, #UH-1-DesktopNav, #YDC-SecondaryNav, #YDC-UH-Stack, .Pos\(f\)\!, .Mb-0.Zoom-1.header.js-applet.M-0, .action-bar, .y-header, .yui-sv-hd.ct-box-hd { position: relative !important; top: 0 !important; transform: none !important; box-shadow: none !important; } incorrect style: #atomic, #Navigation, #UH-0-Header, #UH-1-NavLite, #UH, #UH-0-SportsUH-0-Header, #UH-1-DesktopNav, #YDC-SecondaryNav, #YDC-UH-Stack, .Pos\(f\)\!, .Mb-0.Zoom-1.header.js-applet.M-0, .action-bar, .y-header, .yui-sv-hd.ct-box-hd { position: relative !important; top: 0 !important; transform: none !important; box-shadow: none !important; }
Invalid rule: yahoo.com#$#.HideBottomBar .HideBottomBar_Mt\(-35px\), .Trs\(subnavTransition\).T\(0\).UhHideSubnav_Mt\(-50px\) { margin-top: 0 !important; } incorrect style: .HideBottomBar .HideBottomBar_Mt\(-35px\), .Trs\(subnavTransition\).T\(0\).UhHideSubnav_Mt\(-50px\) { margin-top: 0 !important; }
Invalid rule: yahoo.com#$#.HideBottomBar .HideBottomBar_Op\(0\) { opacity: 1 !important; } incorrect style: .HideBottomBar .HideBottomBar_Op\(0\) { opacity: 1 !important; }

Invalid rule: engadget.com#$#.o-sticky_header\@tp- { position: relative !important; } incorrect style: .o-sticky_header\@tp- { position: relative !important; }

When Apple completely phases out old-style extensions

Example:

Invalid selector: ##################
Invalid selector: ############
Invalid selector: ##############

https://travis-ci.org/AdguardTeam/FiltersRegistry/builds/406610722

Due to the starting comma, AG for Win seems to be mistinterpreting this rule (and applying to all domains):

,101greatgoals.com,activistpost.com,allthetests.com,ancient-origins.net,androidcentral.com,batmanstream.com,bibme.org,biology-online.org,blacklistednews.com,breakingisraelnews.com,chicagotribune.com,citationmachine.net,codeproject.com,collectivelyconscious.net,colourlovers.com,connectedly.com,convert-me.com,convertcase.net,crackberry.com,dailygalaxy.com,datpiff.com,destructoid.com,dictionary.com,diffen.com,drudgereport.com,dwatchseries.to,easybib.com,eurweb.com,freewarefiles.com,fullmatchesandshows.com,gelbooru.com,genfb.com,gorillavid.in,grammarist.com,imagepearl.com,imore.com,indiewire.com,jerusalemonline.com,jewsnews.co.il,kizi.com,kshowonline.com,magnetdl.com,mmorpg.com,movpod.in,nba-stream.com,newsarama.com,nydailynews.com,omgcheckitout.com,onwatchseries.to,opensubtitles.org,playbill.com,pocketnow.com,readmng.com,roadracerunner.com,robinwidget.com,rule34.xxx,sh.st,teslacentral.com,tetrisfriends.com,the4thofficial.net,thebarchive.com,thefreethoughtproject.com,thewatchseries.ac,trifind.com,unblckd.org,usherald.com,veteranstoday.com,videotoolbox.com,vrheads.com,watchseries.sk,windowscentral.com,wnd.com,xda-developers.com##[src*="base64"]

We need to make sure that the $domain modifier is properly composed.

These modifiers STEALTH, EXTENSION, MP4 are supported by the apps only.
REPLACE is supported by the apps and Firefox extension.
PING is unsupported by all products.
The rules with these modifiers must be excluded from the filters.

@seanl-adg commented on Mon Jul 10 2017

According to https://adblockplus.org/development-builds/new-syntax-for-advanced-element-hiding-rules, ABP will use #?# instead of ## to distinguish element hiding rules which make use of -abp-has and -abp-properties pseudo classes.

We need to convert #?# to ##, -abp-has(...) to [-ext-has='...'], and discard if -abp-properties is used.

@Alex-302 commented on Wed Oct 25 2017

Can we close it?

@ameshkov commented on Thu Nov 16 2017

@Alex-302 nope, we'd better do it

https://easylist-downloads.adblockplus.org/easylist.txt

The rules like #$#abort-current-inline-script matchmedia

By the way, how does it pass CSS rules validation?

!+ PLATFORM(ext_ublock)
||code.directadvert.ru/data/*.js$script,redirect=noopjs,important

$redirect= should be allowed, if the rule has hint !+ PLATFORM(ext_ublock)

UPD: error message in compiler:
Invalid rule options: ||code.directadvert.ru/data/*.js$script,redirect=noopjs,important

@Alex-302 commented on Mon Dec 12 2016

Current behaviour: The rules, which are placed after invalid rule, are not working.
Example of issue:
https://forum.adguard.com/index.php?threads/resolved-missed-ad-on-extratorrent.17401/
Screenshot:

Invalid rule:

etmirror.com,etproxy.com,extra.to,extratorrent.cc,extratorrent.works,extratorrentlive.com,extratorrentonline.com##td > table[style*=border: 0px"]

table[style*=border: 0px"] - missed the first quotes

@ameshkov commented on Thu Jun 15 2017

I suggest writing a nodejs script which will do this.

1. You can use nodejs and phantomjs to run an embedded browser and compile a CSS selector.
2. You can also detect extended css rules and validate them as well using extcss library.

For example rule like this:
samdan.com.tr,esquire.com.tr##div[data-mbzone="Textlink" i] > div#id_d_textlink
The problem is with i in div[data-mbzone="Textlink" i]

@Alex-302 commented on Tue Oct 02 2018

Steps to reproduce

1. Open http://example.org/
2. Add:

@@||example.org^$extension
||example.org/favicon.ico

Actual behavior

example.org/favicon.ico is not blocked.

Expected behavior

example.org/favicon.ico must be blocked.

Your environment

• AdGuard AdBlocker extension version: 2.10.11
• Environment name and version: (e.g. Chrome 59.0.3071.115)

https://filters.adtidy.org/android/filters.json

https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters#hints-1

It seems that optimized filter for iOS contains Extended CSS rules which are not supported by iOS, so I think we can remove them.
For example this rule siterankz.com##.chartcontainer > .infotable:has(ins.adsbygoogle) here - https://filters.adtidy.org/ios/filters/2_optimized.txt

@Alex-302 commented on Tue Feb 20 2018

Example:

/^##.adsbox$/

• Actual result:

###adsbox is excluded by: /^##.adsbox$/
###adsbox-left is excluded by: /^##.adsbox$/
###adsbox-right is excluded by: /^##.adsbox$/
###adsbox1 is excluded by: /^##.adsbox$/
###adsbox2 is excluded by: /^##.adsbox$/
###adsbox3 is excluded by: /^##.adsbox$/
###adsbox336x280 is excluded by: /^##.adsbox$/
###adsbox4 is excluded by: /^##.adsbox$/
###adsbox728x90 is excluded by: /^##.adsbox$/
##.adsbox is excluded by: /^##.adsbox$/
##.adsbox-square is excluded by: /^##.adsbox$/
##.adsboxBtn is excluded by: /^##.adsbox$/
##.adsbox_300x250 is excluded by: /^##.adsbox$/
##.adsboxitem is excluded by: /^##.adsbox$/

• Expected result:

###adsbox is excluded by: /^##.adsbox$/

@Alex-302 commented on Wed Jun 21 2017

Feature requests

• rules validating:
  For example:
  ||example.com###advert - the rule is invalid. || are unnecessary for element hiding rules;
  ||example.com^$$domain=domain.org - two option separators.
  ||example.com^$domain=domain.org,domain2.com - invalid domain separator. Must be |.
  If it possible - add validation of CSS rules.
• black list for the files, which must be processed(the list of the files that should be sorted);
• committing changes to the repository;
• show request before/after sorting ;
• print all processed directories;

Currently our script has a bug with sorting order. Sometimes the order of sorting is violated - sorting of strings occurs alphabetically. Expected sorting process:

• sorting of element hiding rules in descending order;
  • sorting by CSS selector in descending order;
  • merging the rules with the same CSS selector;
  • removing duplicated domains;
• sorting of URL rules
  • sorting the rules in descending order;
  • sorting of the rule modifiers(domain=, app=, protobuf=, 'replace=' must be at the and of modifiers list)
  • sorting and merging domains list if the rules contain only $domain= modifier
  • removing duplicates;
• types of rules that must be ignored(allow sorting as string, but don't change them):
  JS(#%#), CSS(#$#), content replacing rules($replace), protobuf($protobuf=), Content Security Policy ($csp), application modifier for Android and Windows('$app=')

Comments ! and hints !+ are separators. Rules that are between the comments should be sorted separately from the rest.

Addition about the hints: the rule with hint should not be sorted - !++rule .
For example:

!+ HINT
some.rule_zzz
some.rule_bbb
some.rule_aaa

result of sorting is

!+ HINT
some.rule_zzz
some.rule_aaa
some.rule_bbb

@vozersky commented on Mon Aug 14 2017

We might also need to add a duplicate check to the script:
AdguardTeam/AdguardForWindows#1850 (comment)

@Alex-302 commented on Wed Aug 09 2017

@vozersky

@ameshkov commented on Mon Aug 14 2017

@Alex-302 need a better example.

For instance, give us a list with 50 rules and show how it should be sorted.

@Alex-302 commented on Mon Aug 14 2017

Before

!
||graph.facebook.com^$app=jp.gocro.smartnews.android
||graph.facebook.com^$app=com.zynga.crosswordswithfriends
||graph.facebook.com^$app=com.urbandroid.sleep
! razlozhi.ru|yandex.by|yandex.com.tr|yandex.kz|yandex.ru|yandex.ua|syl.ru
/:\/\/(otzovik.com)\/[a-zA-Z0-9-_]*==\//$script,domain=otzovik.com
/:\/\/(otzovik.com)\/[a-zA-Z0-9-_]*=\//$script,domain=otzovik.com
!
!+ PLATFORM(iOS, ext_safari)
||oops.rustorka.com^
!
://*.rumedia.ws^$empty,domain=~rumedia.ws
rustorka.com##a[href^="http://www.gearbest.com/"]
rustorka.com##div[align="center"] > a > img
/(rustorka.com\/forum\/misc\/js\/(?!ifix)(?!ajax)(?!main)(?!fancybox)(?!scrolltopcontrol)(?!jquery)(?!ct)(?!bbcode))/$domain=rustorka.com
@@||rustorka.com/forum/shoutbox_view.php
!+ PLATFORM(ext_ff, ext_opera, ext_ublock)
rustorka.com##.post_body > div[align="center"] > a > img
!
myfin1.by##.menu_level_1 > li.left_bt
myfin2.by##.menu_level_1 > li.left_bt
myfin3.by##.menu_level_1 > li.left_bt
myfin4.by##.menu_level_1 > li.left_bt
bgoperator.ru##.l-index__side_a > canvas[width="188"][height="312"]
razlozhi.ru##._23 > div._68._36
tourister.ru##body > div[onclick="TourWindowCClose();"]
torrent-games.net###ubm_div
inosmi.ru##.banner
||r.mradx.net/img/D2/C79060.mp4
||r.mradx.net/img/20/9D02B4.ogg
esports.mail.ru##.rb-video-widget
esports.mail.ru##a.b-bg[target="_blank"]
seedoff.cc##div[id*="Composite"]
||r.mradx.net/img/D2/C79060.mp4
||r.mradx.net/img/20/9D02B4.ogg
esports.mail.ru##.rb-video-widget
esports.mail.ru##a.b-bg[target="_blank"]
seedoff.cc##div[id*="Composite"]
||animespirit.ru/banners/
fotokto.ru###pageContainer > div[class="m2"]
||simpsonsua.com.ua/photos/baner-reklama/
||cy-pr.com/images/stat/tr.png
miytvideo.ru##body > noindex
goldenshara.net##center > a[href="/goldenshara.net.php?url"] > img
zaycev.online##div[data-cookie="mainBanner"]
||bazr.ru/vast?$xmlhttprequest
||image.winudf.com/*/upload/promopure/$domain=apkpure.com
24auto.ru##.brrr_place
mp3cc.com###headerBanner
mp3cc.com##.playlist-ad
mp3cc.com###footerBanner
anistar.me##body > div[class^="heade-"]
chatovod.ru##.chatlist > tbody > tr[class="bold"]:not([class^="chatitem"])
||chatovod.ru/i/promo2/workle.png

After

!
||graph.facebook.com^$app=jp.gocro.smartnews.android|com.zynga.crosswordswithfriends|com.urbandroid.sleep
! razlozhi.ru|yandex.by|yandex.com.tr|yandex.kz|yandex.ru|yandex.ua|syl.ru
/:\/\/(otzovik.com)\/[a-zA-Z0-9-_]*==\//$script,domain=otzovik.com
/:\/\/(otzovik.com)\/[a-zA-Z0-9-_]*=\//$script,domain=otzovik.com
!
!+ PLATFORM(iOS, ext_safari)
||oops.rustorka.com^
!
/(rustorka.com\/forum\/misc\/js\/(?!ifix)(?!ajax)(?!main)(?!fancybox)(?!scrolltopcontrol)(?!jquery)(?!ct)(?!bbcode))/$domain=rustorka.com
://*.rumedia.ws^$empty,domain=~rumedia.ws
@@||rustorka.com/forum/shoutbox_view.php
rustorka.com##a[href^="http://www.gearbest.com/"]
rustorka.com##div[align="center"] > a > img
!+ PLATFORM(ext_ff, ext_opera, ext_ublock)
rustorka.com##.post_body > div[align="center"] > a > img
!
mp3cc.com###footerBanner
mp3cc.com###headerBanner
fotokto.ru###pageContainer > div[class="m2"]
torrent-games.net###ubm_div
razlozhi.ru##._23 > div._68._36
inosmi.ru##.banner
24auto.ru##.brrr_place
chatovod.ru##.chatlist > tbody > tr[class="bold"]:not([class^="chatitem"])
bgoperator.ru##.l-index__side_a > canvas[width="188"][height="312"]
myfin1.by,myfin2.by,myfin3.by,myfin4.by##.menu_level_1 > li.left_bt
mp3cc.com##.playlist-ad
esports.mail.ru##.rb-video-widget
esports.mail.ru##a.b-bg[target="_blank"]
anistar.me##body > div[class^="heade-"]
tourister.ru##body > div[onclick="TourWindowCClose();"]
miytvideo.ru##body > noindex
goldenshara.net##center > a[href="/goldenshara.net.php?url"] > img
zaycev.online##div[data-cookie="mainBanner"]
seedoff.cc##div[id*="Composite"]
||animespirit.ru/banners/
||bazr.ru/vast?$xmlhttprequest
||chatovod.ru/i/promo2/workle.png
||cy-pr.com/images/stat/tr.png
||image.winudf.com/*/upload/promopure/$domain=apkpure.com
||r.mradx.net/img/20/9D02B4.ogg
||r.mradx.net/img/D2/C79060.mp4
||simpsonsua.com.ua/photos/baner-reklama/

@Alex-302 commented on Tue Aug 22 2017

Feature request:

add support of blacklist file with dead domains list, which must be removed from the rules and the rules which contains only one domain.

@Mizzick commented on Mon Oct 30 2017

@Alex-302 pls give us more details about content rules?
example.org$$script[data-src="banner"]

@Alex-302 commented on Mon Oct 30 2017

@Mizzick
What exactly?
Please take a look
https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters#html-filtering-rules-1

@Alex-302 commented on Fri Nov 16 2018

Travis failed to build filters, because of hufilter rules:

prohardver.hu##p:-abp-contains(Amikor a szoftver illeszkedik a meglévő üzleti folyamatokhoz.)
prohardver.hu##div[id="center"] > div:-abp-has(div):-abp-has(a):-abp-has(Mostantól bármely bank ügyfelének elérhető a mobilfizetés élménye.)

Excluded AdguardTeam/FiltersRegistry#123

@ameshkov commented on Mon Apr 10 2017

We currently do these transformations in the programs code. I guess it'd be easier for devs if these transformations were done on the server side.

@Alex-302 commented on Tue Apr 11 2017

Can we add unicode domains to the filters?

@ameshkov commented on Fri Apr 14 2017

Can we add unicode domains to the filters?

Yep, what I want to be done is automatically transform them to punycode on the server side.

@Alex-302 commented on Fri Apr 14 2017

ok, got it.

#?#
#$?#
#@$?#
#@?#

See the details here:
AdguardTeam/AdguardBrowserExtension#1444

It says that it's not a valid filter as it misses the necessary [Adblock Plus 2.0] (and maybe smth else):
https://uploads.adguard.com/up04_68i9v_Adblock_Plus_Options.png

Let's make it compatible.

and ,cookie
It is unsupported in ios, ext_safari, ext_android_cb, ext_ublock

See the details here:
AdguardTeam/ContentBlocker#107 (comment)

Problematic rules:

apkmirror.com##body .google-ad-leaderboard-smaller:style(position: absolute!important; left: -4000px!important; display:block!important;)
apkmirror.com##body .google-ad-square-sidebar:style(position: absolute!important; left: -4000px!important; display:block!important;)

Both rules should've been transformed into the #$# syntax, but they didn't:

apkmirror.com#$#body .google-ad-leaderboard-smaller { position: absolute!important; left: -4000px!important; display:block!important; }
apkmirror.com#$#body .google-ad-square-sidebar { position: absolute!important; left: -4000px!important; display:block!important; }

https://github.com/AdguardTeam/FiltersDownloader

Invalid selector: filmweb.pl#@#+js(abort-current-inline-script.js, Math.random, adbDetect)
Invalid selector: filmweb.pl#@#+js(setTimeout-defuser.js, notDetected)

$first-party to $~third-party
,first-party to ,~third-party
Examples:
||www.ynet.co.il^$important,websocket,first-party,domain=www.ynet.co.il
||zive.cz^*+$script,first-party

For example these rules shouldn't be available in filters for Android app - https://github.com/AdguardTeam/AdguardFilters/blob/3a61398e615de3a475c1c4ba888db30520efa14a/RussianFilter/sections/antiadblock.txt#L1382-L1428, but they are still here - https://filters.adtidy.org/android/filters/1_optimized.txt

Rules like these:
|http*$domain=|example.org
|http*$script,domain=|example.org
should be treated as invalid or converted to:
|http*$domain=example.org
|http*$script,domain=example.org
otherwise they are applied to all websites due to domain=|

It seems that some correct CSS rules with \ are treated as "Incorrect style", for example:

indircafe.net#$#a[class][onclick^="window.open(\""][rel="nofollow"] { position: absolute!important; left: -3000px!important; }
4pda.ru#$#body > * > * > div > *:matches-css(height: 90/\d) { position: absolute!important; left: -3000px!important; }

https://github.com/AdguardTeam/FiltersRegistry/blob/master/filters/filter_14_Annoyances/diff.txt#L2
https://github.com/AdguardTeam/FiltersRegistry/blob/master/filters/filter_1_Russian/diff.txt#L4

I may be wrong, but I guess that it's related to this - #47

EDIT

I changed first rule, so it should be okay.
More examples (search for ! Incorrect style:) - https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/ThirdParty/filter_201_WebAnnoyancesUltralist/diff.txt

Invalid rule: $cookie=cookie_name option: cookie
and ,cookie maybe

Invalid rule: ||delivery.tf1.fr/pub$media,rewrite=abp-resource:blank-mp3,domain=tf1.fr option: rewrite

720hd.club#?##all:style(margin-top: 0 !important) -> 720hd.club#$?##all { margin-top: 0!important }
#?#$ must be #$?#

Redundancy check for Adblock Plus rules + browser extension
https://mega.nz/#!xBlyEQLR!P7LqU02x5cNYer6-zvBLdtF7DFXwFraRNy9y6Q5msCE
https://mega.nz/#!hd1mzagK!cJ3YbLyDSdjkrGx6hTktlNFfmOwIQoqpQ5c8G4IQF0U

another domain checker:
https://github.com/funilrys/PyFunceble

Additionally to the build script we should provide JSON scheme validation (for monitoring purposes).

• https://json-schema.org/
• https://www.npmjs.com/package/jsonschema

Here's what needs to be validated:

• filters.json (for every platform)
• filters_i18n.json (for every platform)
• number of filters in the filters.json (hardcode >= 80)

How it should work:
node validate.js path_to_platforms_directory

For example, this rule breaks the sites when used browser extension, but it exists in Liste-FR. It must be excluded from all filters.

!+ PLATFORM(windows, mac, android)
||piguiqproxy.com^$third-party
``

@ameshkov commented on Wed Feb 08 2017

Adguard scriptlet is a named script, which can be used further by JS or basic rules. Scriptlets are configured in a special "scriptlets" configuration file and maintained by us.

Why do we need it

1. JS rules are simply hard to support. We currently have more than 700 JS rules in all the AG filters, and the number is growing.
2. Some JS rules are almost identical. For instance, Object.defineProperty(window) is used by ~200 rules.
3. AMO and Opera review. It will make reviewers life much easier.

I don't say we should get rid of JS rules completely, though. It should be allowed in the user filter, just not in other filters.

Scriptlets configuration file will be updated automatically along with the filters. However, in the case of AMO and Opera, automatic update will be disabled so that we comply with remote scripts policy.

Scriptlets usage syntax

There are two options.

Injecting and executing scriptlet on a per-domain basis:

example.org#&#scriptlet:scriptletName(param1, param2);

Replacing JS file content with a scriptlet

I am not sure about it, though. I suppose we should start with injection only.
||example.org/file.js$replace=scriptlet:scriptletName(param1, param2);

Scriptlets configuration file

Here is an example of the scriptlet configuration file.

/**
 * Scriptlets configuration object.
 */
var scriptlets = {

	// Scriptlets file version
	version: "1.0.0"
};

/**
 * Empty scriptlet, just to show you the idea.
 *
 * @param context 		Scriptlet execution context
 */
scriptlets["empty"] = function(context) {
	// context.log 		Prints message to the console
	// context.version	Scriptlets configuration file version
	context.log(context.version);

	// scriptlet parameter
	console.log(context.params[0]);
};

/**
 * Defines "window" property.
 *
 * Example: 
 * example.org#%#scriptlet:defineWindowProperty("test", 1);
 */
scriptlets["defineWindowProperty"] = function(context) {

	var propertyName = context.params[0];
	var propertyValue = context.params[1];

	Object.defineProperty(window, propertyName, {
		get: function() {
			if (typeof(propertyValue) === "undefined") {
				return;
			}
			return propertyValue;
		},
		set: function() {}
	});
};

@ameshkov commented on Wed Feb 08 2017

@atropnikov @seanl-adg @Alex-302
Guys, I need your feedback on this proposition.

@Alex-302 commented on Wed Feb 08 2017

Are scriptlets will be hardcoded for AMO version as before?

Replacing JS file content with a scriptlet

is it possible to replace only specific function in the file(for example in some library)?

@ameshkov commented on Wed Feb 08 2017

Are scriptlets will be hardcoded for AMO version as before?

The difference between AMO, Opera, and other versions will be that scriptlets configuration file won't be updated automatically.

is it possible to replace only specific function in the file(for example in some library)?

No, that's possible with $replace rules only.

@ameshkov commented on Wed Feb 08 2017

I've found the first problem with the proposed syntax; it is not backward compatible with the old JS rules, so it will break previous versions of Adguard.

We should "mark" them differently, use #&# mark instead of #%#. I've updated the first message.

@ameshkov commented on Wed Apr 26 2017

What we also need:

1. Add support for the base64-encoded resources (transparent.png and such)
2. Scriptlet should have more props, content type for instance.
3. We should also handle Content-MD5 header properly

@ameshkov commented on Thu Jul 27 2017

@seanl-adg

1. We should have unit-tests for scriptlets.
2. Consider having scriptlets dependencies. Do we actually need it?
3. I suppose that base64-encoded resource redirects should be a separate structure, otherwise it may cause confusion.

For example:
@@||fwmrm.net/crossdomain.xml$xhr,domain=viafree.dk|viafree.no
https://github.com/AdguardTeam/FiltersRegistry/blob/c69e7994d7ef834cec61b911e59fcc4aeaeb8889/filters/ThirdParty/filter_249_NorwegianList/diff.txt#L64

$xhr should be converted to $xmlhttprequest and probably ,xhr to ,xmlhttprequest

Maybe other modifiers should be also converted - https://github.com/NanoAdblocker/NanoCore2/blob/master/notes/filter_syntax.md#convenience-options

css to stylesheet and frame to subdocument