xeverous / filter_spirit Goto Github PK

There should be a way to specify only lower or upper price bound of the query. Currently if you want 100c+ divination cards you can write $divination | price(100, 999999) however such workaround:

• is not very appealing, unintuitive to use
• has a hidden limit defined by the implementation (max integer value)
• may cause hidden problems (when eg a card giving mirror is worth some huge number in chaos)

Current idea: _ keyword, indicating "no value". That is, $divination | price(100, _). One might propose price(100,) but it might look weird and I would like to avoid optional tokens in the grammar.

Any thoughts?

FS GUI application (which does not exist yet) should have a filter block preview window. Basically a window where you can see a single filter block in action, without having to load it in-game.

Possible future additions:

• RGB color picker + fast access to commonly used colors
• access to variables in the filter's code (if loaded an FS template filter)
• font size slider
• sound picker (+ play option)
• beam and minimap icon selection

At the bottom of this post:

https://www.pathofexile.com/forum/view-thread/2771031

Description and/or example
There is already a condition for ShaperMap <Boolean>, would be very nice if we could get the same for ElderMap <Boolean>

Currently FS supports only poe.watch. It would be good to also support poe.ninja.

Things to do:

• Implement parsing of JSON files returned by poe.ninja (poe.ninja API documentation: https://poe.ninja/swagger/index.html)
• Unify the interface - the current implementation is too strongly coupled with poe.watch data format. FS should have a unified format that is agnostic to the API being used
• Find similarities and differences between ninja and watch. Decide how to deal with them.

Generate random loot from specific encounters. Howering over each loot should point display which code line catches that item.

Right now the UI features many basic buttons for rares and items of specific class.

Loot samples (this list will be continuously edited to add more):

• Delve
  • Fossilized Remains
  • ??? (drops uniques)
  • Resonator's Cache
  • Bloodstained Lockbox (sacrifices, corrupted items)
  • Murmuring Lockbox (essences)
  • Sealed Lockbox (???)
  • Heavy Armaments (rare garbage)
  • Abandoned Armaments
  • Abandoned Amulets
  • Abandoned Jewelery
  • city map chest
  • city jewelery chest
  • city currency chest
• Incursion
  • Ancient Hoard (???)
  • Dusty Charts (maps)
  • Vaal vessel (quality/vaal gems)
  • Stockpile (storage room)
  • War Hoard (corrupted equipment)
  • currency chests (common orbs, common orbs + harbinger orbs, quality orbs)
  • Flashpowder Stash
  • Stone of Passage
  • chests that "contain a valueable item"
• Maps
  • blue monster pack
  • regular unique monster drop
  • sextanted T16 boss MF cull (lots of rares, sacrifice fragments, corrupted items, possibly maps)
  • Shaper guardian drop (fragments + uniques)
  • Elder guardian drop (fragments + uniques)
  • Vaal Winds drops (lots of corrupted items, 6Ls)
  • The Wealthy Exile drop (Rogue Exile dropping only rares) (guuaranteed full chaos recipe set)
• Labirynth
  • Silver/Golden/Treasure keys
  • gem treasure chest
  • divination treasure chest
  • rares treasure chest
  • artisan treasure chest
• Legion/Blight chests
  • veiled items
  • shaper/elder rares
  • lab items (enchanted items, offering)
  • essences
  • currency
  • oils
  • incubators
  • blighted maps
  • sealed prophecies
• Strongboxes
  • artisan (quality currency + The Master Artisan)
  • arcanist
  • diviner (cards + stacked deck)
  • ornate/large
  • cartographer
• Abyss
  • Abyssal Trove (rares + jewels)
  • Abyss boss/Stygian Spire (abyss uniques, belt)
• boss encounters / unique maps
  • Atziri (uniques, + a chance to drop prophecized vaults and regal orb)
  • Uber Atziri
  • Pale Court (uniques + rares)
  • Shaper
  • Elder
  • Uber Elder (can drop shaper and elder items at the same time)
  • The Beachhead drop (shards + unique pieces)
• leveling
  • 6s, 5L, 6s 6L
  • 4Ls
  • chrome items
  • hammers
  • low-lvl recipe stuff (sceptres, rings)
  • Hillock (Tabula, Oni-Goroshi)
• Breach (splinters, uniques, rings, 6s)

The goal: we want a feature that eliminates code duplication of actions that frequently are used together (eg text/border/background color, beam + minimap icon).

We also want to be careful with its complexity:

/u/besplash: I felt like your initial example code was very easy and intuitive to use, but your macros are making it more complex than it has to be. The Tag system from FilterBlast on the other hand doesn't need any new kind of programming knowledge which would be way better for people that have 0 programming experience like most of the playerbase.

Example code, containing duplicated actions:

currency = RGB(213, 159, 15) # (yellow)

Class "Currency" {
    # some common styles for all currency here...

    BaseType "Chaos Orb" {
        SetTextColor black
        SetBackgroundColor currency
        SetFontSize 42
        Show
    }

    # other blocks between [...]

    BaseType "Orb of Fusing" {
        SetTextColor black
        SetBackgroundColor currency # again with text color, duplicated in chaos orb
        Show
    }

}

If there was no "other blocks between [...]", we could just create another scope that inherits these but in this scenario we do not have A, B, B, C order but something like A, B, C, B. Bs are separate so we can not enclose them in { [...] }.

action code duplication reduction feature - current idea:

currency_colors = {
    SetTextColor black
    SetBackgroundColor currency
}

beam_and_icon = {
    SetBeam yellow
    SetMinimapIcon MinimapIcon(0, star, yellow)
    # also possible here: ApplyTags ...
}

# [...]    
BaseType ["Orb of Fusing", "Orb of Alchemy"] {
    ApplyTags [currency_colors, beam_and_icon]
    SetFontSize 42
    Show
}

BaseType ["Jeweller's Orb"] {
    ApplyTags [currency_colors]
    Show
}

I think this syntax is faily simple while being consistent with the rest of the grammar. Tags could only be used to group actions, no conditions. Reason:

How filters work: an item goes through it and receives the style of the first block that matches specified conditions, all further blocks are ignored. So you want the most restrictive blocks first (most nested), then decreasing in strictness unitll some generic rule like "hide all leftover magic and rare garbage" hides everything at the bottom. Allowing to override actions is no problem: these do not collide. Filter with styles/colors 1, 2, 3, 4 ..., 97, 98, 99 will work just like 1, 97, 99, 50, 4, 98, 3. But for conditions - if you have blocks in order like A, Z, K, D, Y, C (where A means most restrictive block and Z least restrictive) everything past Z will not work because some items will be catched by A and everything else will be caught by Z, leaving no items for further blocks. If we disallow overriding/tagging conditions in nested blocks, we guuarantee that the filter structure has never "the A, Z, B issue" and always goes from most-specific-item to least-specific-item.

Looking for any opinions & propositions:

• How this feature should be named? Tags do not seem to be a good name for me, this term should be saved for features related to filter strictness/variants. Maybe compound actions, grouped actions?
• Grouped actions are applied to blocks by something like Apply [grp1, grp2] syntax. Grouped actions can also be applied to other grouped actions.
• Block grouping conditions, allow only to group actions. See reasoning above. Is this fine?

Description and/or example
Need to add Prophecy as a condition that allows strings or arrays in case someone dont want to pull the tier lists based on poe.ninja, but would rather tier them self. Self made tiers can be very useful for SSF where cheap prophecies/uniques/fossil/incubator etc. can be very useful even tho they are not worth a lot in trade league. This matters for all of the stuff that can be pulled automatically via price queries.

I understand that the price queries are one of the core functions of this tool, but personally as a HCSSF player, they help me very little as i want to tier everything my self as i dont trust the economy and me to have the same value on certain items.

Code Example
This code snippet i took from NeverSink to display how his filter does prophecies

	Class Currency
	BaseType "Prophecy"
	Prophecy "A Dishonourable Death" "Fated Connections" "Lost in the Pages" "The Bowstring's Music" "The King's Path" "The Queen's Sacrifice" "Trash to Treasure"
	SetFontSize 45
	SetTextColor 130 25 255 255             # TEXTCOLOR:	 Blessing
	SetBorderColor 130 25 255 255           # BORDERCOLOR:	 Breach
	SetBackgroundColor 255 255 255 255      # BACKGROUND:	 T0 Drop
	PlayAlertSound 6 300                    # DROPSOUND:	 T0 Drop
	PlayEffect Red
	MinimapIcon 0 Red Star

If i try the example under, i get a parse error.

BaseType "Prophecy"{
        Prophecy ["The Queen's Sacrifice", "Fated Connections", "Trash to Treasure", "The King's Path", "Lost in the Pages"]{
            SetFontSize font_max
            Show
        }

    }

New actual filter language feature - exact string matches

https://www.pathofexile.com/forum/view-thread/2654584

SocketGroup == GG BBB RR works as it looks like. Currently FS does not support such thing (SocketGroup has separate implementation from any string-based condition).

Few things are absolutely sure:

• Some unique items share their base with other uniques. This makes a lot of drops ambiguous ("Leather Belt") and makes filtering jewels and amulets pretty much impossible. We can not really fix this issue unless GGG implements something to disambiguate. It is a responsibility of filter's creator that ambiguous drops are labeled in special way to indicate the player has to verify them manually.
• Many items can have unreliable prices or no price information at all. Both poe.ninja and poe.watch report this by "low confidence" or simply by not providing price data for such item. It is a responsibility of filter's creator to ensure existence of no-price-query generic blocks and not writing entire filter only on the data supplied by market info.

However, there are some things we can work on:

• 1) We can fight price fixing by storing previous item prices, checking whether there is a significant difference. I'm not sure whether this should be done, because:
  • Economy overview sites (poe.ninja, poe.watch) generally do not store/report past item price information which makes it hard to detect that a specific item price has changed dramatically. And I do not want Filter Spirit to implement its own item price database.
  • Some items will change their value dramatically, eg by "the Mathil effect" or some balance patch
  • A lot of items (especially certain prophecies) have very unstable prices at the league start and it's completely normal
• 2) Both sites report prices both in exalted and in chaos. Which should be used inside filters? Implement and allow both?
• 3) How to deal with ambiguous names? Given code like $uniques | price(10, 1000) - should it output Sadist Garb when Inpulsa is expensive but Tinkerskin is 1c? Should we add another filter like | ambiguous(true)? We need a feature to let users decide what to do when multiple uniques share the same base type and some get in price range while others do not.

My current plans are:

• 1) Sites already report confidence which should be enough. Just implement an option to treat all low-confidence price items as unknown value
• 2) Go with chaos. In case of exalted, majority of items would have a price in range 0.01 - 0.5 which makes it hard to work with. Values 1 - 100 feels much better. We can always add ex-based price querying later.
• 3) ???

Some users are generating filters intended for SSF. They do not use any price queries. Having to download data or read any existing saves is an unnecessary requirement for such filters.

Hence, the new option (-e, --empty-data) which simply proceeds with generation using empty price data. If there are any queries in the filter, they will simply evaluate to nothing (as if no items matched the price range).

The current name is anti-idiomatic (compiler literature uses the term "symbol" for any type of identifiers). New name will also support other entities than constants (built-in functions, compound actions).

The intial syntax idea to query item price looked like this: $divination(10, 100). It is very good for trivial items that have only 1 property (base type name): cards, prophecies, fragments, fossils and few others.

But some items have multiple properies, where each specific combination of them can significantly impact the price:

• gems: quality, level, corrupted
• bases: ilvl, influence
• maps: tier

The current idea is a functional-language-like syntax which allows to apply multiple filters to one query: $bases | ilvl(x) | influence(shaper) | price(10, 100).

Current plans for behaviour:

• The expression should still return an array of strings
• It should be an error if some filters are missing (eg no influence specified for bases)
• The order of filters should have no impact. All of tested items must satisfy all filters.

Things to discuss:

• Should some filters be optional (eg corrupted)? Or should we require the code to explicitly specify that we do not care (eg | corrupted(_))? I would favor "explicit better than implicit".
• Which item types should require which filters? (see the first list in this post)
• Name this | price(10, 100) feature somehow? "Filter" is too conflicting with everything else.
• The current idea still has some code duplication:

x = 86
BaseType $bases | ilvl(x) | influence(shaper) | price(10, 100)
ShaperItem true # How to avoid this? The same condition is already in the query.
ItemLevel x     # How to avoid this? The same condition is already in the query.
Corrupted false

There is still some code from before 3.9 Atlas update, having some assumptions that are no longer true:

At worst, some items will be incorrectly considered as invalid (and ignored) but since FS does not yet support querying for complex items (see #10) this does nothing more than producing some badly looking logs.

Description
When compiling a block that has the diamond MinimapIcon, its missing a "d" after compiling
Steps to reproduce

SetMinimapIcon MinimapIcon(0, white, diamond)

Expected result

MinimapIcon 0 White Diamond

Actual result

MinimapIcon 0 White Diamon

It would be good to be able to specify:

1. Required FS version to generate the filter (if the version user is currently running is lower, give a clear error informing to update FS). I don't like the current grammar.
2. Version of the filter itself (so that everyone who downloads it can check if the author has released a new version). Ideally same subgrammar as for the required FS version.
3. Some metadata fields for the author? 1 generic description string or dedicated (named) fields for each thing like author, forum link, etc?

• The core directory (apart from version number) is only used by the CLI executable. Move the code to the executable.
• Some of the core directory code is too coupled with network code (eg directly using poe_watch API - this should be done on a higher lvl once both item price APIs are supported)
• The generator and lang directories contain mixed code of these 2 - either generator should be removed or contents of both lang and generator reordered

As per this GGG reddit reply the list now needs a thorough resest and update

Description
poe.ninja download does not work

Steps to reproduce
Just run the poe.ninja download option

Additional context

INFO: downloading from poe.ninja/api/data/currencyoverview?type=Currency&league=Metamorph
INFO: downloading from poe.ninja/api/data/currencyoverview?type=Fragment&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Oil&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Incubator&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Scarab&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Fossil&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Resonator&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Essence&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=DivinationCard&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Prophecy&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=SkillGem&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=BaseType&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=HelmetEnchant&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=UniqueMap&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Map&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=UniqueJewel&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=UniqueFlask&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=UniqueWeapon&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=UniqueArmour&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=UniqueAccessory&league=Metamorph
INFO: downloading from poe.ninja/api/data/itemoverview?type=Beast&league=Metamorph
ERROR: could not read the request: body limit exceeded

This is very likely the cause: https://stackoverflow.com/questions/50348516/boost-beast-message-with-body-limit

First and foremost, great program, thank you for your work. Ill describe what im seeing with regards to the eccentricities of the parser when trying to load in pre-existing filters but my question is : Is the parser a wip or is the intention to drive towards one syntax to rule them all? If the former the parser at grinding.zone is very effective accepting everything ive thrown at it thus far, maybe contacting them to pick their brain can help.

Tested this on the expedition and scourge build.

DisableDropSound will cause parsing to fail with an ambiguous "expected eoi" message if followed by True or False. Only accepting the symbol on its own despite "DisableDropSound True" being very common in the wild.

Having more than one condition eg HasExplicitMod in a block throws an error stating ...
"the same condition can not be specified again in the same block or nested blocks" ...
despite having more than one being quite common. eg for a block to match items that have one or more of SetA AND one or more of SetB by writing
...
HasExplicitMod SetA
HasExplicitMod SetB
...

SocketGroup "GGG" is perfectly valid but throws an error because the "'s exist stating that socket_spec was expected and that it got a string instead. A similar situation occurs with HasInfluence only it doesnt complain about the fact that "Shaper" etc is a string.

Thank you for your time.

We need a feature which will allow to generate filters with multiple variants/strictness levels from 1 template.

Current idea: decorate blocks (something akin to Python's decorators) and then specify which blocks should be present in each variant.

@chome_items
SocketGroup "RGB" {
        @chrome_items.small {
                Width 2
                Height <= 2 {
                        SetBorderColor color_chromatic_small
                        Show
                }

                Width 1
                Height <= 4 {
                        SetBorderColor color_chromatic_small
                        Show
                }

                SetBorderColor color_chromatic_big
                Show
        }
}

Edit: cleaner goals - a list of what is must-have:

1. The feature must support all 3 states (generate with Show, generate with Hide, do not generate). In a lot of cases you want the 3rd one - the difference is that items not explicitly hidden may be caught by later blocks - this means that syntax like @Strict(Show) @UberStrict(Hide) is not enough, because (example):
   • You want to hide armourer scraps and whetstones, simply because you don't pick up them and they have no other use.
   • You don't want to hide hammers when you resign from hammer recipe. You want to just not generate a block for them so they can still be catched by chromatic or chaos recipe blocks.
2. A lot of variants will use exactly the same block, with the only differences being disabling sound effect and changing Show to Hide. The feature should allow to avoid code duplication.
3. The feature should not require additional files. 1 filter template file => multiple filter variants. We can already generate multiple filters by choosing different league for item price data.
4. There should a way to specify the file name, since we can not put multiple filters in the same output file. Need ideas for the syntax.
5. The feature should be independent from theming, which is just a selection of colors/sounds etc. When having X different variants and Y different themes, you should be able to generate all X * Y different filters.
6. The feature should not impose requirement that each subsequent variant is a subset of another. It should be possible to have different variants, where each one has an independent set of blocks.
7. Any syntax should be declarative, not imperative.

Explained in this post: https://www.pathofexile.com/forum/view-thread/2806232

create GitHub issue templates

So far FS has only a command-line interface. This is enough to make it work and let others automate it through shell scripts but an average user will definitely expect something "viewable" and more intuitive to use. GUI also allows to implement some fancy features like loot preview, filter rules debug, price data download management etc.

I'm currently contributing to cycfi/elements which will be the GUI library used for FS.

I'm currently using Magnum and discovering what's possible with it.

Meanwhite, post your feature ideas/UI mockups here.

Current plan (outaded):
A UI consisting of few top-level tabs:

• generation
  • browse for input template file
  • browse for output directory path
  • choose which data to use (see download tab)
  • (perhaps some generation tweaks in the future: warnings, variants, themes etc)
• download
  • download from poe.ninja for <league>
  • download from poe.watch for <league>
  • view downloaded data (separated into categories like on respective websites)
  • manage past downloads (copy, delete, etc)
• loot preview/debug
  • this is a bigger topic, see issue #5
  • something similar to Filtration but more of the debug part than editing (FS uses user-written template files)
• settings
  • ???
• about
  • links to repository, author info, version etc
  • (possibly auto-update button in the future)

They can actually drop under some rare conditions:

• Corrupting Tempest
• strongbox mods
• sextant mods

Example code:

Corrupted falsee

The code is parsed as boolean_condition::corrupted, bool(false), e instead of boolean_condition::corrupted, identifier(falsee).

Identifiers can not be searched before keywords, because otherwise no keyword would ever match (any keyword satisfies identifier grammar). Since keywords are searched before identifiers, if first characters much a keyword it is treated as a keyword, leaving some garbage for further processing (here: e) which causes very unclear syntax errors. Looks like some look-ahead needs to be performed to ensure there are no unconsumed letters left.

The feature was eventually abandoned but there is still some dead code left. If it returns in the future (eg strictness/themes) the form will be different.

Description
Trying to use the HasEnchantment "some enchatment name" in a filterspirit file, but its giving error when parsing, making me belive it might not be supported at this moment?
Steps to reproduce
Try putting this in your filter:

AnyEnchantment True {
				Class "Helmets" {
					HasEnchantment "Enchantment Blade Vortex Area Of Effect 2" {
						Set $comp_enchanted_t1
						Show	
					}
				}
			}

Expected result
Should give a block that highlights that enchant only
Actual result
Fails
Additional context
(any other info if needed)

The 3.9 update item filter information is out

https://www.pathofexile.com/forum/view-thread/2687401

https://www.reddit.com/r/pathofexile/comments/e910dq/forum_announcements_conquerors_of_the_atlas_patch/fafxgvj/

Currently filter_spirit does not support the Shaper voice lines, would be nice if they could be added.

These are all the voice lines I could find with Shaper.
Used like this in normal filter syntax:

PlayAlertSound ShMirror 300

So I assume it would be like this in Filter_Spirit

    SetAlertSound ShMirror 300

Or if stored in a variable:

$alert_shaper_mirror = ShMirror 300

Description
Trying to compile a filter that uses a Rarity Normal will fail since it no such name exists for Normal, but using Rarity normal will work as expected

Steps to reproduce
Class "Body Armour" { Rarity Normal { Show } }

Expected result
Filter compiles correctly using Rarity Normal

Actual result

line 27: error: no such name exists
  Rarity Normal {
         ~~~~~~
INFO: filter generation failed

Additional context
I'm guessing it would be nice to match what the actual item filter uses (Normal instead of requiring normal) or making it handle either options?

Hey,

Love this filter generator, its what i have been looking for!

Is there any way to include only some Highlight block that is normally commented out, but i uncomment them if i need to look for a specific item for one or two maps?

So i dont have to uncomment it, then compile, then comment it out again and compile again when I got what i was looking for.

https://www.pathofexile.com/forum/view-thread/2771031

GGG just dropped a news post with a huge change for filters: Continue which lets to continue execution of the filter after the item has been matched. Open to suggestions how this should be integrated/implemented. Perhaps the syntax can be redesigned.

Remove this class.