Comments (11)
Any candidate "validation" should happen in the provider, before providing the candidate to the resolver, within find_matches
.
from resolvelib.
Right, I based my code on the example, which is not very efficient. That is not a problem in the provided code, but it becomes one in this case, where I introduce expensive operations.
The solution was pretty straightforward, make sure find_matches
returns an iterator callback, so that it can consume candidates as it goes, and make sure get_preference
does not wastefully force the consumption of candidates.
from resolvelib.
The code is on https://github.com/FFY00/python-resolver FYI.
from resolvelib.
I did consider a validation step, but the issue is most usages wouldn’t need that step, and would be “dead weight” for implmenetations that do things properly. But maybe we could have something like Resolver(dev=True)
to enable a “dev mode” that enables additional checks whether the provider is doing sane things. (It might not solve your problem though since you were consuming the candidates in get_preference
, which is out of the resolver’s control.)
from resolvelib.
Well, I don't think it should introduce that much overhead if the implementation is just a return True
, which is what would match the current behavior.
The issue here is that consuming candidates in get_preference
triggers expensive operations, even though I don't really need that information there. The current design makes returning a generator callback pretty much useless if I am actually doing something in get_preferrence
, because to do something meaningful there, I will have to consume the candidates.
If we add candidate validation, I could consume everything in get_preference
, have the resolver pick a candidate, and only then verify if it is valid or if the resolver should proceed to the next pick, instead of forcing this verification to happen for all possible candidates.
This would essentially mean that the result of find_matches
starts being possible candidates, not necessarily valid candidates, so that we can postpone the validation step and only do that if we actually need to.
from resolvelib.
The current design makes returning a generator callback pretty much useless if I am actually doing something in
get_preferrence
, because to do something meaningful there, I will have to consume the candidates.
Then you don't return a generator function? You can return a sequence instead and that will be used as-is without overhead. The generator function return type is for implementations that can't do anything meaningful to the candidate list, which is (you guessed it) pip. If you don't have that problem, you don't need to constrain yourself.
from resolvelib.
Yes, but I want the candidates to be lazily generated. Determining if a candidate is actually valid or not is expensive in this use-case, so I don't want to that unless I absolutely have to.
In the current API, I am required to only return valid candidates, this means the expensive operation will have to happen for all candidates I return. Using a generator lets me delay that operation to when the resolver decides to consume it. This is approach is fine if you don't use get_preference
, if you do, you'll consume the candidates anyway.
Giving a more concrete example, I have the following requirement, build[virtualenv]
. For a wheel to be a valid candidate, it needs to provide the virtualenv
extra, which is something I can only know by downloading the wheel and looking at METADATA
.
Right now, for each candidate that comes out of find_matches
, I need to download the wheel and make sure it provides the virtualenv
extra, because the candidate needs to be valid. I would like to be able to return all build
wheels as candidates, and only verify if the wheel provides the virtualenv
extra if the resolver decides to pick it.
Does that make sense?
from resolvelib.
Would #63 provide the same mechanism? Basically an additional hook when each candidate comes out of the iterable to ask “does this one work” and discard when it does not.
from resolvelib.
I don't know if is_satisfied_by
is used anywhere else, where it could cause issues, but if it isn't, then yes. #63 should resolve this.
from resolvelib.
is_satisfied_by
is currently called after get_dependencies
, so what’s required is basically swapping the two. Again, it would make the resolver slightly slower since the hook will be called more than it currently is, but hopefully I can make it not too bad with slight tweaks.
from resolvelib.
It's you call 😛. You can do that or introduce a new callback just to validate the candidate after picking. I am happy with either one.
from resolvelib.
Related Issues (20)
- Typing inconsistency: `AbstractProvider.get_preferences` HOT 1
- Implement lazy consumption of the candidates iterable returned by find_matches() HOT 1
- Sdist releases on pypi.org do not contain files required for tests HOT 2
- Add documentation HOT 5
- Test Failures With Packaging 22.0/23.0 HOT 1
- Drop support for Python prior to 3.7 HOT 3
- Default branch has been renamed to `main` and branch protections enabled
- Resolvelib 0.9.0 passes identifiers to Provider's `get_preference` method without accompanying PreferenceInformation HOT 5
- Please make a new release
- Required Python versions not specified in pip-readable way HOT 1
- 0.9.0: pytest is failing in 6 units HOT 3
- sdist is missing conftest.py HOT 1
- Option to only provide causes of conflict when calling get_preference HOT 2
- An example of resolution failure while it does have a valid resolution. HOT 33
- Awkward wording in docstring for `find_matches()` HOT 5
- Implement Causes Narrowing HOT 1
- Idea: Breadth-first search until reaching breadth level with unpinned requirements
- For n packages there are O(n^2) calls to `_is_current_pin_satisfying` HOT 4
- Use pip-resolver-benchmarks and Pip code to create "real world" resolvelib tests? HOT 3
- Request: New Release HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from resolvelib.