Comments (5)
WRT the session variables, it may be worth considering also the google analytics etc. URL params, like "utm_source"; those attributes will be added by lots of websites and social media tools to all outbound URLs and could probably be safely stripped during canonicalisation.
from w3lib.
Nice links, thanks!
removal of userinfo
What does it mean? username/password?
dots and slashes in path and hostname
Could you please give an example? What's wrong with e.g. dots in hostname?
spaces succeeding and preceding the URL
Arguably this is an issue with link extraction, not with canonicalization. URLs shouldn't have such whitespaces. See also: https://github.com/scrapy/scrapy/issues/1614.
common session id variables and their values
This would be a very good feature to have, but we can't just blindly strip some known session_id parameter names and values by default. See also: scrapy/scrapy#1560.
ip v6 canonicalization
a good call.
from w3lib.
yes, userinfo is username and password.
Could you please give an example? What's wrong with e.g. dots in hostname?
google.com.
obviously these are task-dependent issues, but there is no mechanism to enable such behaviour.
from w3lib.
those attributes will be added by lots of websites and social media tools to all outbound URLs and could probably be safely stripped during canonicalisation.
I don't quite like doing this all by default without an option to turn it off. So the main question for now is how to make this behavior overridable, so that users can implement such rules themselves, without having to modify w3lib or scrapy. We can also provide something by default, but I think it should be a next step, and a separate task.
obviously these are task-dependent issues, but there is no mechanism to enable such behaviour.
Yep, it is discussed in scrapy/scrapy#1560. Changing canonicalize_url to do these actions is not enough, there should be a mechanism in Scrapy to enable it, and it also should be customizable. This part of the ticket is a duplicate of scrapy/scrapy#1560, probably it is better to keep discussion of this feature there. The problem is known, but there was no concrete proposal on how to fix it so far.
from w3lib.
yes, userinfo is username and password.
Is it a real issue in practice? I understand why it can help, but I can also see how it can break some of the use cases if scrapy/scrapy#1466 gets merged, if we do it by default.
from w3lib.
Related Issues (20)
- remove_tags not working on html comments HOT 4
- Redirection may not work depending on order of 'content' and 'http-equiv' in meta tag HOT 5
- Fix CI issue on PyPy 3 and the Rust compiler HOT 1
- test_add_or_replace_parameter fails on Python 3.6.13, 3.7.10, 3.8.8, 3.9.2 due to CVE-2021-23336 fix HOT 1
- w3lib.url.safe_url_string incorrectly encode IDNA domain with port HOT 1
- should the canonize_url function convert an apostrophe to %27 HOT 2
- [request] Update pypi release HOT 1
- basic_auth_header uses the wrong flavor of base64
- safe_url_string URL-encodes already-encoded username and password, breaking idempodency HOT 1
- Issue in safe_url_encoding HOT 8
- BOM should take precedence over Content-Type header when detecting the encoding
- safe_url_string handling IPv6 URLs HOT 5
- Function `convert_entity` does not catch `OverflowError`
- Reimplement safe_url_string based on the URL living standard HOT 4
- Location of libw3.a HOT 1
- test_safe_url_string_url regressed on 3.11.4 HOT 1
- canonicalize_url incorrectly handles port when using hostname that requires IDNA encoding HOT 1
- Space at end of query string is trimmed HOT 1
- Missing PyPI description
- canonicalize_url inappropriately "lowercases" credentials (username/password) in url HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from w3lib.