Comments (3)
The current implementation seems to already basically do the right thing on UTF-8 encoded ByteString
, trying to leave the URL alone except when it's truly invalid (a raw π
half-way through) and then URL-encoding those invalid characters.
It should just be more transparent about the fact that it works on unicode/utf-8 by working on Text
directly, helping avoid a variety of errors in the process.
from req.
ByteString
semantically represents a sequence of 8 bit octets, not a series of ASCII characters, and URLs are semantically sequences of [ASCII] characters. You can see this distinction when looking at howChar8
is too big to store only ASCII, and with how oftenByteString
is used to store non-ASCII data, and with the non-ASCII encoding typically used when outputting to stdout.
Even so, Text
is composed of elements which are even "bigger" allowing all sorts of Unicode in it. Thus it's hardly a better option?
The type which we take as the argument of parseUrlHttp(s)
function is dictated by the http-types
package because we build on functions like decodePathSegments
and parseQueryText
, both of which take ByteString
s and make some Text
pieces out of it. If you want to change this, then perhaps the right place to start is http-types
issue tracker.
Anyway, AFAIU, both Text
and ByteString
are not perfect:
- In case of
ByteString
it's 1) have elements which are too big 2) encoding issues may introduce bugs. - In case of
Text
it's 1) even bigger with Unicode symbols which cannot appear un-escaped in URLs (correct me if I'm wrong, I may have forgotten the subtleties).
Since both options are not ideal and http-types
uses ByteString
I think we should stick to it.
from req.
The flaws in the types are more or less what you said, but I would adjust the phrasing slightly to show why Text really is much less problematic than ByteString:
The two main issues are:
-
Encoding issues: It's relatively easy to have bugs that slip through the type checker if you just use the wrong encoding/decoding function. This issue is unique to ByteString.
-
Too many codepoints: Suffered by both ByteString and Text, but I would actually argue that ByteString is even worse in this regard. At least with Text, these overly large codepoints have a pretty sane default behavior, namely UTF-8 encode + percent encode (e.g. what your browser does when you type arbitrary unicode in the URL).
The http-types
aspect is interesting, and my follow up to that is that http-types
is also doing the wrong thing, as they are currently worrying about encoding and assuming it to be utf8, when they should really only be dealing with charpoints. Occasionally it makes sense to do both such as in XML when you need to try and guess the encoding based on the encoding parameter and change the parser accordingly, but usually it does not make sense.
from req.
Related Issues (20)
- Make Req an instance of MonadUnliftIO HOT 2
- Cannot perform a request with url containing braces HOT 7
- Hard to pass Link header URIs back into req HOT 5
- When req fails with a StatusCodeException the body is truncated HOT 4
- Get access to the Request HOT 5
- Add `MonadMask` instance for `Req`?
- ConnectionTimeout with "req" even though "curl" and browser work just fine HOT 1
- Set a retrying limit by default for request timeouts HOT 2
- not installable on ghc 9.0.1 due to reliance on base <4.15 HOT 3
- Support retry 0.9 HOT 1
- How can I retrieve the exact URL with query parameters? Possible limitation? HOT 6
- Interface with ToForm and FromForm HOT 2
- Support for GHC 9.2 HOT 2
- Not compatible with transformers-0.6 HOT 1
- Couldn't find a way to call odata services HOT 1
- Make responses printable. HOT 1
- Hackage docs not working for 3.3.0 HOT 2
- How can I retry the request even if it errors out? HOT 2
- unexpected program termination HOT 2
- Inspecting Request Created by `req` Function HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from req.