aristidb / http-types Goto Github PK
View Code? Open in Web Editor NEWGeneric HTTP types for Haskell (for both client and server code)
License: BSD 3-Clause "New" or "Revised" License
Generic HTTP types for Haskell (for both client and server code)
License: BSD 3-Clause "New" or "Revised" License
Generic HTTP types for Haskell (for both client and server code). This library also contains some utility functions, e.g. related to URI handling, that are not necessarily restricted in use to HTTP, but the scope is restricted to things that are useful inside HTTP, i.e. no FTP URI parsing.
Why in the world is the name movedPermanently301
? If I wanted to use the number as the name, I wouldn't be using the nice-name variant...
It could be useful to derive Typeable
on various types. HttpVersion
comes to mind. a) do you agree, and b) would you accept a pull request with some changes (limited to sprinkling deriving Typeable around)?
Hi @aristidb,
I found that URI fragment
is missing in the package.
I am talking about secondary part of URI (after hash sign, #
). Despite it does not used in HTTP REST API, it is quite useful in HTTP (e.g. as HTML anchor). I would like to have it as a part of http-types
just not to introduce it separately in several other packages dependent on this package (e.g. yesod-core
).
Consider GitHub, e.g. following URL: https://github.com/aristidb/http-types/blob/master/Network/HTTP/Types/URI.hs#L340-L347
While the content of the page with and without anchor remains the same, the page markup is different.
Imagine the situation where I want to introduce server-side rendering for HTML page with similar (highlighting) functionality based on fragment.
Would you like to accept PR with this missing part?
Thanks,
Andrey
Here https://github.com/aristidb/http-types/blob/master/Network/HTTP/Types/URI.hs#L253 documentation states Performs percent encoding on all unreserved characters, as well as @\:\@\=\+\$@,
while under the hood it calls urlEncodeBuilder False
which does not escape characters :@&=+$,
. Also in the docs the characters :@&=+$,
called unreserved but they are actually reserved https://developers.google.com/maps/url-encoding. I also send related PR here fizruk/http-api-data#119
Configuring http-types-0.6.6...
Setup: At least the following dependencies are missing:
case-insensitive >=0.2 && <0.4
It is defined and exported in Network.HTTP.Types.Status
but is not exported from Network.HTTP.Types
like most other status codes. This is also true for statuses 422,428,429,431 and 511.
Is there a reason that Types.hs does not simply export "module Network.HTTP.Types.Status"?
In this function:
http-types/Network/HTTP/Types/Header.hs
Line 166 in dda6874
The ByteRanges
are render with an =
sign. But according to section 14.16 of RFC2616:
https://www.ietf.org/rfc/rfc2616.txt
When rendering a byte range in a Content-Range
header, there is no '=',
Content-Range = "Content-Range" ":" content-range-spec
content-range-spec = byte-content-range-spec
byte-content-range-spec = bytes-unit SP
byte-range-resp-spec "/"
( instance-length | "*" )
byte-range-resp-spec = (first-byte-pos "-" last-byte-pos)
| "*"
instance-length = 1*DIGIT
This is, of course, an = in the request. This seems like senseless inconsistency in the RFC itself.
Perhaps renderByteRanges
is only intended to render the byte ranges in requests? And there should be a different function for rendering byte ranges in a response? Or I am misunderstanding something?
@aristidb
I've wondered, since this repo seems to have been somewhat abandoned. Would you be open to me having either co-maintainership of this repo, or that I fork this repo and take over maintainership of http-types
on hackage?
Instead of having a (fairly useless) type synonym for HTTP method, and then a (fairly restrictive) enumeration of so-called "standard methods", why not have:
data Method = GET | POST | PROPFIND | ... | OtherMethod ByteString
The status code should be imATeapot418
, rather than the current imATeaPot418
.
After all, this is the whole reason why I added the useQuestionMark parameter in the first place. Otherwise, just adding a ? to the string if necessary would have been easier.
This is how my old renderQuery behaved, but Michael Snoyman's updated version changed the behavior to always include the ?.
This is a problem because I use the result of renderQuery in a message that has to be signed cryptographically for AWS.
The URI spec says:
For consistency, URI producers and
normalizers should use uppercase hexadecimal digits for all percent-
encodings.
Given how widely used http-types is in the Haskell ecosystem, I think it would be wise to follow the spec's recommendation on this.
Additionally, please note that we may keep http-types constrained to 0.10 on Stackage, due to 0.11's adverse effect on amazonka request signing (see brendanhay/amazonka#440) and other such packages.
See also: commercialhaskell/stackage#3226
I had expected (hRange, renderByteRange (ByteRangeFrom 1000)) would be a http request Range header, but in fact renderByteRange does not include the "bytes=" part, so that generates an invalid "Range: 1000-" header.
renderByteRanges does include the "bytes=" part, so the workaround is to use
(hRange, renderByteRanges [ByteRangeFrom 1000])
The only way to learn of either behavior is to read the source code. This at least needs to be documented better.
It might be useful to parameterize the rendering functions with the role the range is being rendered for, eg a Request or a Response. See also #64.
3 characters is a cheap price for easier comprehension when you are already typing long names like statusForbidden
I suggest changing to statusForbidden403
or status403Forbidden
I think it's a good idea to provide pattern synonyms for Header/Method/Status... along aside constants, so people can match them with ease.
Would this be reasonable? toEnum would just be a giant case statement for all the status codes supported, with a fallthrough that used a null ByteString for the message.
Thoughts?
Is the goal to provide some URL handling utilities (like extract the path from a bytestring, which http-types already provides) or to eventually provide comprehensive URL handling (e.g. extracting the scheme and fragment from a URL as well)?
Clarifying this would help users understand the difference between http-types and other libraries like uri-bytestring and network-uri.
I think the set of special symbols not url encoded in query parameters can be expanded from
ALPHA / DIGIT / "-" / "_" / "." / "~"
with
":" / "@" / "/" / "!" / "$" / "'" / "(" / ")" / "*" / ","
e.g ":", "@", "/" (extra pchar values) and sub-delims without "?" (query component delimiter) and "&;=+" (form url encoding sub component delimiters).
This is based on the query ABNF in RFC 3986 Appendix A. Appropriate sections below
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="
pct-encoded = "%" HEXDIG HEXDIG
pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
query = *( pchar / "/" / "?" )
and RFC 3986 Section 2.4: When to Encode or Decode quoted in part below
Under normal circumstances, the only time when octets within a URI
are percent-encoded is during the process of producing the URI from
its component parts. This is when an implementation determines which
of the reserved characters are to be used as subcomponent delimiters
and which can be safely used as data. Once produced, a URI is always
in its percent-encoded form.
But do check my working and I can send over a PR if you agree.
Hi! The new blaze-builder was released, is there a chance to make this libarry work with it?
OK, if you take a look at:
The Haddock says that it returns Builder
from Data.Binary.Builder
, and indeed if I write code that interacts with encodePathSegments
it seems true. While according to the source code you export the builder from Blaze.ByteString.Builder
and older versions of Haddock also indicate that. I tried to blame on GitHub and I don't see when the change happened. Data.Binary.Builder
and Blaze.ByteString.Builder
do not seem to export each other's Builder
type (in fact the packages do not depend on each other).
Any idea what is going on here?
I'm working on a project that has defined some of it's own header types which are in master but not in 0.8.6 on hackage. When do you expect to be making a new release that'll include these new header definitions?
https://github.com/helium/airship/blob/master/src/Airship/Internal/Decision.hs#L47-L69
The browser behavior (and standard if I can dig up the forms spec) of HTML forms with array-like values is to include a query string with repeated keys.
This multi-select:
<select name="colors" multiple>
<option value="orange">orange</option>
<option value="purple">purple</option>
</select>
Will result in ?colors=orange&colors=purple
if both options are selected. Instead of parsing this as a list, parseQuery
ignores the second value. Furthermore, the query is represented as a lookup map of bytestrings, so has no notion of multiple values per key.
The reason I bring this issue up is this use case: basic HTML form with a multi-select and no JS and a haskell backend using http-types
. The http-types
Query
type and parsers fail that use case.
Any plans to push a new release? There have been a few headers added since 0.9.1 (notably "Set-Cookie" would be nice to have).
It would be nice to be able to see what has changed in a new release (e.g. the latest 0.9), by adding a changelog file (which would automatically be shown on hackage).
While i'm tackling #66 , i found an interesting problem: http-type
would create too many ByteString
:
parseQueryText/parseQuery
use urlDecode
which create new ByteString
s.HeaderItem
use CI ByteString
which in turn create a case-folded ByteString
.These ByteString
s are not created via slice thus will be slow and create memory fragment, i would suggest to simply use Text
instead.
The "Source repository" field on Hackage at http://hackage.haskell.org/package/http-types-0.8.1 contains "git clone https://github.com/aristidb/aws.git", which seems to refer to a different repository.
In ghci, if I do :m + Network.HTTP.Types or import Network.HTTP.Types, I suddenly find that none of the Prelude identifiers are in scope any more -- not even True and False.
parseByteRanges cannot parse a response like ("Content-Range","bytes */10000")
This can occur when a server does not support a requested range. In particular, I was resuming the download of a file, but the whole file was already actually downloaded. The server responds with partialContent206 and parsing the Content-Range header is the only way to detect if the whole file was already downloaded.
This is called an "unsatisfied-range" in the HTTP RFC, and says it SHOULD be used for a 206 response (leaving open the possibility that some servers send one of the other range formats). None of the ByteRange constructors seem appropriate to parse that into, so it seems that would need a new ByteRangeUnsatisfied constructor.
Is this project still maintained?
Some PRs and issue reports have not gotten a response in over 2 years.
Coming GHC 7.8 uses array 0.5.0.0. So, http-types cannot be compiled with GHC head at this moment. Please relax the upper bound. A new release would be appreciated.
Is there any reason why the StdMethod
type does not have a Generic
instance ?
I need the Generic
instance to derive another instance (Hashable) and was wondering if it would be possible to add it or if there's a specific reason why it's not present.
I am not seeing on obvious way to render the various types into a Byte String that I can send through a socket.
How would I render the http version, status, and headers into a usable byte string?
The encodePathSegments
function incorrectly prefixes output with a /. This makes it nearly useless for constructing relative URI references, and this behavior also directly contradicts the documentation.
Test program:
{-# LANGUAGE OverloadedStrings #-}
import Blaze.ByteString.Builder
import Network.HTTP.Types
import Data.Text
main = print . toByteString $ encodePathSegments ["foo", "bar", "baz"]
Expected output (taken directly from the haddock documentation):
"foo/bar/baz"
Actual output:
"/foo/bar/baz"
This might be slightly related to #62.
Let's say that I have these data types:
data Something = Something
{ field01 :: Text
, field02 :: Text
}
data StepInstruction = StepInstruction
{ id :: Int
, order :: Int
, description :: Text
}
data Recipe = Recipe
{ id :: Int
, name :: Text
, something :: Something
, instructions :: [StepInstruction]
}
In PHP I could easily submit a form with such data type and get it back in the form of a nested array. Convention for submitting Recipe
is as follows:
?id=0
&name=Recipe01
&instructions[0][id]=0
&instructions[0][order]=1
&instructions[0][description]=somedescription
&instructions[1][id]=1
&instructions[1][order]=2
&instructions[1][description]=someotherdescription
Currently Query
flat and I'd have to do additional parsing on the argument names to get an array of structured arguments.
I'm not sure if this is a standard, but I've also seen the same convention in C#/MVC so I think it would be a good idea to support such feature, because I might want to add repeating "sub forms" to my main form that should be contained in array, and currently there is no good way to do that from what I can see.
Here is also an example of a form that might submit such query:
<form>
<input type="hidden" name="id">
<input type="text" name="name">
<input type="text" name="something[field01]">
<input type="text" name="something[field02]">
<div>
<input type="hidden" name="instructions[0][id]">
<input type="number" name="instructions[0][order]">
<input type="text" name="instructions[0][description]">
</div>
<div>
<input type="hidden" name="instructions[1][id]">
<input type="number" name="instructions[1][order]">
<input type="text" name="instructions[1][description]">
</div>
<div>
<input type="hidden" name="instructions[2][id]">
<input type="number" name="instructions[2][order]">
<input type="text" name="instructions[2][description]">
</div>
</form>
Maybe QueryItem
should look something like this:
data QueryItem
= PlainValue ByteString
| QueryItemList [QueryItem]
| QueryItem (ByteString, Maybe QueryItem)
Cf snoyberg/http-enumerator#13
The standard seems to say semicolon should be supported as an alternative to ampersand in query strings. Once http-types supports this, http-enumerator will be able to fetch darcsweb pages.
Such as:
statusIsRedirect :: Status -> Bool
statusIsRedirect (Status {statusCode=code}) = code >= 300 && code < 400
Classes are http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html:
The issue is, that hspec
is used instead of hspecX
to evaluate the spec (see hspec/hspec#71).
Version 0.12.3
The documentation of urlEncode
for the boolean parameter is:
Whether to decode '+' to ' '
This doesn't appear to be the whole story though. It also controls other characters like @
ghci> import Prelude
ghci> URI.urlEncode False "[email protected]"
"[email protected]"
ghci> URI.urlEncode True "[email protected]"
"test%40example.com"
This documentation appears true for urlDecoding (replacePlus
is only used in one place).
-- | Percent-decoding.
urlDecode :: Bool -- ^ Whether to decode @\'+\'@ to @\' \'@
-> B.ByteString -> B.ByteString
urlDecode replacePlus z = fst $ B.unfoldrN (B.length z) go z
where
go bs =
case B.uncons bs of
Nothing -> Nothing
Just (43, ws) | replacePlus -> Just (32, ws) -- plus to space
Just (37, ws) -> Just $ fromMaybe (37, ws) $ do -- percent
(x, xs) <- B.uncons ws
x' <- hexVal x
(y, ys) <- B.uncons xs
y' <- hexVal y
Just (combine x' y', ys)
Just (w, ws) -> Just (w, ws)
hexVal w
| 48 <= w && w <= 57 = Just $ w - 48 -- 0 - 9
| 65 <= w && w <= 70 = Just $ w - 55 -- A - F
| 97 <= w && w <= 102 = Just $ w - 87 -- a - f
| otherwise = Nothing
combine :: Word8 -> Word8 -> Word8
combine a b = shiftL a 4 .|. b
But in URL encoding, the true/false flag isn't restricted to just spaces:
unreservedQS, unreservedPI :: [Word8]
unreservedQS = map ord8 "-_.~"
unreservedPI = map ord8 "-_.~:@&=+$,"
-- | Percent-encoding for URLs.
urlEncodeBuilder' :: [Word8] -> B.ByteString -> B.Builder
urlEncodeBuilder' extraUnreserved = mconcat . map encodeChar . B.unpack
where
encodeChar ch | unreserved ch = B.word8 ch
| otherwise = h2 ch
unreserved ch | ch >= 65 && ch <= 90 = True -- A-Z
| ch >= 97 && ch <= 122 = True -- a-z
| ch >= 48 && ch <= 57 = True -- 0-9
unreserved c = c `elem` extraUnreserved
-- must be upper-case
h2 v = B.word8 37 `mappend` B.word8 (h a) `mappend` B.word8 (h b) -- 37 = %
where (a, b) = v `divMod` 16
h i | i < 10 = 48 + i -- zero (0)
| otherwise = 65 + i - 10 -- 65: A
-- | Percent-encoding for URLs (using 'B.Builder').
urlEncodeBuilder
:: Bool -- ^ Whether input is in query string. True: Query string, False: Path element
-> B.ByteString
-> B.Builder
urlEncodeBuilder True = urlEncodeBuilder' unreservedQS
urlEncodeBuilder False = urlEncodeBuilder' unreservedPI
-- | Percent-encoding for URLs.
urlEncode :: Bool -- ^ Whether to decode @\'+\'@ to @\' \'@
-> B.ByteString -- ^ The ByteString to encode as URL
-> B.ByteString -- ^ The encoded URL
urlEncode q = BL.toStrict . B.toLazyByteString . urlEncodeBuilder q
I'm not super familiar with this domain. Should URL decoding have an option to work with those other characters? Should just the documentation for URL encoding change?
If I import Network.HTTP.Types.Header I have the HeaderName type, but not when importing Network.HTTP.Types
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.