It would be nice to add ToHttpApiData instances for ToJSON instances like so

instnace (ToJSON a) => ToHttpApiData a where
  toQueryParam = toStrict . encodeToLazyText

Currently they encode slightly differently:

>>> zt :: ZonedTime
2016-12-31 01:00:00 +0000
>>> encode zt
"\"2016-12-31T01:00:00Z\""
>>> toQueryParam zt
"2016-12-31T01:00:00+0000"

And parseQueryParam can't decode aeson's variant:

>>> decode "\"2016-12-31T01:00:00Z\"" :: Maybe ZonedTime
Just 2016-12-31 01:00:00 +0000
>>> parseQueryParamMaybe "2016-12-31T01:00:00Z" :: Maybe ZonedTime
Nothing

While aeson can decode http-api-data variant:

>>> decode "\"2016-12-31T01:00:00+0000\"" :: Maybe ZonedTime
Just 2016-12-31 01:00:00 +0000
>>> parseQueryParamMaybe "2016-12-31T01:00:00+0000" :: Maybe ZonedTime
Just 2016-12-31 01:00:00 +0000

aeson also can decode +hh suffix (when there's no minutes offset):

>>> decode "\"2016-12-31T01:00:00+03\"" :: Maybe ZonedTime
Just 2016-12-31 01:00:00 +0300
>>> parseQueryParamMaybe "2016-12-31T01:00:00+03" :: Maybe ZonedTime
Nothing

README does not currently mention Form, LenientData or any of the useful helper functions.
Also it does not get typechecked. We should probably update README to give an up-to-date description of what this package provides.

Should there exist one?

UTCTime loses precision, this means if you toHeader and then parseHeader you'll get a different value from the original one. I found this problem as I was using UTCTime as a primary key (I know, not a good idea), sometimes I was sending it as a header, sometimes within an aeson JSON object, and in these circumstances, the same UTCTime would never match:

>>> t <- getCurrentTime
>>> t
2017-02-15 02:30:26.988123 UTC
>>> fromJSON $ toJSON t :: Result UTCTime
Success 2017-02-15 02:30:26.988123 UTC
>>> parseHeader $ toHeader t :: Either Text UTCTime
Right 2017-02-15 02:30:26 UTC

Probably with a generic deriving mechanism for records, whose fields have To/FromHttpApiData instances.

I'd suggest to simply set a lower bound base >= 4.7 if you don't want to support base < 4.7

Configuring http-api-data-0.3.3...
Preprocessing library http-api-data-0.3.3...

src/Web/Internal/FormUrlEncoded.hs:50:18:
    Could not find module `Data.Proxy'
    It is a member of the hidden package `tagged-0.8.5'.
    Perhaps you need to add `tagged' to the build-depends in your .cabal file.
    It is a member of the hidden package `tagged-0.8.4'.
    Perhaps you need to add `tagged' to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.

I've been thinking about Generic deriving (suggested in #3 (comment)) and it probably will not work. Consider this example:

data Example = Example Text Text

Text is always encoded as is, so there is no safe way to encode Example.

So it might only make sense to derive instances for constructors with at most one field, e.g.:

data Account = User Text | Org Text
data Action = Create | Read | Update | Delete deriving (Show)

We could provide TH code to generate instances for those, but I don't think it is worth adding anymore.
Besides there's almost no boilerplate for enumerations:

instance ToHttpApiData Action where
  toUrlPiece = showUrlPiece

instance FromHttpApiData Action where
  parseUrlPiece = readEitherUrlPiece

Thoughts?

I'll search for non-failing builds ASAP.

As @phadej mentions in #53 (comment), aeson uses it, so it'd be nice to have the same tests here!

Would need to add dependencies to http://hackage.haskell.org/package/cookie

Motivation: http://hackage.haskell.org/package/servant-auth-server-0.3.0.0 has http://hackage.haskell.org/package/servant-auth-server-0.3.0.0/src/src/Servant/Auth/Server/SetCookieOrphan.hs module, let's have less orphans.

It would be nice if the Form data type is exported from the Web.FormUrlEncoded module. There are a couple functions returning a Form (toForm, fromEntriesByKey), but there is no way to write the type of them in your own code without importing Form from Web.Internal.FormUrlEncoded.

The constructor of Form should probably not be exported.

Hi, I wonder whether there is some preferred, idiomatic way to deal with things like

data A = MkA 
  { a1 :: Text 
  , a2 :: B 
  } 
  deriving stock Generic 
  deriving anyclass FromForm 

data B = MkB 
  { b1 :: Text 
  , ... }
  deriving stock Generic 
  deriving anyclass FromForm 

fails with "No instance FromHttpApiData B arising from..." because of this instance her:

instance {-# OVERLAPPABLE #-} (Selector s, FromHttpApiData c) => GFromForm t (M1 S s (K1 i c)) where
  gFromForm _ opts form = M1 . K1 <$> parseUnique key form
    where
      key = Text.pack $ fieldLabelModifier opts $ selName (Proxy3 :: Proxy3 s g p)

This is quite unfortunate because I would like to somehow indicate that this should be derived recursively somehow; perhaps we can indicate this somehow by using a newtype wrapper in A's recursive field and then deriving everything newtype for it and then add an {-# OVERLAPPING #-} instance to the generic deriving? Though that would be quite the mess. I wonder what is a good way to do this.

Thanks in advance.

GHC-9.6 ships base-4.18 which is not allowed in the cabal file.

Latest version of path-piece on Hackage has PathPiece instance for Maybe a (which is similar to derived Show instance for Maybe).

Latest code in path-piece repo also has PathPiece instance for ():

instance PathPiece () where
    fromPathPiece t = if t == "_" then Just () else Nothing
    toPathPiece () = "_"

Do these instances make sense? Should we provide a Generic mechanism to derive To/FromHttpApiData similar to Read/Show deriving (i.e. why Maybe should be special)?

Or should we just leave them for PathPiece class only "for compatibility"?

Maybe something like this:

data Part
  = Param ByteString
  | File FileInfo

data FileInfo = FileInfo FilePath ContentType

class ToFormMultiPart a where
  toFormMultiPart :: a -> HashMap Text Part

class FromFormMultiPart a where
  parseFormMultiPart :: HashMap Text Part -> Either Text a

I was hoping readTextData and showTextData would form a nice bijection, so that I could easily derive the ability to send my custom types over the wire without too much thought, but for some reason showTextData converts everything to lower case, causing readTextData to fail.

What is the reasoning behind this? IMO the naming is a little misleading and error prone.

Servant.Common.Text defines ToText/FromText instance for Bool to use lowercase Text representation:

>>> fromText ("true"::Text) :: Maybe Bool
Just True
>>> fromText ("false"::Text) :: Maybe Bool
Just False

path-pieces reuses Show/Read instances for Bool.

Which one should we use?

P. S. I personally like path-pieces version because it does not make Bool special.

It's bundled with GHC 8.6.1 which has been released recently.

• This type uses a HashMap: https://hackage.haskell.org/package/http-api-data-0.4.3/docs/Web-FormUrlEncoded.html#t:Form

• The package in which HashMap is defined: Unordered containers specifically mentions: "This package currently provides no defenses against hash collision attacks such as HashDoS. Users who need to store input from untrusted sources are advised to use Data.Map or Data.Set from the containers package instead."

• As a result, Anyone who uses any of these functions is vulnerable:

  • https://hackage.haskell.org/package/http-api-data-0.4.3/docs/Web-FormUrlEncoded.html#v:urlDecodeForm
  • https://hackage.haskell.org/package/http-api-data-0.4.3/docs/Web-FormUrlEncoded.html#v:urlDecodeAsForm

  This includes a bunch of packages, including servant, see

  • https://hackage-search.serokell.io/?q=urlDecodeForm
  • https://hackage-search.serokell.io/?q=urlDecodeAsForm

• More details about a similar issue here: https://cs-syd.eu/posts/2021-09-11-json-vulnerability

  Proposed fix: Use Map instead of HashMap.

How about instances for these guys from base:

• Char;
• Data.Monoid newtypes;
• Version;
• [a] and ZipList a;
• tuples (up to some reasonable number);
• Void.

The examples for implementing instances of FromForm use the Form constructor, but it's not exported form Web.FormUrlEncoded.

Hi @fizruk! I'm a long-time Servant user (and thus of http-api-data), and I'd like to suggest a solution to prevent the anyclass deriving strategy of ToHttpApiData and FromHttpData via a solution that @MangoIV came up with:

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE DefaultSignatures #-}
{-# LANGUAGE DeriveAnyClass #-}
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE StandaloneKindSignatures #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE UndecidableInstances #-}
{-# OPTIONS_GHC -Wno-unrecognised-pragmas, -Wall #-}

module PreventAnyClassDeriving (A (..), C (..), C' (..), C'' (..), MkTypeError) where

import Data.Kind (Type)
import GHC.TypeLits (ErrorMessage (ShowType, Text, (:<>:)), TypeError)

type MkTypeError :: k -> Type
type family MkTypeError c where
  MkTypeError c =
    TypeError ('Text "Don't use anyclass deriving with class " ':<>: 'ShowType c)

type PreventAnyClassDeriving c = MkTypeError c ~ ()

class C a where
  f :: a -> Int
  default f :: PreventAnyClassDeriving C => a -> Int
  f = error "someone removed the PreventAnyClassDeriving constraint"

class C' where
  f' :: Int
  default f' :: PreventAnyClassDeriving C' => Int
  f' = error "someone removed the PreventAnyClassDeriving constraint"

class C'' a b where
  f'' :: a -> b
  default f'' :: PreventAnyClassDeriving C'' => a -> b
  f'' = error "someone removed the PreventAnyClassDeriving constraint"

data A = MkA
-- deriving anyclass C
-- deriving anyclass instance C'' Int Int
-- will throw compile time error "Don't use anyclass deriving with class C"

What do you think of this?

Additionally, fromUrlPiece for NominalDiffTime can only accept integers, so it can't represent all NominalDiffTimes.

Since NominalDiffTime uses Pico from Data.Fixed underneath, perhaps this can be changed once #77 is implemented?

The current version of http-api-data on Hackage has an upper bound in its custom-setup stanza preventing Cabal 2.2. Would it be possible to relax that, either via a new release or a revision?

As @gregwebs pointed out in his comment for yesodweb/persistent#490, libraries that depend on http-api-data should not use .Internal module.

So far the following functions from Web.HttpApiData.Internal seem to be useful in other libraries (for servant-server and persistent):

• parseUrlPieceMaybe (use case);
• parseHeaderMaybe (use case);
• parseQueryParamMaybe (use case);
• parseUrlPieceWithPrefix (use case);
• showTextData (use case);
• readEitherTextData (use case);
• parseBoundedCaseInsensitiveTextData (use case).

@gregwebs suggested to use Web.HttpApiData.Utils or Web.HttpApiData.Parse for these. The latter is not suitable for showTextData though.

OTOH current naming should not overlap with other libraries and we might as well export those from Web.HttpApiData directly.

Thoughts?

[__3] rejecting: base-4.20.0.0/installed-8a80 (conflict: http-api-data => base>=4.10.1.0 && <4.20)

Neither read nor Data.Text.Read readers check integer bounds. E.g.:

>>> maxBound :: Int8
127
>>> read "128" :: Int8
-128
>>> fst <$> decimal "128" :: Either String Int8
Right (-128)

path-pieces package had parseIntegral helper which parsed Integer and then checked if bounds are violated. Instances in Servant.Common.Text do not have this.

Should we perform this check?

Hi, I’m wondering if it’d make sense to have a special handling for Booleans in GFromForm like we have for Maybe. It’d make sense to write an instance that parses like Maybe but @Text and then fmaps with isJust st a checkbox can be recognised without checking their value or encoding it as Maybe Text on the Backend. Perhaps a newtype like “CheckBoxParam” or something with the same name that’s isomorphic to Bool would be worthwhile as well?

I was trying to write my own ToHttpApiData instance for a newtype wrapper over text, and ended up looking at the existing instance for Text and was surprised to find that it isn't really url-encoding the data. Why is this the case? Isn't the idea of this type-class to convert a value to be safe for usage in URLs (either as path fragments or query-params)?

If ToHttpApiData is not supposed to emit url-encoded stuff, what is the type-class/function/mechanism that does it?

This is straightforward to do and uuid is likely to be among other dependencies (at least I think so).

@gregwebs what do you think?

Does it make sense to have:

instance (FromForm a, FromForm b) => FromForm (a, b) where
    fromForm f = (,) <$> fromForm f <*> fromForm f

and ditto for Either?

### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:254: expression `fromEntriesByKey [("name",["Nick"]),("color",["red","blue"])]'
expected: fromList [("color","red"),("color","blue"),("name","Nick")]
 but got: fromList [("name","Nick"),("color","red"),("color","blue")]
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:405: expression `toEntriesByKey [("name", "Nick"), ("color", "red"), ("color", "white")] :: Either Text [(Text, [Text])]'
expected: Right [("color",["red","white"]),("name",["Nick"])]
 but got: Right [("name",["Nick"]),("color",["red","white"])]
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:487: expression `urlEncodeForm [("name", "Julian"), ("lastname", "Arni")]'
expected: "lastname=Arni&name=Julian"
 but got: "name=Julian&lastname=Arni"
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:523: expression `urlDecodeForm "name=Greg&lastname=Weber"'
expected: Right (fromList [("lastname","Weber"),("name","Greg")])
 but got: Right (fromList [("name","Greg"),("lastname","Weber")])
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:580: expression `urlEncodeAsForm Person {name = "Dennis", age = 22}'
expected: "age=22&name=Dennis"
 but got: "name=Dennis&age=22"
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:695: expression `urlEncodeAsForm Project { projectName = "http-api-data", projectSize = 172 }'
expected: "size=172&name=http-api-data"
 but got: "name=http-api-data&size=172"
Examples: 126  Tried: 116  Errors: 0  Failures: 6

These happened only on 32bit linux, but it could mean they're just transient.

To support GHC 8.8.1

There are missing instances for fixed point numbers, for example:

>>> import Data.Fixed
>>> import Web.HttpApiData
>>> toUrlPiece (1.5 :: Milli)

<interactive>:13:1: error:
    • No instance for (ToHttpApiData Milli)
        arising from a use of ‘toUrlPiece’
    • In the expression: toUrlPiece (1.5 :: Milli)
      In an equation for ‘it’: it = toUrlPiece (1.5 :: Milli)

Both path-pieces and servant's ToText frequently have instances that go via String (Show instances). We could avoid that altogether by using the text-show package, which should be more efficient.

This example from the docs is off:

data Person = Person
  { name :: String
  , age  :: Int }

instance FromForm Person where
  fromForm (Form m) = Person
    <$> parseUnique "name" m
    <*> parseUnique "age"  m

Form m should just be m.

Replace parseBoundedTextData with parseBoundedUrlPiece, parseBoundedQueryParam and parseBoundedHeader.

Hello,
I try to use a webform with a Maybe Float

but I get this error message when it tries to parse the parameter resolution.

here the request from servant

Oct 15 11:33:38 process2 autoprocessing-exe[171261]: POST /xdsme
Oct 15 11:33:38 process2 autoprocessing-exe[171261]:   Params: [("dataCollectionId","0"),("resolution",""),("optimize","0"),("celll",""),("spacegroup","")]
Oct 15 11:33:38 process2 autoprocessing-exe[171261]:   Request Body: dataCollectionId=0&resolution=&optimize=0&celll=&spacegroup=
Oct 15 11:33:38 process2 autoprocessing-exe[171261]:   Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Oct 15 11:33:38 process2 autoprocessing-exe[171261]:   Status: 400 Bad Request 0.0003s

could not parse: `' (input does not start with a digit)

Indeed "" is not a float, but in that case I expect Nothing.

So where is the problem ?

Natural is part of base beginning from version 4.8 , could we add instances for it as well?

Hello,

I'm experiencing test failures with GHC 8.2.1-rc2 and http-api-data (9298d25). This is the relevant log:

Failures:

  test/Web/Internal/HttpApiDataSpec.hs:48: 
  1) Web.Internal.HttpApiData, toUrlPiece <=> parseUrlPiece, TimeOfDay
       Falsifiable (after 4 tests): 
       14:57:05
       "14:57:05." : Left "endOfInput" /= Right 14:57:05

  test/Web/Internal/HttpApiDataSpec.hs:48: 
  2) Web.Internal.HttpApiData, toUrlPiece <=> parseUrlPiece, LocalTime
       Falsifiable (after 1 test): 
       1864-05-09 12:11:44
       "1864-05-09T12:11:44." : Left "endOfInput" /= Right 1864-05-09 12:11:44

  test/Web/Internal/HttpApiDataSpec.hs:48: 
  3) Web.Internal.HttpApiData, toUrlPiece <=> parseUrlPiece, ZonedTime
       Falsifiable (after 22 tests): 
       1864-04-22 10:12:33 QJD
       "1864-04-22T10:12:33.-0014" : Left "Failed reading: satisfy" /= Right 1864-04-22 10:12:33 QJD

  test/Web/Internal/HttpApiDataSpec.hs:48: 
  4) Web.Internal.HttpApiData, toUrlPiece <=> parseUrlPiece, UTCTime
       Falsifiable (after 1 test): 
       1864-05-09 18:57:22 UTC
       "1864-05-09T18:57:22.Z" : Left "Failed reading: satisfy" /= Right 1864-05-09 18:57:22 UTC

Randomized with seed 357928475

Any ideas what could be the cause?