Better ergonomics around arbitrary URLs about req HOT 7 CLOSED

I'm going to work today or tomorrow on adding support for using URIs from modern-uri in req. It's going to work in the following manner: you'll still have parseURI for compatibility but also there will be functions to turn URI from modern-uri (this is the type you want to render and store) into (Url Https, Option scheme) which may fail because e.g. the scheme may be not HTTP or HTTPS. You'd need to do the conversion right before you do the requests.

This will also give us a better URI parser which supports more features like fragments.

from req.

Hmm, that seems like a good change.

With that said, I still end up in the same position, where I can't parse the data into a "req-ready" type, because once I do I lose the ability to render or store it.

Is there any particular reason why query strings are not included in the URL type? They seem pretty equivalent to the path in my eyes, both are optional and can be composed, and both are expected to be seen when rendering a URL. I think this is the only http library I have seen that doesn't put query strings in the URL, and unfortunately this essentially means I can't do anything interesting with a URL, as rendering it will be misleading whenever a query string has been chopped off.

from req.

I still end up in the same position, where I can't parse the data into a "req-ready" type, because once I do I lose the ability to render or store it.

The type you are looking for is URI from modern-uri. Of course you'll have to convert that URIs into (Url, Option scheme) pair just before making a request, but there is no problem with this. You could as well define a helper which takes URIs directly and then performs the conversion for you. URI can be manipulated and rendered, see: https://hackage.haskell.org/package/modern-uri.

Is there any particular reason why query strings are not included in the URL type?

We are manipulating the Request type here. When you use the header option it just modifies the Request to include the header. The same happens with paths and query parameters. Req was originally created for making requests against static REST-like APIs. If you think about it that way, it becomes clear that an endpoint path and query parameters are different beasts, hence the separation. Query parameters can go not only in URL, too.

from req.

The type you are looking for is URI from modern-uri. Of course you'll have to convert that URIs into (Url, Option scheme) pair just before making a request, but there is no problem with this.

But that conversion is going to return an Either or Maybe that I am going to have to handle right?

it becomes clear that an endpoint path and query parameters are different beasts, hence the separation.

I'm still not sure I fully understand, what is the big difference between the following?

https://www.example.com/object/<pk>/subobject/<pk>

https://www.example.com/?object=<pk>&subobject=<pk>

I know i've seen the two used pretty interchangeably in various APIs and websites.

Query parameters can go not only in URL, too.

In that case a single QueryString type that can be built up (Monoid and a primitive or two) seems to make sense, and then you can either /? it onto the URL or throw it into the body.

from req.

But that conversion is going to return an Either or Maybe that I am going to have to handle right?

The action of performing a request is already something that can fail. You just add yet another way for it to fail. If you have something that uses a scheme that is different from http or https (one reason you may get Nothing when converting) it is not going to work anyway.

I'm still not sure I fully understand, what is the big difference between the following?

You are calling different endpoints and interact with different resources. Sure one could say "it works all the same for me", but conceptually (keeping REST principles in mind), those are two very different requests. We want to keep things separated like this in the library.

In that case a single QueryString type that can be built up (Monoid and a primitive or two) seems to make sense, and then you can either /? it onto the URL or throw it into the body.

This is already the case. You can use query flags and parameters as options and also as body when you use ReqBodyUrlEnc.

from req.

If you have something that uses a scheme that is different from http or https (one reason you may get Nothing when converting) it is not going to work anyway.

Right. That's why I was planning on parsing it to a req URL as early in the pipeline as possible. That way I know at parse-time which parsed URL did not start with http or https right away.

You are calling different endpoints and interact with different resources. Sure one could say "it works all the same for me", but conceptually (keeping REST principles in mind), those are two very different requests. We want to keep things separated like this in the library.

Maybe a better example would be:

https://www.example.com/endpoint/<pk>

https://www.example.com/endpoint?pk=<pk>

Both are the same endpoint, one just uses a query param to specify the pk, the other uses an extra path segment.

It makes very little sense to me why I get to keep the first <pk> in the URL and can render and store it, but the second <pk> disappears irrecoverably into an opaque Option type.

I'm also not saying the above (or my previous example) should be the exact same, they would be different values that return False when compared with ==, but I do think they both count as being URLs.

As far as RFC3986 is concerned, a URI/URL includes the query string. The above RFC is what REST and friends all build on top of, and every other REST framework that I know of stores query strings in the URL.

This is already the case. You can use query flags and parameters as options and also as body when you use ReqBodyUrlEnc.

Sure. But I would like to directly put the query flags and parameters into the URL type via something like /?, rather than put them in the opaque Option type that I cannot render back out.

Adding query strings to the URL type would also simplify the type of the parseUrl* functions, as they no longer need to spit out an Option type.

from req.

Right. That's why I was planning on parsing it to a req URL as early in the pipeline as possible. That way I know at parse-time which parsed URL did not start with http or https right away.

You can still do that. Just create your own wrapper.

It makes very little sense to me why I get to keep the first <pk> in the URL and can render and store it, but the second <pk> disappears irrecoverably into an opaque Option type.

They both disappear irrecoverably. Both Url and Options are opaque. You want URI.

As far as RFC3986 is concerned, a URI/URL includes the query string. The above RFC is what REST and friends all build on top of, and every other REST framework that I know of stores query strings in the URL.

I admit that Url type from Req is not truly a URL, it's just path. The naming is unfortunate.

Sure. But I would like to directly put the query flags and parameters into the URL type via something like /?, rather than put them in the opaque Option type that I cannot render back out.

Req is not about URL manipulation, modern-uri is. Req is just about sending a request and getting a response back.

No Haskell HTTP client libraries I know of provide special safe Url types for manipulation. They work on Strings. modern-uri does what you want. If you need to guarantee that parsed URLs are going to work with Req you can make an opaque product type with URI and the result of its conversion to (Url, Option scheme). When you need to print it back, use URI. Otherwise use the second component to perform the call.

from req.