Git Product home page Git Product logo

sttp's Introduction

sttp

Ideas, suggestions, problems, questions CI Maven Central Open in Gitpod

The Scala HTTP client that you always wanted!

This is the development version of the upcoming sttp client 4. For the current stable version, see sttp 3 on GitHub and its documentation.

Welcome!

sttp client is an open-source library which provides a clean, programmer-friendly API to describe HTTP requests and how to handle responses. Requests are sent using one of the backends, which wrap lower-level Scala or Java HTTP client implementations. The backends can integrate with a variety of Scala stacks, providing both synchronous and asynchronous, procedural and functional interfaces.

Backend implementations include the HTTP client that is shipped with Java, as well as ones based on akka-http, http4s, OkHttp. They integrate with Akka, Monix, fs2, cats-effect, scalaz and ZIO. Supported Scala versions include 2.12, 2.13 and 3, Scala.JS and Scala Native; supported Java versions include 11+.

Here's a quick example of sttp client in action:

import sttp.client4._

val sort: Option[String] = None
val query = "http language:scala"

// the `query` parameter is automatically url-encoded
// `sort` is removed, as the value is not defined
val request = basicRequest.get(uri"https://api.github.com/search/repositories?q=$query&sort=$sort")
  
val backend = DefaultSyncBackend()
val response = request.send(backend)

// response.header(...): Option[String]
println(response.header("Content-Length")) 

// response.body: by default read into an Either[String, String] to indicate failure or success 
println(response.body)                                 

Documentation

sttp (v4) documentation is available at sttp.softwaremill.com/en/latest.

sttp (v3) documentation is available at sttp.softwaremill.com/en/stable.

sttp (v2) documentation is available at sttp.softwaremill.com/en/v2.

sttp (v1) documentation is available at sttp.softwaremill.com/en/v1.

scaladoc is available at https://www.javadoc.io

Quickstart with scala-cli

Add the following directive to the top of your scala file to add the core sttp dependency: If you are using scala-cli, you can quickly start experimenting with sttp by copy-pasting the following:

//> using dep "com.softwaremill.sttp.client4::core:4.0.0-M13"
import sttp.client4.quick._
quickRequest.get(uri"http://httpbin.org/ip").send()

The quick package import brings in the sttp API and a pre-configured, global synchronous backend instance.

Quickstart with Ammonite

Similarly, using Ammonite:

import $ivy.`com.softwaremill.sttp.client4::core:4.0.0-M13`
import sttp.client4.quick._
quickRequest.get(uri"http://httpbin.org/ip").send()

Quickstart with sbt

Add the following dependency:

"com.softwaremill.sttp.client4" %% "core" % "4.0.0-M13"

Then, import:

import sttp.client4._

Type basicRequest. and see where your IDE’s auto-complete gets you!

Other sttp projects

sttp is a family of Scala HTTP-related projects, and currently includes:

  • sttp client: this project
  • sttp tapir: Typed API descRiptions
  • sttp model: simple HTTP model classes (used by client & tapir)
  • sttp shared: shared web socket, FP abstractions, capabilities and streaming code.
  • sttp apispec: OpenAPI, AsyncAPI and JSON Schema models.

Contributing

If you have a question, suggestion, or hit a problem, feel free to ask on our discourse forum!

Or, if you encounter a bug, something is unclear in the code or documentation, don’t hesitate and open an issue on GitHub.

We are also always looking for contributions and new ideas, so if you’d like to get into the project, check out the open issues, or post your own suggestions!

Note that running the default test task will run the tests using both the JVM and JS backends, and is likely to run out of memory. If you'd like to run the tests using only the JVM backend, execute: sbt rootJVM/test.

Importing into IntelliJ

By default, when importing to IntelliJ or Metals, only the Scala 2.13/JVM subprojects will be imported. This is controlled by the ideSkipProject setting in build.sbt (inside commonSettings).

If you'd like to work on a different platform or Scala version, simply change this setting temporarily so that the correct subprojects are imported. For example:

// import only Scala 2.13, JS projects
ideSkipProject := (scalaVersion.value != scala2_13) || !thisProjectRef.value.project.contains("JS")

// import only Scala 3, JVM projects
ideSkipProject := (scalaVersion.value != scala3) || thisProjectRef.value.project.contains("JS") || thisProjectRef.value.project.contains("Native"),

// import only Scala 2.13, Native projects
ideSkipProject := (scalaVersion.value != scala2_13) || !thisProjectRef.value.project.contains("Native")

Modifying documentation

The documentation is typechecked using mdoc. The sources for the documentation exist in docs. Don't modify the generated documentation in generated-docs, as these files will get overwritten!

When generating documentation, it's best to set the version to the current one, so that the generated doc files don't include modifications with the current snapshot version.

That is, in sbt run: set version := "4.0.0-M13", before running mdoc in docs.

Testing the Scala.JS backend

In order to run tests against JS backend you will need to install Google Chrome.

Building & testing the scala-native backend

By default, sttp-native will not be included in the aggregate build of the root project. To include it, define the STTP_NATIVE environmental variable before running sbt, e.g.:

STTP_NATIVE=1 sbt

You might need to install some additional libraries, see the scala native documentation site. On macos, you might additionally need:

ln -s /usr/local/opt/openssl/lib/libcrypto.dylib /usr/local/lib/
ln -s /usr/local/opt/openssl/lib/libssl.dylib /usr/local/lib/

Commercial Support

We offer commercial support for sttp and related technologies, as well as development services. Contact us to learn more about our offer!

Copyright

Copyright (C) 2017-2024 SoftwareMill https://softwaremill.com.

sttp's People

Contributors

adamw avatar adpi2 avatar aeons avatar amorfis avatar benzwreck avatar fristi avatar ghik avatar ghostbuster91 avatar ghostdogpr avatar guymers avatar ikhoon avatar kamilkloch avatar kciesielski avatar kevinmeredith avatar kubinio123 avatar kubukoz avatar lglo avatar lolgab avatar magdzikk avatar mergify[bot] avatar micossow avatar mkrzemien avatar omainegra avatar pask423 avatar pawelkaczor avatar pcejrowski avatar scala-steward avatar softwaremill-ci avatar svroonland avatar yanns avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sttp's Issues

NoSuchMethodError when creating SttpBackendStub

Hello, I have just started using sttp. It looks promising but I am hitting the wall on the first execution.

My setup is as follows:

scala 2.11
monix 2.3.0
"com.softwaremill.sttp" %% "core" % "1.0.2"
"com.softwaremill.sttp" %% "async-http-client-backend-monix" % "1.0.2"

I have simple http client performing POST requests and I want to test it. I am providing a stub like this:

 implicit val sttpStub: SttpBackendStub[Task, Observable[ByteBuffer]] =
      SttpBackendStub(AsyncHttpClientMonixBackend())
        .whenRequestMatches(_ => true)
        .thenRespond(entityFoundHtml)

and upon execution of the test I am seeing this error:

[2017-11-08 19:32:34,257] [DEBUG] [i.n.u.i.l.InternalLoggerFactory] [] Using SLF4J as the default logging framework
[2017-11-08 19:32:34,263] [DEBUG] [io.netty.util.ResourceLeakDetector] [] -Dio.netty.leakDetection.level: simple
[2017-11-08 19:32:34,269] [DEBUG] [io.netty.util.ResourceLeakDetector] [] -Dio.netty.leakDetection.maxRecords: 4
[2017-11-08 19:32:34,275] [DEBUG] [i.n.util.internal.PlatformDependent0] [] java.nio.Buffer.address: available
[2017-11-08 19:32:34,276] [DEBUG] [i.n.util.internal.PlatformDependent0] [] sun.misc.Unsafe.theUnsafe: available
[2017-11-08 19:32:34,276] [DEBUG] [i.n.util.internal.PlatformDependent0] [] sun.misc.Unsafe.copyMemory: available
[2017-11-08 19:32:34,276] [DEBUG] [i.n.util.internal.PlatformDependent0] [] java.nio.Bits.unaligned: true
[2017-11-08 19:32:34,277] [DEBUG] [i.n.util.internal.PlatformDependent] [] Java version: 8
[2017-11-08 19:32:34,277] [DEBUG] [i.n.util.internal.PlatformDependent] [] -Dio.netty.noUnsafe: false
[2017-11-08 19:32:34,277] [DEBUG] [i.n.util.internal.PlatformDependent] [] sun.misc.Unsafe: available
[2017-11-08 19:32:34,277] [DEBUG] [i.n.util.internal.PlatformDependent] [] -Dio.netty.noJavassist: false
[2017-11-08 19:32:34,343] [DEBUG] [i.n.util.internal.PlatformDependent] [] Javassist: available
[2017-11-08 19:32:34,343] [DEBUG] [i.n.util.internal.PlatformDependent] [] -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
[2017-11-08 19:32:34,344] [DEBUG] [i.n.util.internal.PlatformDependent] [] -Dio.netty.bitMode: 64 (sun.arch.data.model)
[2017-11-08 19:32:34,344] [DEBUG] [i.n.util.internal.PlatformDependent] [] -Dio.netty.noPreferDirect: false

An exception or error caused a run to abort: io.netty.handler.ssl.SslContextBuilder.protocols([Ljava/lang/String;)Lio/netty/handler/ssl/SslContextBuilder; 
java.lang.NoSuchMethodError: io.netty.handler.ssl.SslContextBuilder.protocols([Ljava/lang/String;)Lio/netty/handler/ssl/SslContextBuilder;
	at org.asynchttpclient.netty.ssl.DefaultSslEngineFactory.buildSslContext(DefaultSslEngineFactory.java:45)
	at org.asynchttpclient.netty.ssl.DefaultSslEngineFactory.init(DefaultSslEngineFactory.java:69)
	at org.asynchttpclient.netty.channel.ChannelManager.<init>(ChannelManager.java:116)
	at org.asynchttpclient.DefaultAsyncHttpClient.<init>(DefaultAsyncHttpClient.java:85)
	at com.softwaremill.sttp.asynchttpclient.AsyncHttpClientBackend$.defaultClient(AsyncHttpClientBackend.scala:314)
	at com.softwaremill.sttp.asynchttpclient.monix.AsyncHttpClientMonixBackend$.apply(AsyncHttpClientMonixBackend.scala:68)

I googled a bit and it looks like method io.netty.handler.ssl.SslContextBuilder.protocols was introduced in netty 4.1.9 (https://github.com/AsyncHttpClient/async-http-client/issues/1406). I run dependencyTree on my project and I can see that:

+-com.softwaremill.sttp:async-http-client-backend-monix_2.11:1.0.2 [S]
[info]   | +-com.softwaremill.sttp:async-http-client-backend_2.11:1.0.2 [S]
[info]   | | +-com.softwaremill.sttp:core_2.11:1.0.2 [S]
[info]   | | +-org.asynchttpclient:async-http-client:2.0.37
[info]   | |   +-com.typesafe.netty:netty-reactive-streams:1.0.8
[info]   | |   | +-io.netty:netty-handler:4.0.41.Final (evicted by: 4.0.52.Final)

what am I doing wrong? It doesn't look like I have mixed any dependency as the project just has 4 dependencies pulled in (scalatest, monix and two libraries from sttp ).

Request: Support non-string bodies for failed requests

This library makes a conscious decision to always parse the bodies of failed requests as strings. This is a needless restriction that makes this otherwise very nice looking library unusable in an environment where 4xx's are returned to clients that expect binary payloads with error information (in our case serialized protobufs).

The easiest change in my mind would to simply change the body Either formEither[String, T] to Either[Array[Byte], T] with a helper on Response to lazily deserialize the bytes into a string.

This looks like a relatively simple refactor to do (I'm considering doing it in a fork) - but I imagine this would be an annoying API break for existing users. Because of this I'm not trying to force this change upstream, but I wanted to flag the issue so it was on the project's radar.

Support exceptions in SttpBackendStub

Would be nice if something like this was supported:

implicit val testingBackend: SttpBackendStub[Future, Nothing] = SttpBackendStub(AkkaHttpBackend())
    .whenRequestMatches(_.uri.path.nonEmpty)
    .thenRespond(throw SomeException("Timeout or something else happened"))

This way you can write tests that describes certain cases, such as timout or connection errors.

Match any url in SttpBackendStub

Currently (I think) the only way to match any url is using .whenRequestMatches(_.uri.path.nonEmpty).
A simpler option would be handy, something like:

implicit val testingBackend: SttpBackendStub[Future, Nothing] = SttpBackendStub(AkkaHttpBackend())
    .whenAnyRequest
    .thenRespond( ... )

or perhaps

implicit val testingBackend: SttpBackendStub[Future, Nothing] = SttpBackendStub(AkkaHttpBackend())
    .whenRequestMatches(any)
    .thenRespond( ... )

URI params: `?` gets wrongly in some situations

I seem to have run into a situation where I see ? getting wrongly encoded.
Consider:

import com.softwaremill.sttp._

def uriP(param: Boolean) = uri"https://someapi.com${if (param) "?f=1" else ""}"

uriP(false) // https://someapi.com -- as expected
uriP(true) // https://someapi.com%3Ff=1 -- the `?` seems to be encoded

Version: 1.1.5

I can understand this may be a bit of an edge case, but in any case, the leading ? shouldn't be escaped.

Escaping embedded colons

Howdy! Thanks for the great project. It's made my work a lot easier.

I seem to have found a bug with escaping colons. From what you've written here it seems like this case should work, but it doesn't seem to.

Using 1.0.6 and Scala 2.11.8

import com.softwaremill.sttp._

println(uri"localhost:8998")
// http://localhost:8998

val str = "localhost:8998"
println(uri"$str")
// http://localhost%3A8998

Is this a bug or a user error?

Support for chunked HTTP?

Greetings --

I'm seeing the following error when working with an endpoint that responds with chunked HTTP:

HTTP chunk size exceeds the configured limit of 1048576 bytes

I use the AkkaHttpBackend backend w/ streaming so I assumed chunked http was handled out of the box. Did I miss something?

Thanks,

-Dragos

FutureAsyncHttpClientHandler hangs the application

I'm probably doing something wrong or not doing something obvious. I'm playing around with sttp and I have a simple object extending the App trait that makes a call using FutureAsyncHttpClientHandler backend. It works and I'm getting a response. The problem is the app seems to hang and I can exit it with either System.exit or even providing a custom executor and then shutting it down.
I tried doing this

object MyApp {
  def main(args: Array[String]): Unit = {
    import java.util.concurrent.ForkJoinPool
    val executor = new ForkJoinPool()
    implicit val context = ExecutionContext.fromExecutor(executor)
   /* 
   Todo.run uses FutureAsyncHttpClientHandler backend and issues a simple http request 
   returning a Future. Todo.run has an implicit ExecutionContext obviously so the context 
   that I create is used 
  */
   val result = Await.result(Todo.run, 3 seconds)
    try {
      import java.util.concurrent.TimeUnit
      executor.shutdown()
      executor.awaitTermination(1, TimeUnit.MINUTES)
    } catch {
      case e: Exception => println("Something went wrong!")
    }
  }
}

But it didn't change anything

sbt netty eviction warning

[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn] 	* io.netty:netty-handler:4.1.22.Final is selected over 4.1.13.Final
[warn] 	    +- org.asynchttpclient:async-http-client:2.4.4        (depends on 4.1.22.Final)
[warn] 	    +- com.typesafe.netty:netty-reactive-streams:2.0.0    (depends on 4.1.13.Final)
[warn] Run 'evicted' to see detailed eviction warnings

screen shot 2018-06-01 at 10 13 15

Comma is not interpreted correctly by `uri` interpolator

scala> import com.softwaremill.sttp._
import com.softwaremill.sttp._

scala> val a = "a"
a: String = a

scala> val b = "b"
b: String = b

scala> uri"http://example.com?q=$a,$b"
res0: com.softwaremill.sttp.Uri = http://example.com?q=a&q=,&q=b

scala> uri"http://example.com?q=${a + "," + b}"
res1: com.softwaremill.sttp.Uri = http://example.com?q=a,b

res0 should be equal to res1.

Websocket support

Any plans to add it? There is no solid websocket client library that I know in Scala. Except of akka-http but that you are bound into akka-stream context.

could not import AsyncHttpClientScalazBackend

I'm trying to use scalaz backend for my sttp but could not find one. Here's my sbt

name := "customer-order-specs"

version := "0.1"

scalaVersion := "2.12.5"

libraryDependencies += "com.softwaremill.sttp" %% "core" % "1.1.12"
libraryDependencies += "com.softwaremill.sttp" %% "async-http-client-backend" % "1.1.12"

libraryDependencies += "com.softwaremill.sttp" %% "async-http-client-handler-scalaz" % "0.0.13"

libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.0-SNAP10" % Test

libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.13.4" % "test"

I do not see ScalazBackend in async-http-client-handler-scalaz.jar.

screen shot 2018-04-21 at 6 47 06 pm

I'm trying to simple scalaz backed http client example, as described in README with HttpURLConnectionBackend

  test("scalaz example") {
    import com.softwaremill.sttp.asynchttpclient.scalaz._

    implicit val backend: SttpBackend[Id, Nothing] = AsyncHttpClientScalazBackend()
    val httpRequest = sttp
      .post(uri"http://httpbin.org/post")
      .body("Hello, world!")

    val httpResponse = httpRequest.send()

    httpResponse.body.isRight shouldBe true
  }

How to ignore ssl?

Is it possible to ignore invalid ssl certificate?

Im using HttpURLConnectionBackend but can switch to any other if needed.

a little bit awkward when trying to log the response

I want to log the response before converting to json, so I wrote

  def logRequest(uri: String, resStr: String): String = {
    println(resStr)
    resStr
  }

  val request = sttp.get(uri"https://httpbin.org/get")
    .mapResponse(logRequest(uri, _))
    .response(asJson[HttpBinResponse])
    .send().foreach(println)

It's not working, because mapResponse would apply map to ResponseAs,
and response would set a new ResponseAs, ignore the previous one completely.

  • Is it possible to add compile error or a warning for such code?

and I end up logging with res: Response[String] like this

    if (res.isSuccess) {
      val resStr = res.body.right.get
      val respTruncated = if (resStr.length > 300) resStr.substring(0, 300) + "..." else resStr
      clientLog.info("request to {} with {} got {}", uri, reqBody, respTruncated)
      Success(resStr)
    } else {
      clientLog.error("response to {} with {} got code {}, body {}", uri, reqBody, res.code + "", res.body.left.get)
      Failure(UnknownServerError)
    }
  • Is there some kind of best practice for logging the raw response?

Can't get responses for an async request?

Hi, I tried sttp to send some asynchronous request, but it can't get response and timeout at last.

My code is:

package my

import com.softwaremill.sttp._
import com.softwaremill.sttp.asynchttpclient.future.AsyncHttpClientFutureBackend

import scala.concurrent.Future
import scala.util.{Failure, Success}

object HttpClientDemo2 extends App {

  import scala.concurrent.ExecutionContext.Implicits.global

  implicit val backend: SttpBackend[Future, Nothing] = AsyncHttpClientFutureBackend()
  val query = "http language:scala"
  val request = sttp
    .get(uri"https://api.github.com/search/repositories?q=$query")
  println("send request!")

  request.send().onComplete {
    case Success(response) =>
      println("--------------- async response ----------------")
      println(response.header("Content-Length"))
      println(response.unsafeBody)
      backend.close()
    case Failure(e) => e.printStackTrace()
  }
}

and my sbt dependencies is:

scalaVersion := "2.12.4"

sbtVersion := "1.0.4"

libraryDependencies ++= Seq(
  "com.softwaremill.sttp" %% "core" % "1.1.2",
  "io.netty" % "netty-handler" % "4.0.54.Final",
  "io.netty" % "netty-codec-http" % "4.0.54.Final",
  "org.asynchttpclient" % "netty-resolver" % "2.0.38",
  "org.slf4j" % "slf4j-nop" % "1.7.25",
  "com.softwaremill.sttp" %% "akka-http-backend" % "1.1.2",
  "com.softwaremill.sttp" %% "json4s" % "1.1.2",
  "org.reactivestreams" % "reactive-streams" % "1.0.2",
  "org.asynchttpclient" % "async-http-client" % "2.0.38",
  "com.softwaremill.sttp" %% "async-http-client-backend" % "1.1.2",
  "com.softwaremill.sttp" %% "async-http-client-backend-future" % "1.1.2",
  "org.scalatest" %% "scalatest" % "3.0.4" % "test"
)

Could you help where is wrong in my code? I also search sttp document and found there is an akka backend demo, which has the same problem in my computer. Thanks!

URI parameter: symbol `;` not encoded properly.

import com.softwaremill.sttp._

scala> val u = uri"http://localhost:8080/query?q=firstQuery;secondQuery"
u: com.softwaremill.sttp.Uri = http://localhost:8080/query?q=firstQuery;secondQuery

it should looks like: http://localhost:8080/query?q=firstQuery%3BsecondQuery.
Using latest sttp version.

Encoding of URL paths is incorrect.

Version: okhttp-handler-monix 0.0.9.

The URI.toString method is encoding URL paths using URLEncoder.encode.

This String is then used to build the OkHttpRequest here.

As URLEncoder.encode is meant to encode query parameters, it can generate incorrect path for chararacters being usually encoded in queries.

The instance I ran into is regarding the character '=' in the following URL:

http://localhost:8080/key=something encoded into http://localhost:8080/key%3Dsomething

This causes the request to fail with status code 404 as the path does not exist.

I believe it should be a rather straightforward fix.

FYI I have found some similar discussions online:

Thank you for this nice lib 👍

Support raw json in Response

Would be useful if you can test that a json response is decoded and handled correctly.

Something like:

implicit val testingBackend: SttpBackendStub[Future, Nothing] = SttpBackendStub(AkkaHttpBackend())
    .whenRequestMatches(_.uri.path.nonEmpty)
    .thenRespond(json"{ 'myjson' : 'here' }")

Cross-compile to Scala.js

I'm working on this in this branch https://github.com/joshlemer/sttp/tree/scalajs

Here I will keep a list of any java dependencies we would need to eliminate, or implement in scala:

java.net.IDN

[error] Referring to non-existent class java.net.IDN$
[error]   called from com.softwaremill.sttp.Uri.encodeHost()java.lang.String
[error]   called from com.softwaremill.sttp.Uri.toString()java.lang.String
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.$$anonfun$new$5(com.softwaremill.sttp.Uri,java.lang.String)org.scalatest.compatible.Assertion
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.$$anonfun$new$4(java.lang.String,scala.Tuple2)scala.Unit
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.$$anonfun$new$2(scala.Tuple2)scala.Unit
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.<init>()
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.__exportedInits
[error]   exported to JavaScript with @JSExport

java.net.URLEncoder

[error] Referring to non-existent class java.net.URLEncoder$
[error]   called from com.softwaremill.sttp.Uri.encodeQuery(java.lang.String,com.softwaremill.sttp.Uri$QueryFragmentEncoding)java.lang.String
[error]   called from com.softwaremill.sttp.Uri.encodeQueryFragments$1(scala.collection.immutable.List,scala.Boolean,scala.collection.mutable.StringBuilder)java.lang.String
[error]   called from com.softwaremill.sttp.Uri.toString()java.lang.String
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.$$anonfun$new$5(com.softwaremill.sttp.Uri,java.lang.String)org.scalatest.compatible.Assertion
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.$$anonfun$new$4(java.lang.String,scala.Tuple2)scala.Unit
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.$$anonfun$new$2(scala.Tuple2)scala.Unit
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.<init>()
[error]   called from com.softwaremill.sttp.UriInterpolatorTests.__exportedInits
[error]   exported to JavaScript with @JSExport

java.io.File

[error] Referring to non-existent class java.io.File
[error]   called from com.softwaremill.sttp.testing.SttpBackendStub$.tryAdjustResponseBody(com.softwaremill.sttp.ResponseAs,java.lang.Object)scala.Option
[error]   called from com.softwaremill.sttp.testing.SttpBackendStub$.tryAdjustResponseType(com.softwaremill.sttp.ResponseAs,com.softwaremill.sttp.Response)com.softwaremill.sttp.Response
[error]   called from com.softwaremill.sttp.testing.SttpBackendStub.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.RequestT.send(com.softwaremill.sttp.SttpBackend,scala.Predef$$eq$colon$eq)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.$$anonfun$new$12()org.scalatest.compatible.Assertion
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.<init>()
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.__exportedInits
[error]   exported to JavaScript with @JSExport

java.nio.files.Files

[error] Referring to non-existent class java.nio.file.Files$
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.writeBasicBody(com.softwaremill.sttp.BasicRequestBody,java.io.OutputStream)scala.Unit
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.writeBody(com.softwaremill.sttp.RequestBody,java.net.HttpURLConnection)scala.Option
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)com.softwaremill.sttp.Response
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.sendWithCounter(com.softwaremill.sttp.RequestT,scala.Int)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStub.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.RequestT.send(com.softwaremill.sttp.SttpBackend,scala.Predef$$eq$colon$eq)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.$$anonfun$new$12()org.scalatest.compatible.Assertion
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.<init>()
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.__exportedInits
[error]   exported to JavaScript with @JSExport

java.nio.channels.Channels

[error] Referring to non-existent class java.nio.channels.Channels$
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.writeBasicBody(com.softwaremill.sttp.BasicRequestBody,java.io.OutputStream)scala.Unit
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.writeBody(com.softwaremill.sttp.RequestBody,java.net.HttpURLConnection)scala.Option
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)com.softwaremill.sttp.Response
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.sendWithCounter(com.softwaremill.sttp.RequestT,scala.Int)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStub.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.RequestT.send(com.softwaremill.sttp.SttpBackend,scala.Predef$$eq$colon$eq)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.$$anonfun$new$12()org.scalatest.compatible.Assertion
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.<init>()
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.__exportedInits
[error]   exported to JavaScript with @JSExport

java.util.zip.InflaterInputStream

[error] Referring to non-existent class java.util.zip.InflaterInputStream
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.wrapInput(scala.Option,java.io.InputStream)java.io.InputStream
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.readResponse(java.net.HttpURLConnection,java.io.InputStream,com.softwaremill.sttp.ResponseAs)com.softwaremill.sttp.Response
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)com.softwaremill.sttp.Response
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.sendWithCounter(com.softwaremill.sttp.RequestT,scala.Int)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStub.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.RequestT.send(com.softwaremill.sttp.SttpBackend,scala.Predef$$eq$colon$eq)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.$$anonfun$new$12()org.scalatest.compatible.Assertion
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.<init>()
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.__exportedInits
[error]   exported to JavaScript with @JSExport

java.util.zip.GZIPInputStream

[error] Referring to non-existent class java.util.zip.GZIPInputStream
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.wrapInput(scala.Option,java.io.InputStream)java.io.InputStream
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.readResponse(java.net.HttpURLConnection,java.io.InputStream,com.softwaremill.sttp.ResponseAs)com.softwaremill.sttp.Response
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)com.softwaremill.sttp.Response
[error]   called from com.softwaremill.sttp.HttpURLConnectionBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.sendWithCounter(com.softwaremill.sttp.RequestT,scala.Int)java.lang.Object
[error]   called from com.softwaremill.sttp.FollowRedirectsBackend.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStub.send(com.softwaremill.sttp.RequestT)java.lang.Object
[error]   called from com.softwaremill.sttp.RequestT.send(com.softwaremill.sttp.SttpBackend,scala.Predef$$eq$colon$eq)java.lang.Object
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.$$anonfun$new$12()org.scalatest.compatible.Assertion
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.<init>()
[error]   called from com.softwaremill.sttp.testing.SttpBackendStubTests.__exportedInits

java.net.HttpCookie

sttp uses this class in a few cases for parsing cookies. and things like that. Should be a medium amount of work to port an existing implementation to scala (again, not the jdk version because of licensing). OkHttp's implementation is licensed under Apache, so it is ok to fork/port I think.

A generic backend for request throttling

Maybe it would be possible to create a generic backend wrapper (maybe generic for all backends, maybe only for asynchronous ones?) which would allow throttling of requests?

Throttling could be per-host or per-host+path.

Or maybe there is a throttling library which we could use?

Artifact resolution problem

I can't manage to get it working with other handlers like Monix or Future, because of a resolution problem with dependency async-http-client-handler.

Tested in version 0.0.2

Timeout when using MonixAsyncHttpClientHandler

The following snippet throws TimeoutException

val sort: Option[String] = None
val query = "http language:scala"
val request = sttp.get(uri"https://api.github.com/search/repositories?q=$query&sort=$sort")

implicit val handler = MonixAsyncHttpClientHandler()
val response =
  Observable.fromTask {
    request.response(asStream[Observable[ByteBuffer]]).send()
  }
  .flatMap(_.body)
  .map(StandardCharsets.UTF_8.decode)
  .consumeWith(Consumer foreach print)
Await.result(response.runAsync(Scheduler.Implicits.global), Duration.Inf)

Stacktrace

Exception in thread "main" java.util.concurrent.TimeoutException: Request timeout to api.github.com/192.30.253.117:443 after 60000 ms
	at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43)
	at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:48)
	at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663)
	at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738)
	at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466)
	at java.lang.Thread.run(Thread.java:745)

But this one works perfectly

val response =
  Observable.fromTask {
    request.send()
  }
  .flatMap(str => Observable.apply(str.body))
  .consumeWith(Consumer foreach print)
Await.result(response.runAsync(Scheduler.Implicits.global), Duration.Inf)

The only difference is the body streaming. I can't think in a cause for the timeout

URL encoded '/' characters are unencoded back to slash

Tried using the uri format string and passing in a java.net.Uri with pre-escaped query string, but both incorrectly handle slashes in the parameter itself (the latter UNENCODES the slashes?!?)

sample:
val testparm = "this/has/slashes"
val encparm = URLEncoder.encode(testparm,"UTF-8")
val uri = new java.net.URI(s"http://notahost.com?query=$encparm")

sttp.get(Uri(uri))
sttp.get(uri"http://notahost.com?query=$testparm")

testparm: String = this/has/slashes
encparm: String = this%2Fhas%2Fslashes
  uri: java.net.URI = http://notahost.com?query=this%2Fhas%2Fslashes

res0: com.softwaremill.sttp.Request[String,Nothing] = RequestT(Method(GET),http://notahost.com?query=this/has/slashes,NoBody,Vector((Accept-Encoding,gzip, deflate)),ResponseAsString(utf-8),RequestOptions(true,1 minute),Map())
res1: com.softwaremill.sttp.Request[String,Nothing] = RequestT(Method(GET),http://notahost.com?query=this/has/slashes,NoBody,Vector((Accept-Encoding,gzip, deflate)),ResponseAsString(utf-8),RequestOptions(true,1 minute),Map())

Plus sign is not escaped correctly

scala> uri"http://localhost:8081/+1"
res5: com.softwaremill.sttp.Uri = http://localhost:8081/%201

scala> uri"http://localhost:8081/ 1"
res6: com.softwaremill.sttp.Uri = http://localhost:8081/%201

scala> res5 == res6
res7: Boolean = true

Workaround:

scala> uri"http://localhost:8081/%2B1"
res10: com.softwaremill.sttp.Uri = http://localhost:8081/+1

MalformedInputException when using HttpURLConnectionBackend

Hi, the HttpURLConnectionBackend backend has exception at getting some sites , while Akka backend successeed.
I also experiment on Httpie, that work fine.
I think it is related to charset problem, but what confuse me is

  1. other backend success.
  2. if it was codec error, it should return Either.Left as Response body since it is parsing error of responce(asString) .

This is HttpURLConnectionBackend backend to get google.com

@ {
  import $ivy.`com.softwaremill.sttp::core:1.1.1`
  import com.softwaremill.sttp._
  implicit val backend = HttpURLConnectionBackend()
  sttp.get(uri"http://google.com").send()
  }
java.nio.charset.MalformedInputException: Input length = 1
  java.nio.charset.CoderResult.throwException(CoderResult.java:281)
  sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
  sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
  java.io.InputStreamReader.read(InputStreamReader.java:184)
  java.io.BufferedReader.read1(BufferedReader.java:210)
  java.io.BufferedReader.read(BufferedReader.java:286)
  java.io.Reader.read(Reader.java:140)
  scala.io.BufferedSource.mkString(BufferedSource.scala:94)
  com.softwaremill.sttp.HttpURLConnectionBackend.asString$1(HttpURLConnectionBackend.scala:231)
  com.softwaremill.sttp.HttpURLConnectionBackend.readResponseBody(HttpURLConnectionBackend.scala:241)
  com.softwaremill.sttp.HttpURLConnectionBackend.readResponse(HttpURLConnectionBackend.scala:220)
  com.softwaremill.sttp.HttpURLConnectionBackend.send(HttpURLConnectionBackend.scala:47)
  com.softwaremill.sttp.HttpURLConnectionBackend.send(HttpURLConnectionBackend.scala:16)
  com.softwaremill.sttp.FollowRedirectsBackend.sendWithCounter(FollowRedirectsBackend.scala:15)
  com.softwaremill.sttp.FollowRedirectsBackend.followRedirect(FollowRedirectsBackend.scala:57)
  com.softwaremill.sttp.FollowRedirectsBackend.$anonfun$followRedirect$2(FollowRedirectsBackend.scala:37)
  scala.Option.fold(Option.scala:158)
  com.softwaremill.sttp.FollowRedirectsBackend.followRedirect(FollowRedirectsBackend.scala:33)
  com.softwaremill.sttp.FollowRedirectsBackend.$anonfun$sendWithCounter$1(FollowRedirectsBackend.scala:19)
  com.softwaremill.sttp.IdMonad$.flatMap(MonadError.scala:38)
  com.softwaremill.sttp.FollowRedirectsBackend.sendWithCounter(FollowRedirectsBackend.scala:17)
  com.softwaremill.sttp.FollowRedirectsBackend.send(FollowRedirectsBackend.scala:10)
  com.softwaremill.sttp.RequestT.send(RequestT.scala:242)
  ammonite.$sess.cmd4$.<init>(cmd4.sc:4)
  ammonite.$sess.cmd4$.<clinit>(cmd4.sc)

this is Akka backend that has response body.

@ {
    import $ivy.`com.softwaremill.sttp::core:1.1.1`
    import $ivy.`com.softwaremill.sttp::akka-http-backend:1.1.1`
    import com.softwaremill.sttp._
    import com.softwaremill.sttp.akkahttp._
    import scala.concurrent.ExecutionContext.Implicits.global
    implicit val backend = AkkaHttpBackend()
    sttp.get(uri"http://google.com").send().foreach(println)
  }
@ Response(Right(<!doctype html><html ... ommit long HTML.

add statusText/reason/message to Response

very common, how to name it better?

lib int text
sttp code add statusText/reason/message?
js fetch status statusText
gigahorse status statusText
akka http StatusCode.intValue StatusCode.reason
HttpURLConnection getResponseCode getResponseMessage
asynchttpclient getStatusCode getStatusText
okhttp3 code message

SttpBackendStub.thenRespondOk() creates a Response with Left(body) instead of Right(body)

This is not what I expected. I was given to understand that the contract for Sttp is that successful responses should always be a Right(body). I think this should change from:

 def thenRespondOk(): SttpBackendStub[R, S] =
      thenRespondWithCode(200)

to

 def thenRespondOk(): SttpBackendStub[R, S] =
      thenRespond("")

Another option is to have thenRespondWithCode change to:

def thenRespondWithCode(code: Int,
                            msg: String = ""): SttpBackendStub[R, S] =
      val body = if (code >= 200 && code < 300) Right(msg) else Left(msg)
      thenRespond(Response[Nothing](body, code, Nil, Nil))

Which also allow more sensible mocking of 201 etc responses.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.