fukamachi / dexador Goto Github PK

View Code? Open in Web Editor NEW

362.0 16.0 42.0 609 KB

A fast HTTP client for Common Lisp

Home Page: http://ultra.wikia.com/wiki/Dexador

Common Lisp 99.26% HTML 0.13% Makefile 0.11% Dockerfile 0.50%

dexador's Issues

What about ssl?

I glanced through code and it looks like dexador only supports client certificates. But what about

Server certificate validation with ability to provide custom cafile/capath
Hostname validation?

read-byte blocks and waits for the next input.

(ql:quickload '(:clack :dexador))
(clack:clackup
 (lambda (env)
   (declare (ignore env))
   '(204 () nil))
 :server :woo :port 5000)

(dex:get "http://localhost:5000")

and nothing will be returned.

Output operation on closed SSL stream

(handler-bind ((dex:http-request-failed #'dex:retry-request))
    (dexador:post "https://api001.backblazeb2.com/b2api/v1/b2_get_upload_url" 
        :content "{\"bucketId\": \"b78178f0df3db6975dd20544\"}" 
        :headers (list (cons "Authorization" "Something"))))

returns

output operation on closed SSL stream
   [Condition of type SIMPLE-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1002EB7FA3}>)

Backtrace:
  0: ((:METHOD SB-GRAY:STREAM-FORCE-OUTPUT (CL+SSL::SSL-STREAM)) #<unavailable argument>) [fast-method]
  1: (FORCE-OUTPUT #<CL+SSL::SSL-STREAM for #<FD-STREAM for "socket xxx.xxx.xxx.xxx:35752, peer: xxx.xxx.xxx.xxx:443" {10082339E3}>>)
  2: (DEXADOR.BACKEND.USOCKET:REQUEST #<unavailable argument> :METHOD :POST :CONTENT "{\"bucketId\": \"b78178f0df3db6975dd20513\"}" :HEADERS (("Authorization" . "Password")))
  3: ((LAMBDA ()))

It looks like in usocket.lisp an assumption is made that the stream is still open when force-output is called on it. A quick look at cl-plus-ssl would seem to indicate that using ssl-stream-handle can be used to see if a stream is still open. If the stream is not open then maybe the retry code in usocket.lisp should start a new connection?

dex:post fails with simple request

Hello @fukamachi ,

First of all thank you for writing this library!

I'm having issues using it though. Whilst a simple cURL call works:

$ curl -X POST \
> https://xyz.com/rest/auth/1/session \
> -H 'content-type: application/json' \
> -d '{"username":"alberto.ferreira", "password":"xxx"}'

{"session":{"name":"JSESSIONID","value":"71CE8.......................}

dex:post fails:

(dex:post *SERVER-AUTH-URI*
          :headers '(("content-type" . "application/json"))
          :content `(("username" . ,*user*)
                     ("password" . ,*pass*)))

;; stacktrace

An HTTP request to "https://xyz.com/rest/auth/1/session" returned 400 bad request.

{"errorMessages":["Unexpected character ('u' (code 117)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: org.apache.catalina.connector.CoyoteInputStream@534e4f67; line: 1, column: 2]"]}
   [Condition of type DEXADOR.ERROR:HTTP-REQUEST-BAD-REQUEST]

Strangely, if I change "username" to "dusername", it starts complaining of character 'd' instead of 'u', which seems to be issues with some UTF-8 thing? I don't need UTF-8 here, as all my inputs are ASCII.

Can you help? Thank you!

Using with crontab throws `/etc/mime.types` error

Description
When Dexador is used in the crontab script, the crontab throws the following error:

Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (Unhandled SB-INT:STREAM-DECODING-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (                                                    {10005385B3}>:)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (  :ASCII stream decoding error on)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (  #<SB-SYS:FD-STREAM for "file /etc/mime.types" {1003F0A383}>:)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (    the octet sequence #(194) cannot be decoded.)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT ()
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10005385B3}>)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SB-INT:STREAM-DECODING-ERROR {1003F63BE3}> #<unused argument> :QUIT T))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK* #<SB-INT:STREAM-DECODING-ERROR {1003F63BE3}>))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (2: (INVOKE-DEBUGGER #<SB-INT:STREAM-DECODING-ERROR {1003F63BE3}>))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (3: (ERROR #<SB-INT:STREAM-DECODING-ERROR {1003F63BE3}>))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (4: (SB-KERNEL:WITH-SIMPLE-CONDITION-RESTARTS ERROR NIL SB-INT:STREAM-DECODING-ERROR :EXTERNAL-FORMAT :ASCII :STREAM #<SB-SYS:FD-STREAM for >
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (5: (SB-IMPL::STREAM-DECODING-ERROR-AND-HANDLE #<SB-SYS:FD-STREAM for "file /etc/mime.types" {1003F0A383}> 1))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (6: (SB-IMPL::FD-STREAM-READ-N-CHARACTERS/ASCII #<SB-SYS:FD-STREAM for "file /etc/mime.types" {1003F0A383}> #<(SIMPLE-ARRAY CHARACTER (512))>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (application/vnd.geocube+xml                        g3 g3)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (application/vnd.fujitsu.oasysgp                        fg5)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (application/vnd.fujitsu.oasysprs                bh2)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (application/vnd.fujixerox... {1003F0A4EF}> 4 508 NIL))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (7: (SB-INT:FAST-READ-CHAR-REFILL #<SB-SYS:FD-STREAM for "file /etc/mime.types" {1003F0A383}> NIL))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (8: (SB-IMPL::ANSI-STREAM-READ-LINE-FROM-FRC-BUFFER #<SB-SYS:FD-STREAM for "file /etc/mime.types" {1003F0A383}> NIL NIL))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (9: (BUILD-MIME-DB #P"/etc/mime.types"))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (10: (SB-FASL::LOAD-FASL-GROUP #S(SB-FASL::FASL-INPUT :STREAM #<SB-SYS:FD-STREAM for "file /home/sheep/.cache/common-lisp/sbcl-1.5.1-linux-x>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (The file should have the following structure:)
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT ()
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (MIME-TYPE FILE-EXTENSION*" #5=#(#(2 4 9 15 34 11 61 48) *MIME-DB* *REVERSE-MIME-DB* %READ-TOKENS FIND-MIME.TYPES)) #4# #5# *MIME-DB* *REVER>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (11: (SB-FASL::LOAD-AS-FASL #<SB-SYS:FD-STREAM for "file /home/sheep/.cache/common-lisp/sbcl-1.5.1-linux-x64/home/sheep/.roswell/lisp/quickl>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (12: ((FLET SB-FASL::THUNK :IN LOAD)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (13: (SB-FASL::CALL-WITH-LOAD-BINDINGS #<CLOSURE (FLET SB-FASL::THUNK :IN LOAD) {7F9FC86FD8DB}> #<SB-SYS:FD-STREAM for "file /home/sheep/.ca>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (14: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) #<SB-SYS:FD-STREAM for "file /home/sheep/.cache/common-lisp/sbcl-1.5.1-linux-x64/home/sheep/.rosw>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (15: (LOAD #P"/home/sheep/.cache/common-lisp/sbcl-1.5.1-linux-x64/home/sheep/.roswell/lisp/quicklisp/dists/quicklisp/software/trivial-mimes->
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (16: (UIOP/UTILITY:CALL-WITH-MUFFLED-CONDITIONS #<CLOSURE (LAMBDA NIL :IN UIOP/LISP-BUILD:LOAD*) {1003EFDADB}> ("Overwriting already existin>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (17: ((SB-PCL::EMF ASDF/ACTION:PERFORM) #<unused argument> #<unused argument> #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (18: ((LAMBDA NIL :IN ASDF/ACTION:CALL-WHILE-VISITING-ACTION)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (19: ((:METHOD ASDF/ACTION:PERFORM-WITH-RESTARTS (ASDF/LISP-ACTION:LOAD-OP ASDF/LISP-ACTION:CL-SOURCE-FILE)) #<ASDF/LISP-ACTION:LOAD-OP > #<>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (20: ((:METHOD ASDF/ACTION:PERFORM-WITH-RESTARTS :AROUND (T T)) #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "trivial-mime>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (21: ((:METHOD ASDF/PLAN:PERFORM-PLAN (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {10023C20A3}>)[fast-method])
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (21: ((:METHOD ASDF/PLAN:PERFORM-PLAN (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {10023C20A3}>) [fast-method])
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (22: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (23: ((:METHOD ASDF/PLAN:PERFORM-PLAN :AROUND (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {10023C20A3}>) [fast-method])
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (24: ((:METHOD ASDF/OPERATE:OPERATE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)) #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/SYSTEM:SYSTEM ">
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (25: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/SYSTEM:SYSTEM "dexador"> >
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (26: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (27: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/SYSTEM:SYSTEM "dexador"> :VERBOSE NIL) [fast-method])
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (28: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> ASDF/LISP-ACTION:LOAD-OP "dexador" :VERBOSE NIL))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (29: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (30: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:LOAD-OP "dexador" :VERBOSE NIL) [fast-method])
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (31: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<CLOSURE (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {10023AD98B}> :OVERRIDE T :KEY NIL :OVERRIDE-CACHE>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (32: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (33: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<CLOSURE (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {100239C84B}> :OVERRIDE NIL :KEY NIL :OVERRIDE-CAC>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (34: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:LOAD-OP "dexador" :VERBOSE NIL) [fast-method])
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (35: (ASDF/OPERATE:LOAD-SYSTEM "dexador" :VERBOSE NIL))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (36: (QUICKLISP-CLIENT::CALL-WITH-MACROEXPAND-PROGRESS #<CLOSURE (LAMBDA NIL :IN QUICKLISP-CLIENT::APPLY-LOAD-STRATEGY) {100239A9FB}>))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (37: (QUICKLISP-CLIENT::AUTOLOAD-SYSTEM-AND-DEPENDENCIES "dexador" :PROMPT NIL))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (38: ((:METHOD QL-IMPL-UTIL::%CALL-WITH-QUIET-COMPILATION (T T)) #<unused argument> #<CLOSURE (FLET QUICKLISP-CLIENT::QL :IN QUICKLISP-CLIEN>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (39: ((:METHOD QL-IMPL-UTIL::%CALL-WITH-QUIET-COMPILATION :AROUND (QL-IMPL:SBCL T)) #<QL-IMPL:SBCL {10032D4873}> #<CLOSURE (FLET QUICKLISP-C>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (40: ((:METHOD QUICKLISP-CLIENT:QUICKLOAD (T)) (COMMON-LISP-USER::DEXADOR) :PROMPT NIL :SILENT T :VERBOSE NIL) [fast-method])
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (41: (QL-DIST::CALL-WITH-CONSISTENT-DISTS #<CLOSURE (LAMBDA NIL :IN QUICKLISP-CLIENT:QUICKLOAD) {10021B69CB}>))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (42: (SB-INT:SIMPLE-EVAL-IN-LEXENV (QUICKLISP-CLIENT:QUICKLOAD (QUOTE (COMMON-LISP-USER::DEXADOR)) :SILENT T) #<NULL-LEXENV>))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (43: (SB-INT:SIMPLE-EVAL-IN-LEXENV (PROGN (ROSWELL:ENSURE-ASDF) (QUICKLISP-CLIENT:QUICKLOAD (QUOTE (COMMON-LISP-USER::DEXADOR)) :SILENT T)) >
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (44: (SB-EXT:EVAL-TLF (PROGN (ROSWELL:ENSURE-ASDF) (QUICKLISP-CLIENT:QUICKLOAD (QUOTE (COMMON-LISP-USER::DEXADOR)) :SILENT T)) NIL NIL))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (45: ((LABELS SB-FASL::EVAL-FORM :IN SB-INT:LOAD-AS-SOURCE) (PROGN (ROSWELL:ENSURE-ASDF) (QUICKLISP-CLIENT:QUICKLOAD (QUOTE (COMMON-LISP-USE>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (46: (SB-INT:LOAD-AS-SOURCE #<CONCATENATED-STREAM :STREAMS (#<SB-SYS:FD-STREAM for "file /home/sheep/temp/test.ros" {1002135C13}> #<SB-IMPL:>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (47: ((FLET SB-FASL::THUNK :IN LOAD)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (48: (SB-FASL::CALL-WITH-LOAD-BINDINGS #<CLOSURE (FLET SB-FASL::THUNK :IN LOAD) {7F9FC86FF53B}> #<CONCATENATED-STREAM :STREAMS (#<SB-SYS:FD->
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (49: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) #<CONCATENATED-STREAM :STREAMS (#<SB-SYS:FD-STREAM for "file /home/sheep/temp/test.ros" {1002135C>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (50: (LOAD #<CONCATENATED-STREAM :STREAMS (#<SB-SYS:FD-STREAM for "file /home/sheep/temp/test.ros" {1002135C13}> #<SB-IMPL::STRING-INPUT-STR>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (51: ((FLET ROSWELL::BODY :IN ROSWELL:SCRIPT) #<SB-SYS:FD-STREAM for "file /home/sheep/temp/test.ros" {1002135C13}>))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (52: (ROSWELL:SCRIPT "/home/sheep/temp/test.ros"))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (53: (ROSWELL:RUN ((:EVAL "(ros:quicklisp)") (:SCRIPT "/home/sheep/temp/test.ros") (:QUIT NIL))))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (54: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ROSWELL:RUN (QUOTE ((:EVAL "(ros:quicklisp)") (:SCRIPT "/home/sheep/temp/test.ros") (:QUIT NIL)))) #<NUL>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (55: (EVAL (ROSWELL:RUN (QUOTE ((:EVAL "(ros:quicklisp)") (:SCRIPT "/home/sheep/temp/test.ros") (:QUIT NIL))))))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (56: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:EVAL . "(progn #-ros.init(cl:load \"/usr/etc/roswell/init.lisp\"))") (:EVAL . "(ros:run '((:eval>
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (57: (SB-IMPL::TOPLEVEL-INIT))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (58: ((FLET SB-UNIX::BODY :IN SB-EXT:SAVE-LISP-AND-DIE)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (59: ((FLET "WITHOUT-INTERRUPTS-BODY-14" :IN SB-EXT:SAVE-LISP-AND-DIE)))
Jun 13 10:46:03 work CROND[21648]: (sheep) CMDOUT (60: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE)))

When executed from the RPEL there is no problem at all.
Also when I used Drakma instead of Dexador this problem disappears.

How to reproduce

Create a Roswell script using the following command.

ros init test

Open it and use Dexador to randomly visit a website.

(dex:get "http://www.google.com")

Setup crontab to execute this script on some time.
Open the journal of crontab and you will see the error.

journalctl -u cronie

First guess
My guess is, the problem is caused by library trivial-mimes, which reads /etc/mimes.type file.

Any fix?

:timeout documentation is outdated

:timeout keyword parameter was split into connection timeout and read timeout but documentation was not updated.

HTTPS request hangs

Example:

(dex:get "https://api.github.com/")

This simply hangs, if I interrupt the process, the backtrace looks like:

Backtrace:
  0: (SB-THREAD:THREAD-YIELD) [external]
  1: (SB-SYS:WAIT-UNTIL-FD-USABLE 400 :INPUT NIL T)
  2: (CL+SSL:MAKE-SSL-CLIENT-STREAM #<unavailable argument> :CERTIFICATE #<unavailable argument> :KEY #<unavailable argument> :PASSWORD #<unavailable argument> :METHOD #<unavailable argument> :EXTERNAL-FOR..
  3: (CL+SSL::CALL-WITH-GLOBAL-CONTEXT #.(SB-SYS:INT-SAP #X07444650) T #<CLOSURE (LAMBDA NIL :IN DEXADOR.BACKEND.USOCKET:REQUEST) {1008B453AB}>)
  4: (DEXADOR.BACKEND.USOCKET:REQUEST #<unavailable argument> :METHOD :GET)

I'm on Win7 x64, SBCL 1.3.15, OpenSSL 1.0.2g 1 Mar 2016, using the quicklisp distribution of Dexador (version 0.9.10).

unread-char for decoding-stream broken for multibyte encodings

In STREAM-READ-CHAR, the first value returned from the BABEL:CODE-POINT-COUNTER function was used to track LAST-CHAR-SIZE (the number of bytes consumed while reading a character). However, it is actually the number of characters decoded, and for this particular call, it is always 1. As a result, a later STREAM-UNREAD-CHAR of a multibyte character will cause BUFFER-POSITION to point into the middle of the encoded byte sequence, ultimately leading to a decoding error.

added a fix in #38

Streaming upload

Master branch fixes SSL issue

I was getting these errors:
X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN
X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY
and after cloning the master-branch to quicklisp/local-projects they went away.

Just posting this in case anyone else gets stuck.

The alien function "CRYPTO_num_locks" is undefined

Hi,

dex:get raises an undefined function error when getting a https URL,
for example:

(defvar *url* "https://lispcookbook.github.io/cl-cookbook/web-scraping.html")
(defvar *html* (dex:get *url*))

will result in:

The alien function "CRYPTO_num_locks" is undefined.
     [Condition of type SB-KERNEL::UNDEFINED-ALIEN-FUNCTION-ERROR]

Restarts:
   0: [RETRY] Retry SLIME interactive evaluation request.
   1: [*ABORT] Return to SLIME's top level.
   2: [ABORT] abort thread (#<THREAD "worker" RUNNING {1007B459B3}>)

Backtrace:
    0: ("undefined function")
    1: (CL+SSL::CRYPTO-NUM-LOCKS)
    2: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN CL+SSL:ENSURE-INITIALIZED))
    3: ((FLET #:WITHOUT-INTERRUPTS-BODY-386 :IN SB-THREAD::CALL-WITH-RECURSIVE-LOCK))
    4: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN CL+SSL:ENSURE-INITIALIZED) {7FFFEE5A56BB}> #<SB-THREAD:MUTEX "SSL initialization" owner: #<SB-THREAD:THREA..
    5: (CL+SSL:ENSURE-INITIALIZED :METHOD #<unavailable argument> :RAND-SEED #<unavailable argument>)
    6: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://lispcookbook.github.io/cl-cookbook/web-scraping.html>)
    7: (DEXADOR.BACKEND.USOCKET:REQUEST #<unavailable argument> :METHOD :GET)
    8: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEXADOR:GET *URL*) #<NULL-LEXENV>)
    9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (UNLESS (BOUNDP (QUOTE *HTML*)) (DEXADOR:GET *URL*)) #<NULL-LEXENV>)
   10: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SB-IMPL::%DEFVAR (QUOTE *HTML*) (SB-C:SOURCE-LOCATION) (UNLESS (BOUNDP #) (DEXADOR:GET *URL*))) #<NULL-LEXENV>)
   11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEFVAR *HTML* (DEXADOR:GET *URL*)) #<NULL-LEXENV>)
   12: (EVAL (DEFVAR *HTML* (DEXADOR:GET *URL*)))
   13: ((LAMBDA NIL :IN SWANK:INTERACTIVE-EVAL))
   --more--

why is that?

versions:

CL implementation: SBCL 1.3.18-1.fc26
dexador-20170516-git

Thanks.

Connection stream is not closed properly in request function

When using dexador on Linux (Ubuntu 20.04) with SBCL 2.0.9, SBCL will promptly crash on the second POST request made with a return code of 141 and when using :content. This seems to be because the stream handle doesn't seem to be properly closed, either in this form, or in the (finalize-connection) function, perhaps due to the if logic there. I was able to fix it for my purposes by explicitly closing the stream in the final unwind-protect:

(unwind-protect
                    (let ((body (convert-body body
                                              (gethash "content-encoding" response-headers)
                                              (gethash "content-type" response-headers)
                                              content-length
                                              transfer-encoding-p
                                              force-binary
                                              force-string
                                              (connection-keep-alive-p
                                               (gethash "connection" response-headers)))))
                      ;; Raise an error when the HTTP response status code is 4xx or 50x.
                      (when (<= 400 status)
                        (with-restarts
                          (http-request-failed status
                                               :body body
                                               :headers response-headers
                                               :uri uri
                                               :method method)))
                      (return-from request
                        (values body
                                status
                                response-headers
                                uri
                                (when (and keep-alive
                                           (not (equalp (gethash "connection" response-headers) "close")))
                                  stream))))
		 (close stream)  ;; added explicit stream here*********
                 (finalize-connection stream (gethash "connection" response-headers) uri))

error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

When uusing dexador on kickass.cd I get the following


CL-USER> (dex:get "https://kickass.cd/")
; Debugger entered on #<CL+SSL::SSL-ERROR-SSL {1001F53BD3}>

A failure in the SSL library occurred on handle #.(SB-SYS:INT-SAP #X0021EBB0) (return code: 1).
SSL error queue:
error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
   [Condition of type CL+SSL::SSL-ERROR-SSL]

Restarts:
 0: [RETRY-REQUEST] Retry the same request.
 1: [RETRY] Retry SLY mREPL evaluation request.
 2: [*ABORT] Return to SLY's top level.
 3: [ABORT] abort thread (#<THREAD "sly-channel-1-mrepl-remote-1" RUNNING {100434DD83}>)

Backtrace:
  0: (CL+SSL::SSL-SIGNAL-ERROR #.(SB-SYS:INT-SAP #X0021EBB0) #<FUNCTION CL+SSL::SSL-CONNECT> 1 -1)
  1: (CL+SSL:MAKE-SSL-CLIENT-STREAM #<unavailable argument> :CERTIFICATE #<unavailable argument> :KEY #<unavailable argument> :PASSWORD #<unavailable argument> :METHOD #<unavailable argument> :EXTERNAL-FOR..
  2: (CL+SSL::CALL-WITH-GLOBAL-CONTEXT #.(SB-SYS:INT-SAP #X00216260) T #<CLOSURE (LAMBDA NIL :IN DEXADOR.BACKEND.USOCKET:REQUEST) {1001E58B2B}>)
  3: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://kickass.cd/>)
  4: (DEXADOR.BACKEND.USOCKET:REQUEST #<unavailable argument> :METHOD :GET)
  5: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEXADOR:GET "https://kickass.cd/") #<NULL-LEXENV>)
  6: (EVAL (DEXADOR:GET "https://kickass.cd/"))
  7: ((LAMBDA NIL :IN SLYNK-MREPL::MREPL-EVAL-1))
  8: (SLYNK::CALL-WITH-RETRY-RESTART "Retry SLY mREPL evaluation request." #<CLOSURE (LAMBDA NIL :IN SLYNK-MREPL::MREPL-EVAL-1) {1001E4A9EB}>)
  9: ((LAMBDA NIL :IN SLYNK-MREPL::MREPL-EVAL-1))
 10: ((LAMBDA NIL :IN SLYNK::CALL-WITH-LISTENER))
 11: (SLYNK::CALL-WITH-BINDINGS ((*PACKAGE* . #<PACKAGE "COMMON-LISP-USER">) (*) (**) (*** . #(239 204 9)) (/ NIL) (// NIL) ...) #<CLOSURE (LAMBDA NIL :IN SLYNK::CALL-WITH-LISTENER) {1001E4A65B}>)
 12: (SLYNK-MREPL::MREPL-EVAL-1 #<SLYNK-MREPL::MREPL mrepl-1-1> "(dex:get \"https://kickass.cd/\")")
 13: (SLYNK-MREPL::MREPL-EVAL #<SLYNK-MREPL::MREPL mrepl-1-1> "(dex:get \"https://kickass.cd/\")")
 14: (SLYNK::PROCESS-REQUESTS NIL)
 15: ((LAMBDA NIL :IN SLYNK::SPAWN-CHANNEL-THREAD))
 16: ((LAMBDA NIL :IN SLYNK::SPAWN-CHANNEL-THREAD))
 17: (SLYNK-SBCL::CALL-WITH-BREAK-HOOK #<FUNCTION SLYNK:SLYNK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA NIL :IN SLYNK::SPAWN-CHANNEL-THREAD) {100435002B}>)
 18: ((FLET SLYNK-BACKEND:CALL-WITH-DEBUGGER-HOOK :IN "/Users/toni/.emacs.d/.cask/27.0/elpa/sly-20180117.533/slynk/backend/sbcl.lisp") #<FUNCTION SLYNK:SLYNK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA NIL :IN SLYNK:..
 19: ((LAMBDA NIL :IN SLYNK::CALL-WITH-LISTENER))
 20: (SLYNK::CALL-WITH-BINDINGS ((*PACKAGE* . #<PACKAGE "COMMON-LISP-USER">) (*) (**) (*** . #(239 204 9)) (/ NIL) (// NIL) ...) #<CLOSURE (LAMBDA NIL :IN SLYNK::CALL-WITH-LISTENER) {100435006B}>)
 21: ((LAMBDA NIL :IN SLYNK::SPAWN-CHANNEL-THREAD))
 22: ((FLET SB-UNIX::BODY :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
 23: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
 24: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
 25: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
 26: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE) {4A41D3B}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THREAD "..
 27: (SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE #<SB-THREAD:THREAD "sly-channel-1-mrepl-remote-1" RUNNING {100434DD83}> NIL #<CLOSURE (LAMBDA NIL :IN SLYNK::SPAWN-CHANNEL-THREAD) {100434D33B}> NIL)
 28: ("foreign function: call_into_lisp")
 29: ("foreign function: new_thread_trampoline")
 30: ("foreign function: _pthread_body")
 31: ("foreign function: _pthread_body")
 32: ("foreign function: thread_start")

I believe, but I'm not sure that the cl+ssl system is using this openssl in emacs PATH is the first /usr/local/bin

Welcome to the Emacs shell

~/learn/lisp/cl-university $ $PATH
/Users/toni/.rvm/gems/ruby-2.4.2/bin:/Users/toni/.rvm/gems/ruby-2.4.2@global/bin:/Users/toni/.rvm/rubies/ruby-2.4.2/bin/:/usr/texbin:/Library/TeX/texbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/toni/.rvm/bin: command not found
~/learn/lisp/cl-university $ openssl version
OpenSSL 1.1.0e  16 Feb 2017
~/learn/lisp/cl-university $ which openssl
/usr/local/bin/openssl

╭─ ~  2.4.2@learn  sbcl-bin 
╰─ openssl version                         1 ↵  14.52 Dur  10013  07:32:41
OpenSSL 1.1.0e  16 Feb 2017

and with curl it works, also inside emacs with restclient

╰─ curl -v https://kickass.cd/                           ✓  10014  07:32:56
*   Trying 104.24.104.224...
* TCP_NODELAY set
* Connected to kickass.cd (104.24.104.224) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate: sni57005.cloudflaressl.com
* Server certificate: COMODO ECC Domain Validation Secure Server CA 2
* Server certificate: COMODO ECC Certification Authority
> GET / HTTP/1.1
> Host: kickass.cd
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 31 Jan 2018 07:05:21 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=daafb96fd87e73e919922a781ec43fc6b1517382321; expires=Thu, 31-Jan-19 07:05:21 GMT; path=/; domain=.kickass.cd; HttpOnly
< X-Powered-By: PHP/5.6.32
< Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< Server: cloudflare
< CF-RAY: 3e5ad27669b13c7d-CDG
<
<!DOCTYPE html>
<html>
     . . .
     . . .
     . . .
</HtML>
* Connection #0 to host kickass.cd left intact

Request timeout leaves socket in CLOSE_WAIT state

If a request times out, the socket will not be closed and hang in CLOSE_WAIT state forever. Eventually, this will use up all ports.

Test case:
1. Set up server:

(defparameter *hunchentoot-server* (hunchentoot:start 
				  (make-instance 'hunchentoot:easy-acceptor :port 8080)))
(hunchentoot:define-easy-handler (say-yo :uri "/test") ()
           (setf (hunchentoot:content-type*) "text/plain")
           (sleep 5)
           (format nil "Hey!"))

2. Send request:

(dex:get "http://localhost:8080/test"
                  :use-connection-pool nil
                  :keep-alive nil
                  :connect-timeout 3
                  :read-timeout 3)

Behold now that netstat -tupan shows this:

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      13306/sbcl          
tcp      168      0 127.0.0.1:46202         127.0.0.1:8080          CLOSE_WAIT  13306/sbcl

And for any additional request like in step 2 another entry will appear:

tcp        0      0 127.0.0.1:8080          127.0.0.1:46268         FIN_WAIT2   -                   
tcp      168      0 127.0.0.1:46202         127.0.0.1:8080          CLOSE_WAIT  13306/sbcl          
tcp      168      0 127.0.0.1:46264         127.0.0.1:8080          CLOSE_WAIT  13306/sbcl          
tcp      168      0 127.0.0.1:46268         127.0.0.1:8080          CLOSE_WAIT  13306/sbcl

HTTP/2 support

Right now if you try fetching a resource that is using HTTP/2 you'll get DEXADOR.ERROR:HTTP-REQUEST-NOT-FOUND error.
Sample call to reproduce (using dexador from quicklisp):

(dexador:get "https://www.theglobeandmail.com/life/health-and-fitness/health/number-of-us-adhd-diagnoses-astronomical/article10606200/?cmpid=rss1")

RIght now only cl-http2 related project I'm seeing on github is https://github.com/akamai/cl-http2-protocol. 🤔

The function quri.parser::parse-scheme-string is undefined.

Hi,
I suddenly have this error on a simple dex:get.

The function quri.parser::parse-scheme-string is undefined.

What's going on ? I don't find where this function appears. I'm on Quicklisp 20180328. Looking at dexador in quicklisp's directory, an rgrep doesn't find parse-scheme-string.

I think this error appeared when I tried to upgrade QL but finally downgraded to this version (due to a Slime error).

Lack of "Transfer-Encoding" and "Content-Length" making an empty body

In usocket.lisp, in the definition of read-response, a response without both "Content-Length" and "Transfer-Encoding" is being considered as an empty message.

However, this does not follow the specification of HTTP/1.1, as the last rule says that:

Otherwise, this is a response message without a declared message
body length, so the message body length is determined by the
number of octets received prior to the server closing the
connection.

(see also previous rules in order to see which are the cases)

The current code with this problem is below

(cond
 ...
 ((or (not transfer-encoding-p)
      (let ((status (http-status http)))
        (or (= status 100)    ;; Continue
            (= status 101)    ;; Switching Protocols
            (= status 204)    ;; No Content
            (= status 304)))) ;; Not Modified
  (setq body +empty-body+))
 ...)

Is this cross implementation ?

Hi,
I'm doing advertising for dexador and I'm being asked: does dexador work on ABCL, CLISP, SBCL, ECL (or a lisp in that family), and CCL? I can't find that information.

Thanks ! cheers

Options :use-connection-pool and :keep-alive make request non thread-safe

Here is the traceback I'm receiving when trying to use dex:get from multiple threads:

Corrupt NEXT-chain in #<HASH-TABLE :TEST EQ :COUNT 22 {10020CF1F3}>. This is probably caused by multiple threads accessing the same hash-table without locking.
[Condition of type SIMPLE-ERROR]

Restarts:
0: [TRANSFER-ERROR] Transfer this error to a dependent thread, if one exists.
1: [KILL-ERRORS] Kill errors in workers (remove debugger instances).
2: [ABORT] abort thread (#<THREAD "lparallel" RUNNING {10031CBC33}>)

Backtrace:
0: (SB-IMPL::SIGNAL-CORRUPT-HASH-TABLE-BUCKET #<HASH-TABLE :TEST EQ :COUNT 22 {10020CF1F3}>)
1: (SB-IMPL::GETHASH/EQ # <unavailable argument> # <unavailable argument> # <unavailable argument> )
2: (DEXADOR.CONNECTION-CACHE::GET-CONNECTION-POOL)
3: (DEXADOR.CONNECTION-CACHE:STEAL-CONNECTION "https://www.reddit.com")
4: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :MAX-REDIRECTS 4 :HEADERS ((:HOST . "www.reddit.com")) :METHOD :GET :USE-CONNECTION-POOL T)
5: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :MAX-REDIRECTS 4 :HEADERS ((:HOST . "www.reddit.com")) :METHOD :GET :USE-CONNECTION-POOL T)
6: (CL-REDDIT::GET-JSON "http://www.reddit.com/comments/gxb7j1.json?comment=ft0giiu" # <unused argument> )
7: (CL-REDDIT:GET-COMMENTS "gxb7j1" #<CL-REDDIT:USER {1005081B83}> :ARTICLE NIL :COMMENT "ft0giiu" :CONTEXT NIL :DEPTH NIL :LIMIT NIL :SORT NIL :THREADED NIL :SHOWMORE NIL)
8: ((LAMBDA NIL :IN LPARALLEL.COGNATE::PMAP-INTO/POWDER/LIST))
9: ((FLET "BODY-FN0" :IN LPARALLEL.KERNEL::MAKE-CHANNELED-TASK))
10: (LPARALLEL.KERNEL::EXEC-TASK/WORKER # <unavailable argument> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL..
11: (LPARALLEL.KERNEL::WORKER-LOOP #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 32 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {10031CA583}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #..
12: (LPARALLEL.KERNEL::%CALL-WITH-TASK-HANDLER # <unavailable argument> )
13: ((LAMBDA NIL :IN LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT))
14: (LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT #<CLOSURE (LAMBDA NIL :IN LPARALLEL.KERNEL::ENTER-WORKER-LOOP) {100320835B}> # <FUNCTION FUNCALL> #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 3..
15: ((LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-WORKER-THREAD))
16: ((LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS))

Tested under SBCL 2.0.2

Workaround for SBCL

I do this at the start of the program:

(setf cl-dbi::*threads-connection-pool*
        (make-hash-table :test 'equal :synchronized t))

WANT-STREAM unsafely returns stream to connection pool

When want-stream and use-connection-pool are both true, the stream is returned to the pool before the returned stream has been completely read from. This can result in an error if a second request to the same server is sent before the caller has completely read the stream.

This is definitely a corner case, but, I think, still within the realm of something that's reasonable to want. As an example, this function signals an error that the stream is closed before it finishes reading it.

(defun uh-oh ()
  (let ((stream-1 (dex:get "http://example.com" :want-stream t))
        (stream-2 (dex:get "http://example.com" :want-stream t)))
    (loop
      :for line := (read-line stream-1 nil :eof)
      :until (eql line :eof)
      :do (print line))))

A more realistic example would be a multi-threaded program that performs HTTP requests on one thread and passes the streams to another thread for processing.

I'm handling this in my own connection pool on top of Dexador by requiring that the user close the returned stream when they're done with it. The close method then reads from the underlying stream to its EOF before returning it to the pool. Happy to share my code for that once I bang on it a bit more (most likely next weekend).

Headers with NIL values are still written

In the documentation it says that headers with a NIL value are not written, however this does not seem the case. The loop that writes the custom headers does not contain a NIL check whatsoever, so this seems to be the culprit.

get: special chars in url string are not converted, results in 400 bad request

A call like
(dexador:get "https://www.last.fm/music/Mötley+Crüe")

results in

An HTTP request to "https://www.last.fm/music/Mötley+Crüe" returned 400 bad request.
[Condition of type HTTP-REQUEST-BAD-REQUEST]

I'm assuming the special chars ö and ü, in this case, are not converted so that the call is not converted automatically to
(dexador:get "https://www.last.fm/music/M%C3%B6tley+Cr%C3%BCe")

which would be a call to the correct page.

curl seems to be doing this, as well as the browser before actually sending the request.

I'm not sure if this should be something dexador should handle or if it should be something the user needs to take care of. If the second, any nice, clean suggestions for how the user should handle this conversion?

SSL verify error: 20

on travis ci environment I couldn't access https://api.github.com using dexador.
does anyone know how to fix it?
detail https://github.com/snmsts/experiment/tree/dexador201701

winhttp issues on Windows

I had Dexador from 2019-02-02 installed from Quicklisp and it served me well. Today I upgraded to the new version, which apparently uses winhttp instead of cl+ssl (which I had zero problems installing on Windows).

The first thing I noticed is that timeouts are unreasonably short. After bumping *default-connect-timeout* and dexador:*default-read-timeout* to 10000, I was actually able to do something, but with the default value of 10 I've been getting winhttp timeout on any non-trivial POST requests. Could these values be in milliseconds instead of seconds as opposed to cl+ssl?

Also the very first example (dex:get "http://lisp.org/") doesn't work for me, resulting in

ERROR 12175: Secure failure
   [Condition of type WINHTTP::WIN-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1004FB9D73}>)

Backtrace:
  0: (WINHTTP::GET-LAST-ERROR)
  1: (WINHTTP:SEND-REQUEST #.(SB-SYS:INT-SAP #X009146C0) #() :START 0 :END NIL)
  2: (DEXADOR.BACKEND.WINHTTP:REQUEST #<QURI.URI.HTTP:URI-HTTPS https://lisp.org/> :MAX-REDIRECTS 4 :METHOD :GET :METHOD :GET)
  3: (DEXADOR.BACKEND.WINHTTP:REQUEST "http://lisp.org/" :METHOD :GET)
  4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEXADOR:GET "http://lisp.org/") #<NULL-LEXENV>)
  5: (EVAL (DEXADOR:GET "http://lisp.org/"))

There might be something with this specific site's certificate but I'm not sure what. It uses Let's Encrypt just like my own site http;//ichi.moe which does work.

Overall I wish there was a way to continue using CL+SSL even on Windows if it's installed. I couldn't find such option in the code since using winhttp seems to depend on :windows feature.

Add `force-string` option

There are many cases where one might prefer to return a string even if the Content-Type isn't text/*

It's trivial to add a conversion afterwards so it's not a huge issue, but if force-binary exists it makes sense for force-string to exist too.

"SSL initialization error: Can't load certificate" when using .pem and .key certs

I have a certificate and a key file generated by a vendor with whom I'm integrating:

client-2048.pem

-----BEGIN CERTIFICATE-----
Neighoofeib8edahv8ieK3eePai0wequohm6vahm7kao4aeTeo3aephie8laiW3
...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
Ath3SeifooDairaithovohno4ahdeeth4eiJeeDoogoo9Ahjo0aiBe1OeWei9Ei
...
-----END RSA PRIVATE KEY-----

client-2048.key

-----BEGIN RSA PRIVATE KEY-----
Ath3SeifooDairaithovohno4ahdeeth4eiJeeDoogoo9Ahjo0aiBe1OeWei9Ei
...
-----END RSA PRIVATE KEY-----

If I attempt to use these with dexador, I get an unhandled condition:

CL-USER> (dexador:request "https://example.com/api/certlogin"
                    :method :post
                    :content `(("username" . "REDACTED") ("password" . "REDACTED"))
                    :ssl-cert-file "~/path/to/cert/client-2048.pem"
                    :ssl-key-file "~/path/to/cert/client-2048.key")

SSL initialization error: Can't load certificate ~/path/to/cart/client-2048.pem
SSL error queue is empty.
   [Condition of type CL+SSL:SSL-ERROR-INITIALIZE]

Restarts:
 0: [RETRY-REQUEST] Retry the same request.
 1: [RETRY] Retry SLIME REPL evaluation request.
 2: [*ABORT] Return to SLIME's top level.
 3: [ABORT] abort thread (#<THREAD "new-repl-thread" RUNNING {10038B8E03}>)

Backtrace:
  0: (CL+SSL::INSTALL-KEY-AND-CERT #<unavailable argument> #<unavailable argument> #<unavailable argument>)
  1: (CL+SSL:MAKE-SSL-CLIENT-STREAM #<unavailable argument> :CERTIFICATE #<unavailable argument> :KEY #<unavailable argument> :PASSWORD #<unavailable argument> :METHOD #<unavailable argument> :EXTERNAL-FOR..
  2: (CL+SSL::CALL-WITH-GLOBAL-CONTEXT #.(SB-SYS:INT-SAP #X7FC114002450) T #<CLOSURE (LAMBDA NIL :IN DEXADOR.BACKEND.USOCKET:REQUEST) {1003F9F6AB}>)
  3: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://example.com/api/certlogin>)
  4: (DEXADOR.BACKEND.USOCKET:REQUEST #<unavailable argument> :METHOD :POST :CONTENT (("username" . "REDACTED") ("password" . "REDACTED")) :SSL-CERT-FILE "~/path/to/cart/client-2048...
  5: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEXADOR.BACKEND.USOCKET:REQUEST *BETFAIR-LOGIN-URI* :METHOD :POST :CONTENT (SB-INT:QUASIQUOTE (# #)) ...) #<NULL-LEXENV>)
  6: (EVAL (DEXADOR.BACKEND.USOCKET:REQUEST *BETFAIR-LOGIN-URI* :METHOD :POST :CONTENT (SB-INT:QUASIQUOTE (# #)) ...))
 --more--

For what it's worth, the certificate and keyfile work when used with Ruby, on the same machine:

pem_filename = '~/path/to/cert/client-2048.pem'
key_filename = '~/path/to/cert/client-2048.key'

ssl = {
  client_cert: OpenSSL::X509::Certificate.new(File.read(pem_filename)),
  client_key: OpenSSL::PKey::RSA.new(File.read(key_filename)),
  verify: false
}

@client = Faraday.new(:url => 'https://example.com/api/certlogin', :ssl => ssl) do |faraday|
  # snip ...
end

Versions of things:

Ubuntu 19.10 (Eoan Ermine)
SBCL 1.5.7
Quicklisp 2019-10-08
OpenSSL 1.1.1c 28 May 2019

new version broke dexador on ubuntu 18.04

Service

(defhandler (app "/" :method :get)
    "quack quack i'm a server.")

Curl

curl 127.0.0.1:9001
--> 
{"message": "quack quack i'm a server."}

Drakma

(drakma:http-request "http://127.0.0.1:9001/" :method :get)
-->
#(123 34 109 101 115 115 97 103 101 34 58 32 34 113 117 97 99 107 32 113 117 97
  99 107 32 105 39 109 32 97 32 115 101 114 118 101 114 46 34 125)
200
((:DATE . "Wed, 18 Sep 2019 08:01:14 GMT") (:CONTENT-TYPE . "application/json")
 (:TRANSFER-ENCODING . "chunked"))
#<PURI:URI http://127.0.0.1:9001/>
#<FLEXI-STREAMS:FLEXI-IO-STREAM {100679D4C3}>
T
"OK"

Dexador

(dex:get "http://127.0.0.1:9001/")
--> 
An HTTP request to "http://127.0.0.1:9001/" returned 400 bad request.

#(66 97 100 32 82 101 113 117 101 115 116)
   [Condition of type DEXADOR.ERROR:HTTP-REQUEST-BAD-REQUEST]

Restarts:
 0: [RETRY-REQUEST] Retry the same request.
 1: [IGNORE-AND-CONTINUE] Ignore the error and continue.
 2: [RETRY] Retry SLIME REPL evaluation request.
 3: [*ABORT] Return to SLIME's top level.
 4: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1004831BC3}>)

Backtrace:
  0: (DEXADOR.ERROR:HTTP-REQUEST-FAILED 400 :BODY #(66 97 100 32 82 101 ...) :HEADERS #<HASH-TABLE :TEST EQUAL :COUNT 3 {1007CA4563}> :URI #<QURI.URI.HTTP:URI-HTTP http://127.0.0.1:9001/> :METHOD :GET)
  1: (DEXADOR.BACKEND.USOCKET:REQUEST #<unavailable argument> :METHOD :GET)
  2: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEXADOR:GET "http://127.0.0.1:9001/") #<NULL-LEXENV>)
  3: (EVAL (DEXADOR:GET "http://127.0.0.1:9001/"))
  4: (SWANK::EVAL-REGION "(dex:get \"http://127.0.0.1:9001/\") ..)
  5: ((LAMBDA NIL :IN SWANK-REPL::REPL-EVAL))
  6: (SWANK-REPL::TRACK-PACKAGE #<CLOSURE (LAMBDA NIL :IN SWANK-REPL::REPL-EVAL) {1007CA264B}>)
  7: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation request." #<CLOSURE (LAMBDA NIL :IN SWANK-REPL::REPL-EVAL) {1007CA25EB}>)
  8: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA NIL :IN SWANK-REPL::REPL-EVAL) {1007CA25CB}>)
  9: (SWANK-REPL::REPL-EVAL "(dex:get \"http://127.0.0.1:9001/\") ..)
 10: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK-REPL:LISTENER-EVAL "(dex:get \"http://127.0.0.1:9001/\") ..)
 11: (EVAL (SWANK-REPL:LISTENER-EVAL "(dex:get \"http://127.0.0.1:9001/\") ..)
 12: (SWANK:EVAL-FOR-EMACS (SWANK-REPL:LISTENER-EVAL "(dex:get \"http://127.0.0.1:9001/\") ..)
 13: (SWANK::PROCESS-REQUESTS NIL)
 14: ((LAMBDA NIL :IN SWANK::HANDLE-REQUESTS))
 15: ((LAMBDA NIL :IN SWANK::HANDLE-REQUESTS))
 16: (SWANK/SBCL::CALL-WITH-BREAK-HOOK #<FUNCTION SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA NIL :IN SWANK::HANDLE-REQUESTS) {10048400FB}>)
 17: ((FLET SWANK/BACKEND:CALL-WITH-DEBUGGER-HOOK :IN "/home/jacknchou/.roswell/lisp/slime/2019.08.13/swank/sbcl.lisp") #<FUNCTION SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA NIL :IN SWANK::HANDLE-REQUEST..
 18: (SWANK::CALL-WITH-BINDINGS ((*STANDARD-INPUT* . #<SWANK/GRAY::SLIME-INPUT-STREAM {10047104D3}>)) #<CLOSURE (LAMBDA NIL :IN SWANK::HANDLE-REQUESTS) {100484011B}>)
 19: (SWANK::HANDLE-REQUESTS #<SWANK::MULTITHREADED-CONNECTION {1003D5C763}> NIL)
 20: ((FLET SB-UNIX::BODY :IN SB-THREAD::NEW-LISP-THREAD-TRAMPOLINE))
 21: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::NEW-LISP-THREAD-TRAMPOLINE))
 22: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::NEW-LISP-THREAD-TRAMPOLINE))
 23: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
 24: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::NEW-LISP-THREAD-TRAMPOLINE) {7FA848B96D7B}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THREAD "rep..
 25: (SB-THREAD::NEW-LISP-THREAD-TRAMPOLINE #<SB-THREAD:THREAD "repl-thread" RUNNING {1004831BC3}> NIL #<CLOSURE (LAMBDA NIL :IN SWANK-REPL::SPAWN-REPL-THREAD) {1004831B6B}> NIL)
 26: ("foreign function: call_into_lisp")
 27: ("foreign function: new_thread_trampoline")

Missing support for not-verify-ssl (self-signed certificates)

Attempting to request data from a server with a self-signed certificate. In the default configuration, this fails. I want to skip/ignore the certificate verification process.

Attempted to set the not-verify-ssl flag but, so far, unsuccessful.

Any workaround ?

USOCKET:NS-HOST-NOT-FOUND-ERROR

Please catch this condition so I can use dex:ignore-and-continue. This occurs when a 301 redirect points to a URL that cannot be resolved/connected to. This may be a desired 301 redirect so we should handle this in dex. Thank you

(dex:get "http://doesnotexist.google.com")

=>

Condition USOCKET:NS-HOST-NOT-FOUND-ERROR was signalled.
   [Condition of type USOCKET:NS-HOST-NOT-FOUND-ERROR]

stream return (:want-stream t) timing out on retrieval

Although retrieving through the stream works extremely well, it consistently fails at about the same place each time (in this case, at 90ish of 150K triples). Drakma, on the other hand, doesn't have the issue. Is there something wrong with how I've configured the request? (see commented out section below)

(defun sparql-stream->db (query &key db)
"The query must return ?s ?p ?o triples for the streaming json parse"
;(handler-case
(let ((db (if db db
(make-solid-db (make-temp-graph-namespace))))
)
#|
;;can only get about 90 triples until failure with dex
(multiple-value-bind (result http-status response-hash uri stream)
(dex:post (string+ (get-server) "/blazegraph/sparql")
:content (list ("query".,query)) :headers (list ("accept". ,(getf (config :sparql-output) :json)))
:want-stream t
:keep-alive t
:use-connection-pool t
)
(print (list result http-status (alexandria:hash-table-alist response-hash) uri stream))
|#

(multiple-value-bind (stream status-code headers uri http-stream must-close status-text)
(drakma:http-request
 (string+ (get-server) "/blazegraph/sparql")
 :method :post
 ;:content-type "text/plain"
 :accept (getf (config :sparql-output) :json)
 :want-stream t
 :parameters (list `("query".,query)))
  (print (list stream status-code headers uri http-stream must-close status-text))
 
  (with-open-stream (stm stream)
  (with-open-stream (s (flexi-streams:make-flexi-stream stm :external-format :utf-8))
;(read-line s)
(json-streams:with-open-json-stream (js (json-streams:make-json-input-stream  s))
  (loop for item = (json-streams:json-read js)
     until (or (eql item :eof)
	       (string= item "bindings")))
  (json-streams:json-read js);;advancing to triples section
  (loop for j-triple = (json-streams::parse-single js)
     until (or (eql j-triple :end-array)
	       (eql j-triple :eof))
     do (json-stream-triple->db db j-triple)
     )))))
  db))

Add read-timeout to dex:get

Right now it is possible to DOS a service which uses dexador to access external resources, if these resources are extremely slow or just accept TCP connection but dont respond.

Right now, I've added this workaround into my application code:

(ql:quickload 'trivial-timeout)

(with-timeout (read-timeout)
      (dex:get url :timeout connect-timeout))

read-byte blocks when use :want-stream

actually it will return after a long time
(dex:get "http://lisp.org" :want-stream t)
maybe the problem is from:
(loop while (read-byte body nil nil))
and this code works fine:
(loop while (listen body) do (read-byte body nil nil))

Stream response would be blocked when read-sequence

Stream response that requests with :want-stream t return would be blocked when read-sequence. It because the other peer doesn't send EOF as the stream is KeepAlive.

It has fixed at a branch end-of-decoding-stream when Content-Type header exists, however it still occurs when the response is chunked.

To fix it, I suppose CHUNGA has to be replaced by something else.

can't save a cookie.

Now I try to make an ask.fm client ezoe.

To ask someone, we needs an ask.fm account.

Following code login to ask.fm.

I expect dexador that

first, dex:get access to "ask.fm/login"
get authenticity_token and _ask.fm_session cookie. from body
and keep a connection.
next, dex:post access to "ask.fm/session" with authenticity_token's value (from '2'), username , password and _ask.fm_session cookie.
dex:post save auth_token="foobarhogefuga..." to cookie.

but, dex:get couldn't keep _ask.fm_session's cookie and save auth_token to cookie.

In Firefox, at first, auth_token is initialize with "".
next set a value (like "foobarhogefuga...").
dexador is only initialize..

dexador saticefies my expectation?

thanks.

(ql:quickload
 '(:dexador
   :cl-cookie
   :cl-html-parse
   :get-element-by
   :quri))

(defun login (user password)
  (let* ((url "http://ask.fm")
     (cookie (cl-cookie:make-cookie-jar))
     (header '(("User-Agent" . "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0")))
     (body (dex:get url :cookie-jar cookie
                :headers header
                :keep-alive t
                :verbose t))
     (tree (html-parse:parse-html body))
     (form (geby:get-element-by
        :name "authenticity_token" tree))
     (token (quri:url-encode (getf (cdar form) :value)))
     (params`(("authenticity_token" . ,token)
          ("login"              . ,user)
          ("password"           . ,password)
          ("commit"             . "ログイン"))))
    (print cookie)
    (dex:post "http://ask.fm/session"
          :cookie-jar cookie
          :headers header
          :content params
          :keep-alive t
          :verbose t)
    (print cookie)))

Random memory faults

For some reason, on certain sites, when using SSL, SBCL complains about a memory fault and Dexador fails to retrieve the page.
Command used: (dex:get "https://google.com/" :verbose t)
Result: CORRUPTION WARNING in SBCL pid 5061(tid 0x7ffff7fadfc0):
Memory fault at (nil) (pc=0x40fb62, sp=0x7ffff6cfeef8)
The integrity of this image is possibly compromised.
Continuing with fingers crossed.
Using SBCL with Arch Linux.

Using dexador:retry-request can cause "Control stack exhausted" error

Here is approach I've used to retry on some network errors:

(handler-bind
    ((usocket:ns-host-not-found-error
      (lambda (c)
        (declare (ignorable c))
        (invoke-restart
         ;; STACK WILL END HERE BECAUSE OF RECURSION
         (find-restart 'dexador:retry-request)))))
  (dexador:get url
               :read-timeout 10
               :connect-timeout 10))

And here is the stacktrace I've got:


Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.

PROCEED WITH CAUTION.
[Condition of type SB-KERNEL::CONTROL-STACK-EXHAUSTED]

Restarts:
0: [ABORT] abort thread (#<THREAD "Candles Updater" RUNNING {10099FD7A3}>)

Backtrace:
0: (SB-KERNEL::CONTROL-STACK-EXHAUSTED-ERROR)
1: ("foreign function: call_into_lisp")
2: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> #<unavailable &REST argument>)
3: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
4: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
5: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
6: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
7: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
8: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
9: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
10: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
11: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
12: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
13: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
14: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
15: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
16: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
17: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
18: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
19: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
20: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
21: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
22: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
23: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
24: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
25: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$
26: ((LABELS DEXADOR.BACKEND.USOCKET::MAKE-NEW-CONNECTION :IN DEXADOR.BACKEND.USOCKET:REQUEST) #<QURI.URI.HTTP:URI-HTTPS https://api-invest.tinkoff.ru/openapi/marke$
27: (DEXADOR.BACKEND.USOCKET:REQUEST # <unavailable argument> :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :USE-CONNECTION-POOL NIL :US$2
...

What is the best way to process network issues when using dexador?

Probably, this recursive call should be replaced with tagbody/go?

The bounding indices 851 and 0 are bad for a sequence of length 0.

Save this text into some file

HTTP/1.1 204 No Content^M
Server: GitHub.com^M
Date: Fri, 31 Jul 2015 05:08:01 GMT^M
Status: 204 No Content^M
X-RateLimit-Limit: 5000^M
X-RateLimit-Remaining: 4968^M
X-RateLimit-Reset: 1438321051^M
X-OAuth-Scopes: gist^M
X-Accepted-OAuth-Scopes: ^M
X-GitHub-Media-Type: github.v3^M
X-XSS-Protection: 1; mode=block^M
X-Frame-Options: deny^M
Content-Security-Policy: default-src 'none'^M
Access-Control-Allow-Credentials: true^M
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval^M
Access-Control-Allow-Origin: *^M
X-GitHub-Request-Id: 6AA79DDD:44D3:F45611:55BB02B1^M
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload^M
X-Content-Type-Options: nosniff^M
Vary: Accept-Encoding^M
X-Served-By: a241e1a8264a6ace03db946c85b92db3^M
^M

and %s/\^M/\r/g, then

(let ((stream (open "path-to-file" :element-type '(unsigned-byte 8))))
   (dexador.backend.usocket::read-response stream t nil t))

will raise the error
The bounding indices 851 and 0 are bad for a sequence of length 0.
.
Actually, the string above is the response returned by GitHub API.

error: Evaluation aborted on #<TYPE-ERROR expected-type: SINGLE-FLOAT datum: 1.1d0>.

Hi, I can't send HTTP requests using dexador, and I get a type error whether I use dex: get or dex: post.

(dex:get "http://lisp.org/")
; Evaluation aborted on #<TYPE-ERROR expected-type: SINGLE-FLOAT datum: 1.1d0>.

The value
  1.1d0
is not of type
  SINGLE-FLOAT
when binding DEXADOR.BACKEND.USOCKET::VERSION
   [Condition of type TYPE-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1001F41B93}>)

Backtrace:
  0: (DEXADOR.BACKEND.USOCKET:REQUEST #<unavailable argument> :METHOD :GET)
  1: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEXADOR:GET "http://lisp.org/") #<NULL-LEXENV>)
  2: (EVAL (DEXADOR:GET "http://lisp.org/"))
 --more--

Env: macOS 10.14.2 + SBCL 1.4.13

Proxy support

SSL issue

(dexador:get "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json")

Fails with

A failure in the SSL library occurred on handle #.(SB-SYS:INT-SAP #X05C009B0) (return code: 1). SSL error queue: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure [Condition of type CL+SSL::SSL-ERROR-SSL]

Read-timeout does not work on SBCL and ClozureCL

Here is the code to reproduce:

CL-USER> (time (nth-value 1 (dex:get "https://httpbin.org/delay/10" :read-timeout 2)))
Evaluation took:
  10.553 seconds of real time
  0.056000 seconds of total run time (0.052000 user, 0.004000 system)

Url https://httpbin.org/delay/10 respond in 10 seconds, but I expect dex:get signal error after the 2 seconds.

If network flaps, this leads to situations where requests are hanging forever.

Argument :read-timeout was added in this pull https://github.com/fukamachi/dexador/pull/71/files. It calls (usocket:socket-option connection :receive-timeout) method for a new socket, which has implementation for SBCL and ClozureCL, the lisps I'm tested this option:

https://github.com/usocket/usocket/blob/820e760abae25471d0f90d50e8e306d31a132852/option.lisp#L42

I was able to reproduce this bug on:

SBCL 2.0.5 Linux
SBCL 2.0.8 OSX
ClozureCL 1.12-dev (v1.12-dev.5) OSX

dex:post returns an invalid result after a dex:head call

I have a problem when trying to connect to a server requiring digest authentication.

I first try to connect with a HEAD request to get the authentication challenge in the server response headers, then I compute the authentication data and make a POST request.

The code is something like this:

(let ((auth (handler-case (progn (dex:head server-uri) nil)
              (dex:http-request-unauthorized (e)
                (compute-digest-authentication-response ...))
              (t () nil))))
  (dex:post server-uri
            :headers (if auth
                         (list (cons "authorization" auth)
                               (cons "content-type" "application/json"))
                         (list (cons "content-type" "application/json")))
            :content json-request))

The call to (dex:post ...) throws a dex:http-request-unauthorized condition (status 401), but it shouldn't, as analysing the TCP stream with Wireshark shows that the POST request is valid and the server accepts it and returns an answer (status 200).

I looks as if the (dex:post ...) call used the server response to the previous (dex:head ...) call instead of the new server response...

Workaround: I found that if I replace the call to (dex:head ...) by a call to (dex:get ...) or (dex:post ...), everything works fine (the "HTTP/1.1 200 Ok" response is seen correctly).

Octets vector content

I want to post an octets vector as content.

Drakma lets me do that, it is straightforward:

    (drakma:http-request (render-uri* url)
                             :method :post
                             :content content
                             :additional-headers
                             (imbo-auth-headers :post (render-uri* url) imbo-user))

where content is an octets vector.

But Dexador does not support that (signals an error):

(dex:post (render-uri* url)
                  :content content
                  :headers (imbo-auth-headers :post (render-uri* url) imbo-user))

=>

fell through ETYPECASE expression.
Wanted one of (NULL STRING PATHNAME).

Can that be added?

dexador and network unreachable

Hi,

I don't understand why dexador takes so long to detect that the network is unreachable.

I turn off the network and try this:

(time (handler-case (dex:get "https://www.google.com")
         (error (e) (warn "~s" e))))

and I get this (at the first time, after it works quickly):

WARNING: #<USOCKET:NS-HOST-NOT-FOUND-ERROR {1004689883}>
Evaluation took:
1080.867 seconds of real time
[blalba]

18 minutes! it's tooooo long, no ?

did i miss something ?

Backtrace:

  0: (USOCKET::HANDLE-CONDITION #<SB-BSD-SOCKETS:HOST-NOT-FOUND-ERROR {10056916A3}> NIL "www.google.com")
  1: (SB-KERNEL::%SIGNAL #<SB-BSD-SOCKETS:HOST-NOT-FOUND-ERROR {10056916A3}>)
  2: (ERROR SB-BSD-SOCKETS:HOST-NOT-FOUND-ERROR :ERROR-CODE -2 :SYSCALL "getaddrinfo")
  3: (SB-BSD-SOCKETS::ADDRINFO-ERROR "getaddrinfo" -2)
  4: (SB-BSD-SOCKETS:GET-HOST-BY-NAME #<unavailable argument>)
  5: (USOCKET:GET-HOSTS-BY-NAME "www.google.com")
  6: (USOCKET:SOCKET-CONNECT "www.google.com" 443 :PROTOCOL :STREAM :ELEMENT-TYPE (UNSIGNED-BYTE 8) :TIMEOUT 10 :DEADLINE NIL :NODELAY T :LOCAL-HOST NIL :LOCAL-PORT NIL)

SSL verification is not thread safe

On this line (https://github.com/fukamachi/dexador/blob/master/src/backend/usocket.lisp#L408), dexador makes:

(setf (cl+ssl:ssl-check-verify-p) (not insecure))

But if my program is accessing different resources with different :insecure options, then threads can interfere to each other by changing global variable cl+ssl::*ssl-check-verify-p*.

More over, this setf interfere with other libraries, who use cl+ssl. For example, I want to (dex:get "https://some-self-signed.com/resource" :insecure t) and then put data to a postgres with the Postmodern (which uses cl+ssl under the hood to establish SSL connections to the database) – FAIL.

Probably, it will be better idea to rebind this variable with let?

PARAMETERS keyword argument alternative

What is the proper way to replace PARAMETERS keyword argument of DRAKMA:HTTP-REQUEST when I use GET method?

Content-Length Header is not Sent on Empty POST Requests

Some servers refuse requests that do not include a Content-Length header. Dexador does not seem to send this header along with the request body, so some sites cannot be queried successfully.

More examples for dex:post are required

As a newbee,I have to face many difficulties,It maybe easier for me to follow the examples.When I want to try a web-automatic,like to use dex:post to login in a website,and submit text or click the button automatically ， I don't know how to achieve it。

Unable to POST json content because content-type is not set to application/json

I need to send json content to a server, but it seems that the content-type header does get filtered out in the usocket.lisp request function (see custom headers), and it is not set to application/json as it should be. other headers like Accept etc. do get set properly.

Uploading of Files with Unicode in the Filename Fails

This seems like the same issue as edicl/drakma#49 except that it doesn't error ahead of time and instead just produces a garbage request.