→ ↺

HTTP2 in Common Lisp

Table of Contents

[in package HTTP2]

← ↑ → ↺

1 Overview

This is an HTTP/2 implementation in Common Lisp. It provides both high-level interface as well as components to build an optimized tools for specific use cases.

For quick start, quickload HTTP2 and see Tutorials that show how to use a simple client to fetch a resource or how to start the server and serve some content.

For more documentation consult API documentation.

← ↑ → ↺

2 Tutorials

← ↑ → ↺

2.1 Client tutorials

[in package HTTP2/CLIENT]

The client tutorials show how to fetch a web resource using built-in Drakma-style interface, and how to make more advanced client that does several requests in parallel using HTTP/2 streams.

← ↑ → ↺

2.1.1 Using built-in HTTP/2 client

You can use RETRIEVE-URL to fetch a a web resource.

(http2/client:retrieve-url "https://example.com")
==> "<!doctype html>
... <html>
... <head>
...     <title>Example Domain</title>
...
...     <meta charset="utf-8" />
...     <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
...     <meta name="viewport" conten...[sly-elided string of length 1256]"
==> 200 (8 bits, #xC8, #o310, #b11001000)
==> (("content-length" . "1256") ("x-cache" . "HIT") ("vary" . "Accept-Encoding")
...  ("server" . "ECS (bsb/27E0)")
...  ("last-modified" . "Thu, 17 Oct 2019 07:18:26 GMT")
...  ("expires" . "Thu, 28 Sep 2023 19:38:44 GMT")
...  ("etag" . "\"3147526947+ident\"") ("date" . "Thu, 21 Sep 2023 19:38:44 GMT")
...  ("content-type" . "text/html; charset=UTF-8")
...  ("cache-control" . "max-age=604800") ("age" . "151654"))
==> "/"
==> #<VANILLA-CLIENT-CONNECTION >
==> NIL
==> "HTTP2 does not provide reason phrases"

It was designed to be similar to DRAKMA:HTTP-REQUEST, but is not completely same. See below what the function does, some notable differences are:

Basically, the differences fall into two areas, that it does not (yet) provide all the features of Drakma, and HTTP/2 is different.

← ↑ → ↺

2.1.2 Build your own client

Let us see what it takes to build simplified RETRIEVE-URL function from components. It will use CL+SSL to build a Lisp stream over TLS stream over network stream.

HTTP/2 requests are done over TLS connection created with an ALPN indication that it is to be used for HTTP/2. The helper function here is CONNECT-TO-TLS-SERVER, and then WITH-OPEN-STREAM can be used:

  (defun my-retrieve-url (url)
    (let ((parsed-url (puri:parse-uri url)))
      (with-open-stream (network-stream
                         (connect-to-tls-server (puri:uri-host parsed-url)
                                                :sni (puri:uri-host parsed-url)
                                                :port (or (puri:uri-port parsed-url) 443)))
        (my-retrieve-url-using-network-stream network-stream url))))

Now that we have a Lisp STREAM to communicate over, we can establish HTTP/2 connection of class VANILLA-CLIENT-CONNECTION over it, send client request, and then PROCESS-PENDING-FRAMES until server fully sends the response. That invokes restart FINISH-STREAM with the processed stream that we handle. We can get the data from it using DRAKMA-STYLE-STREAM-VALUES.

  (defun my-retrieve-url-using-network-stream (lisp-stream url)
    (with-http2-connection (connection 'vanilla-client-connection lisp-stream)
      (my-send-client-request connection url)
      (restart-case
          (process-pending-frames connection nil)
        (finish-stream (stream)
          (drakma-style-stream-values stream)))))

Sending the request involves creating a new HTTP2 stream with OPEN-HTTP2-STREAM and proper parameters.

  (defun my-send-client-request (connection url)
    (open-http2-stream connection
                              (request-headers :GET (puri:uri-path (puri:parse-uri url))
                                               (puri:uri-host (puri:parse-uri url)))
                              :end-stream t))

← ↑ → ↺

2.2 Server tutorials

Server related interfaces are exported from the HTTP2/SERVER package and are part of the HTTP2/SERVER system. This system is also loaded when HTTP2 is loaded.

← ↑ → ↺

2.2.1 Starting HTTP/2 server

[in package HTTP2/SERVER with nicknames HTTP2/SERVER/SHARED, HTTP2/SERVER/POLL, HTTP2/SERVER/THREADED]

Start server on foreground with RUN, or on background with START. You can stop server on background with STOP.

This creates (as of this version) a multithreaded server that serves 404 Not found responses on any request.

(http2/server:start 8443)
==> #<HTTP2/SERVER:DETACHED-TLS-THREADED-DISPATCHER HTTP/2 server on https://localhost:8443/>
==> #<PURI:URI https://localhost:8443/>
;; run curl -k https://localhost:8443/
127.0.0.1:58565 Connected, using VANILLA-SERVER-CONNECTION
127.0.0.1:58565 / [#1] - processing

← ↑ → ↺

2.2.2 Define content for HTTP/2 server

[in package HTTP2/SERVER with nicknames HTTP2/SERVER/SHARED, HTTP2/SERVER/POLL, HTTP2/SERVER/THREADED]

To server something else than 404 Not found, you need to define handlers for specific paths. Simple handler definition can look like

(define-exact-handler "/hello-world"
  (handler (foo :utf-8 nil)
    (with-open-stream (foo foo)
      (send-headers
       '((:status "200")
         ("content-type" "text/html; charset=utf-8")))
      (format foo "Hello World, this is random: ~a" (random 10)))))

This defines a handler on "/hello-world" path that sends reasonable headers, writes some text to the stream and closes the stream (via WITH-OPEN-STREAM). The text written is passed to the client as data (body).

In general, the handlers are set using DEFINE-PREFIX-HANDLER or DEFINE-EXACT-HANDLER, and are functions typically created by HANDLER macro, or (in simple cases) by REDIRECT-HANDLER or SEND-TEXT-HANDLER functions.

← ↑ → ↺

2.2.3 Getting request details

[in package HTTP2/SERVER with nicknames HTTP2/SERVER/SHARED, HTTP2/SERVER/POLL, HTTP2/SERVER/THREADED]

Sometimes you need to get some data from the request.

These data can be carried by querying the HTTP/2 stream object involved. If you define handlers by HANDLER macro, it is available in a lexically bound STREAM variable.

(define-exact-handler "/body"
  (handler (foo :utf-8 nil)
    (with-open-stream (foo foo)
      (send-headers
       '((:status "200")
         ("content-type" "text; charset=utf-8")))
      (format foo "Hello World, this is a ~s request.~3%content~%~s~3%headers~%~s~%~3%body~%~s~%"
         (http2/core::get-method stream)
         (http-stream-to-string stream)
         (http2/core::get-headers stream)
         (http2/core::get-body stream)))))

← ↑ → ↺

Body of the request

Sometimes there is a body in the client request.

When sending a such a request, you can use CONTENT parameter of the HTTP2/CLIENT:RETRIEVE-URL, together with CONTENT-TYPE.

 (http2/client:retrieve-url "https://localhost:8080/body" :content "Hello")
 (http2/client:retrieve-url "https://localhost:8080/body"
       :content #(1 2 3) :content-type "application/octet-stream")

When you write a handler for such a request, you should know if you want binary or text data. The vanilla class for the server streams looks at the headers, and if they look like UTF-8 (as per IS-UTF8-P), it processes the data as text, if not, they are collected as binary vector.

When your client systematically send headers that do not make it TEXT and you want to read text, as last resort change class of your streams to include FALLBACK-ALL-IS-ASCII (or improve IS-UTF8-P, or add some other decoding function).

If you do not want to see text at all, change class to NOT include UTF8-PARSER-MIXIN or any other conversion mixin.