google / nginx-sxg-module Goto Github PK
View Code? Open in Web Editor NEWNGINX SXG module
License: Apache License 2.0
NGINX SXG module
License: Apache License 2.0
Hey guys, so we tried following the guide below using debian packages only (not installing from source), we got the DigiCert Certificate, installed everything properly, we already have SSL, the site is configured fine but we just wanted to test out the SXG Extension:
https://web.dev/how-to-set-up-signed-http-exchanges/
And when adding the sxg on
; anywhere inside the server block, it shows:
nginx: [emerg] unknown directive "sxg" in /etc/nginx/sites-enabled/xxxx.conf
nginx: configuration file /etc/nginx/nginx.conf test failed
our main nginx conf file looks like this:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
client_max_body_size 100m;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
add_header X-Content-Type-Options nosniff;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
So the module is enabled, everything seems to be fine except for when adding that directive. I can also send the actual nginx conf if necessary.
Just tried building this with nginx but it appears that the build failed with libsxg.h missing.
Was this missed in the initial commit?
nginx-sxg-module/ngx_http_sxg_filter_module.c
Lines 326 to 327 in 9d11d8a
nginx-sxg-module/ngx_http_sxg_filter_module.c
Lines 823 to 824 in 9d11d8a
https://tools.ietf.org/html/rfc7230#section-5.3 suggests that clients can make requests of the form GET http://host.example/path HTTP/1.1
, though in practice they don't for typical clients/servers.
It seems that this line:
nginx-sxg-module/ngx_http_sxg_filter_module.c
Lines 702 to 703 in 97ea78b
https://host.examplehttp://host.example/path
.
TODO:
unparsed_uri
isn't as unparsed as its name implies?).Given the rarity of this case, it seems low-priority, and I'd recommend the easiest solution, e.g. don't generate a SXG when unparsed_uri
is absolute.
apt install valgrind
into the containervalgrind --exit-on-first-error=yes --error-exitcode=1 nginx -g 'daemon off;' &
instead of nginx
for ((x=0; x<100; x++)); do curl http://localhost:8080/certs/cert.cbor >/dev/null 2>&1 & done
(and ditto for an SXG)Optionally, research non-default args to valgrind, or other dynamic analysis tools.
In current implementation, evaluation of qvalue
is done with simple comparison of number.
When multiple candidates have the same qvalue
, the plugin will give priority to one appeared first in the list.
As a natural motivation of the user, app/sxg
header should have priority in spite of the order in the accept-header list when it have the same qvalue
.
Then, add tie-break to prioritize app/sxg
for such situation.
Main SXG response is timeout when subresources are stored to the file in NGINX server with proxy_cache setting. This timeout issue only happens when there is a cache file of subresource in NGINX server and the modules sends a subrequest.
How to reproduce:
/static
)proxy_cache_path /tmp/cache levels=1:2 keys_zone=nodejs:100m max_size=2g inactive=60m;
upstream node {
server app:3000;
}
server {
listen 80;
listen 443 ssl;
ssl_certificate /etc/nginx/server.crt;
ssl_certificate_key /etc/nginx/server.key;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name example.com;
sxg on;
sxg_certificate /etc/nginx/cert.pem;
sxg_certificate_key /etc/nginx/priv.key;
sxg_cert_url https://sxg-sandbox.firebaseapp.com/cert.cbor;
sxg_validity_url https://example.com/validity/resource.msg;
location = / {
proxy_cache nodejs;
proxy_cache_key $host$uri;
proxy_cache_valid 200 60m;
add_header X-Content-Type-Options nosniff;
add_header Vary Accept;
add_header Cache-Control "public, max-age=30";
proxy_pass http://node;
}
location /static {
proxy_cache nodejs;
proxy_cache_key $host$uri;
proxy_cache_valid 200 60m;
add_header X-Content-Type-Options nosniff;
add_header Vary Accept;
proxy_pass http://node;
}
}
node.js server example:
const express = require('express');
const path = require('path')
const PORT = 3000;
const HOST = '0.0.0.0';
const app = express();
// maxAge option is required because express server sets "Cache-Control: public, max-age=0" as a default, which leads NGINX not to cache response.
app.use('/static', express.static('public', {maxAge: 1000 * 60}))
app.get('/', (req, res) => {
res.append('Link', '</static/style.css>;rel="preload";as="style"')
res.sendFile(path.join(__dirname,'./public/index.html'), {cacheControl: false})
});
app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);
// on your local machine
curl 'https://example.com/static/style.css'
// on NGINX server. Check if the cache file exists
cat /tmp/cache/e/99/abda96163e15bae457c85f803036699e | less
curl -H 'Accept: application/signed-exchange;v=b3' 'https://example.com' --output - -v
kSignedExchangeEnabledAcceptHeaderForPrefetch
is used for <link rel=(preload|prefetch)>
even outside of SXG context. This can interact with https://crbug.com/1180441 resulting in broken subresources for pages that use that tag and run this worker.
To avoid this, the module should serve SXG if and only if its q-value is 1, until https://crbug.com/1243065 is fixed.
Hi,
I have SXGs enabled for my domain Mustakbil.com.
But when I search google.com for Mustakbil, and click the very first link which points to the home page, it opens a random wrong page of my website.
Here's the markup of the link from google search results:
<a href="https://www.mustakbil.com/" data-sxg-url="https://www-mustakbil-com.webpkgcache.com/doc/-/s/www.mustakbil.com/"></div></span><div><span class="VuuXrf">Mustakbil.com</span><div class="byrV5b"><cite class="tjvcx GvPZzd cHaqb" role="text" style="max-width:315px">https://www.mustakbil.com</cite></div></div></div></a>
The href is pointing correctly to https://www.mustakbil.com/
and SXGs url is also pointing correctly to https://www-mustakbil-com.webpkgcache.com/doc/-/s/www.mustakbil.com/
But when I click this link it opens wrong url. Which is a random page of my website but not the actual homepage.
from discussion in https://github.com/google/nginx-sxg-module/pull/37/files
make-debs.yml and end-to-end-test.yml are both failing with "The job was canceled because "ubuntu_eoan_eoan" failed." Details from the logs:
Step 5/12 : RUN apt-get update && apt-get install -y --no-install-recommends -q build-essential ca-certificates cmake debhelper devscripts dpkg-dev fakeroot git libssl-dev lsb-release moreutils wget
---> Running in caa6101fb843
Ign:1 http://security.ubuntu.com/ubuntu eoan-security InRelease
Ign:2 http://archive.ubuntu.com/ubuntu eoan InRelease
Err:3 http://security.ubuntu.com/ubuntu eoan-security Release
404 Not Found [IP: 91.189.88.152 80]
Ign:4 http://archive.ubuntu.com/ubuntu eoan-updates InRelease
Ign:5 http://archive.ubuntu.com/ubuntu eoan-backports InRelease
Err:6 http://archive.ubuntu.com/ubuntu eoan Release
404 Not Found [IP: 91.189.88.142 80]
Err:7 http://archive.ubuntu.com/ubuntu eoan-updates Release
404 Not Found [IP: 91.189.88.142 80]
Err:8 http://archive.ubuntu.com/ubuntu eoan-backports Release
404 Not Found [IP: 91.189.88.142 80]
Reading package lists...
E: The repository 'http://security.ubuntu.com/ubuntu eoan-security Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu eoan Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu eoan-updates Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu eoan-backports Release' does not have a Release file.
The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends -q build-essential ca-certificates cmake debhelper devscripts dpkg-dev fakeroot git libssl-dev lsb-release moreutils wget' returned a non-zero code: 100
I think "#else" is correct.
Hello,
I inspect this url https://github.com/google/nginx-sxg-module/blob/master/packaging/build_deb but I can't find libnginx-mod.nginx.skeleton in related the path so I can't create deb file.
Serving static resources such as html, js, image etc from local, that is a common situation of nginx use case. But now the module expects they come only from upstream
if subrequest is called, and cause the worker process error (this will be fixed in #46).
We can easily reproduce it in below situation.
If the response header of a document includes:
Link: <https://example.com/static/preload.js>;rel="preload";as="script"
nginx conf:
location /static {
root /var/www/app;
try_files $uri $uri/;
}
Is is possible to support this use case? Anyway the current behavior is a little bit confusing, so let's have a documentation at the moment.
Approximately 50% of CBOR requests in our build will fail with an empty server reply, due to the worker process crashing:
Nginx log:
2021/11/01 02:08:32 [alert] 16#16: worker process 17 exited on signal 11
2021/11/01 02:08:32 [alert] 17#17: *8 ignoring stale global SSL error (SSL: error:2706F06C:OCSP routines:OCSP_response_get1_basic:no response data), client: 95.217.86.147, server: www.xxxx.com, request: "GET /xxxx.com.cert.cbor HTTP/1.0", host: "1.2.3.4"
Stack trace:
Program received signal SIGSEGV, Segmentation fault.
0x00007f7863cbc18e in OCSP_response_get1_basic () from target:/lib/x86_64-linux-gnu/libcrypto.so.1.1
(gdb) bt
#0 0x00007f7863cbc18e in OCSP_response_get1_basic () from target:/lib/x86_64-linux-gnu/libcrypto.so.1.1
#1 0x000055925675be49 in refresh_if_needed ()
#2 0x000055925675c91b in ?? ()
#3 0x000055925669b540 in ngx_http_core_access_phase ()
#4 0x0000559256696e7d in ngx_http_core_run_phases ()
#5 0x00005592566a1edf in ?? ()
#6 0x00005592566a22d4 in ?? ()
#7 0x000055925668a183 in ?? ()
#8 0x0000559256680717 in ngx_process_events_and_timers ()
#9 0x00005592566882d0 in ?? ()
#10 0x00005592566867ed in ngx_spawn_process ()
#11 0x0000559256689019 in ngx_master_process_cycle ()
#12 0x0000559256660141 in main ()
I will attempt to reproduce on a debug build with source available and report back when available.
Versions:
FROM ubuntu:21.04 AS build
RUN wget https://github.com/cloudflare/quiche/raw/master/extras/nginx/nginx-1.16.patch
RUN cd nginx-module-vts && git checkout 3c6cf41315bfcb48c35a3a0be81ddba6d0d01dac
RUN cd nginx-rtmp-module && git checkout v1.2.2
RUN cd nginx-statsd && git checkout ef52718c0e1cf6b52899c89da89b28933eb11557
RUN cd ngx_brotli && git checkout 9aec15e2aa6feea2113119ba06460af70ab3ea62
RUN cd libsxg && git checkout 672b807e9a1a0032c73f1eeb7c9761d8cfd013d0
RUN cd nginx && git checkout release-1.16.1
RUN cd nginx-sxg-module && git checkout e748c0406bf0ab297a9dea97ea844fd1f75eefb0 # f742dad1a56e54c7af629f8a06efc3316d6700fa
RUN git clone https://github.com/openresty/headers-more-nginx-module.git
RUN cd headers-more-nginx-module && git checkout a4a0686605161a6777d7d612d5aef79b9e7c13e0
Configure:
RUN PATH=$HOME/.cargo/bin:$PATH ./auto/configure \
--with-cc-opt='-O2 -fstack-protector-strong -Wformat -Wno-error=implicit-fallthrough -Werror=format-security -Werror=maybe-uninitialized -D_FORTIFY_SOURCE=2' \
--with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' \
--prefix=/usr \
--conf-path=/etc/nginx/nginx.conf \
--http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log \
--lock-path=/var/lock/nginx.lock \
--pid-path=/var/run/nginx.pid \
--modules-path=/usr/loacl/lib/nginx/modules \
--http-client-body-temp-path=/var/lib/nginx/body \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
--http-proxy-temp-path=/var/lib/nginx/proxy \
--http-scgi-temp-path=/var/lib/nginx/scgi \
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
--with-compat \
--with-pcre-jit \
--with-ipv6 \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_image_filter_module \
--with-http_realip_module \
--with-http_auth_request_module \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_dav_module \
--with-http_slice_module \
--with-threads \
--with-http_addition_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_sub_module \
--with-stream \
--with-stream_ssl_module \
\
--add-module=../nginx-module-vts \
--add-module=../nginx-rtmp-module \
--add-module=../nginx-statsd \
--add-module=../ngx_brotli \
--add-module=../njs/nginx \
--add-module=../headers-more-nginx-module \
--add-module=../nginx-sxg-module
This function blocks outside the event loop.
nginx-sxg-module/ngx_http_sxg_filter_module.c
Line 861 in 0ffe86c
nginx-sxg-module/ngx_sxg_utils.c
Lines 397 to 398 in 0ffe86c
can we avoid blocking on the refreshing certificate?
If a subrequest response is larger than subrequest_output_buffer_size
, the main response is empty. Instead, that particular preload URL should be omitted (since it wasn't possible to compute header-integrity), and the rest of processing should continue.
If there are two server directives, SXG doesn't work. Normal HTML file is returned even if the client has Accept: application/signed-exchange;v=b3
header.
Here is the nginx conf file I tested.
/etc/nginx/sites-enabled/default
upstream node {
server app:3000;
}
server {
listen 80;
# listen [::]:80 default_server;
listen 443 ssl;
# listen [::]:443 ssl default_server;
ssl_certificate /etc/nginx/server.crt;
ssl_certificate_key /etc/nginx/server.key;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name example.com;
add_header X-Content-Type-Options nosniff;
add_header Vary Accept;
sxg on;
sxg_certificate /etc/nginx/cert.pem;
sxg_certificate_key /etc/nginx/priv.key;
sxg_cert_url https://sxg-sandbox.firebaseapp.com/cert.cbor;
sxg_validity_url https://example.com/validity/resource.msg;
location / {
proxy_pass http://node;
}
}
server {
server_name another-server;
listen 8000;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
location / {
try_files $uri $uri/ =404;
}
}
nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
sendfile on;
tcp_nopush on;
# tcp_nodelay on;
keepalive_requests 100;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
subrequest_output_buffer_size 4096k;
charset utf-8;
ssl_protocols TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log debug;
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Testing subresource feature with a single line of Link header.
This doesn't work.
link: <https://example.com/static/style.css>; rel="preload"; as="style"
This works.
link: <https://example.com/static/style.css>;rel="preload";as="style"
The task extracting a url from the header might have some issues, but not sure.
So far I still don't test with multiple link headers, or comma separated values yet.
The module should remove cache related headers in upstream response when it change the response by transforming to SXG. Otherwise static resources in a client cache wouldn't be updated even if SXG is out of date.
Static resources from upstream usually have cache related headers, such as ETag
and Last-Modified
. Those are useful to let a client know if the cache is still fresh or not. But so far the module doesn't change them when transforming the response to SXG. That causes expired SXG files (mainly static sub-resources) being stored for a long time unexpectedly.
At the moment, just removing cache related headers seems good way I think.
/setups/nginx-sxg-module/ngx_sxg_utils.c
/setups/nginx-sxg-module/ngx_sxg_utils.c: In function ‘get_term_length’:
/setups/nginx-sxg-module/ngx_sxg_utils.c:38:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
for (size_t i = 0; i < len; ++i) {
^
/setups/nginx-sxg-module/ngx_sxg_utils.c:38:3: note: use option -std=c99 or -std=gnu99 to compile your code
/setups/nginx-sxg-module/ngx_sxg_utils.c: In function ‘sxg_qvalue_is_1’:
/setups/nginx-sxg-module/ngx_sxg_utils.c:165:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
for (const char* const end = str + len; str < end;) {
^
/setups/nginx-sxg-module/ngx_sxg_utils.c: In function ‘check_refresh_needed’:
/setups/nginx-sxg-module/ngx_sxg_utils.c:365:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
for (int i = 0; i < resps; ++i) {
^
tried compiling this module with ngnix getting above error
If i try CFLAGS="$CFLAGS -std=c99" getting error for u_char datatype
Please help me with this how to compile this module with Ngnix
I initially installed libsxg to a custom path using -DCMAKE_INSTALL_PREFIX=$HOME/usr
. I believe using -DCMAKE_PREFIX_PATH=$HOME/usr
in this project should enable this to include/link with that location, but it didn't work. I believe some combination of find_library
/target_link_libraries
/etc. in CMakeLists.txt is necessary. Possible help from:
The signed fallback URL does not include the ?foo
from the original URL.
The construct_fallback_url function:
nginx-sxg-module/ngx_http_sxg_filter_module.c
Lines 430 to 431 in 56e7556
After this fix, I'm guessing that URLs with a ?
at the end of them will still be signed improperly, but that's a relatively minor bug.
When upstream response header is having the preload Link:
in which the url is full path including domain name, the sxg filter makes an subrequest with the wrong URL, then a worker process terminates abnormally. It seems that ngx_http_subrequest expects only path string, not including domain. However, the current implementation set the URL which is passed from the link header value from upstream response as it is. That causes the issue.
I put logs below.
Response header from upstream:
Link: <https://example.com/static/style.css>;rel="preload";as="style"'
Error log:
2020/05/28 14:57:15 [debug] 8#8: *1 http upstream request: "/?"
2020/05/28 14:57:15 [debug] 8#8: *1 http upstream process header
2020/05/28 14:57:15 [debug] 8#8: *1 malloc: 00005648477A9F20:4096
2020/05/28 14:57:15 [debug] 8#8: *1 recv: eof:0, avail:1
2020/05/28 14:57:15 [debug] 8#8: *1 recv: fd:14 871 of 4096
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy status 200 "200 OK"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "X-Powered-By: Express"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Link: <https://example.com/static/style.css>;rel="preload";as="style""
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Accept-Ranges: bytes"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Cache-Control: public, max-age=0"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Last-Modified: Tue, 26 May 2020 08:54:35 GMT"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "ETag: W/"200-1725030ccf8""
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Content-Type: text/html; charset=UTF-8"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Content-Length: 512"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Date: Thu, 28 May 2020 14:57:15 GMT"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header: "Connection: close"
2020/05/28 14:57:15 [debug] 8#8: *1 http proxy header done
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: param is: [<https://example.com/static/style.css>]
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: param is: [rel="preload"]
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: param is: [as="style"]
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: found as statement in link preload header: style
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: param is: [<https://example.com/static/style.css>]
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: param is: [rel="preload"]
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: param is: [as="style"]
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: found as statement in link preload header: style
2020/05/28 14:57:15 [debug] 8#8: *1 nginx-sxg-module: non preload header:
2020/05/28 14:57:15 [debug] 8#8: *1 invoke request for: https://example.com/static/style.css as style
2020/05/28 14:57:15 [debug] 8#8: *1 posix_memalign: 00005648477AAF30:4096 @16
2020/05/28 14:57:15 [debug] 8#8: *1 http subrequest "https://example.com/static/style.css?"
...
2020/05/28 14:57:15 [debug] 8#8: *1 http filename: "/var/www/htmlhttps://example.com/static/style.css"
2020/05/28 14:57:15 [debug] 8#8: *1 add cleanup: 00005648477ABE18
2020/05/28 14:57:15 [error] 8#8: *1 open() "/var/www/htmlhttps://example.com/static/style.css" failed (2: No such file or directory), client: 172.25.0.1, server: example.com, request: "GET / HTTP/1.1", subrequest: "https://example.com/static/style.css", host: "example.com"
2020/05/28 14:57:15 [debug] 8#8: *1 http finalize request: 404, "https://example.com/static/style.css?" a:1, c:2
2020/05/28 14:57:15 [alert] 1#1: worker process 8 exited on signal 11
It ends-up returning an empty response to a client.
$ curl --insecure -H 'Accept: application/signed-exchange;v=b3' https://example.com --output -
curl: (52) Empty reply from server
Of course a full URL is a valid format in Link header, so that would be great if the module supports it. Maybe the module should extract only a path string from the passed URL before making a subrequest? If it can't be supported soon, can we consider having the error handling for this issue as a quick fix? It can be critical for users.
I spotted this line cause this error.
nginx-sxg-module/ngx_http_sxg_filter_module.c
Line 276 in b19a5c6
I'm working on implementing signed exchanges for a website and I have a question and maybe a potential feature requests. Most of the HTML I'm serving has an expiration date of just a few days, however a lot of static content (hashed) uses 365 days. Signed exchanges can be generated for both, however in front of my Nginx servers I also have a load balancer with CDN capabilities. If it starts caching requests for signatures, then these will be stale. I couldn't find a way to differentiate expires
and cache-control
headers based on the fact if it's an SGX request or not. Should the module automatically set cache-control
and expires
to a reasonable value?
To make it more clear, that's my setup:
Requests made from AMP caches would also end up on the load balancer and possibly be served by the CDN, unless there's a way I can prevent that from happening.
Change build_deb
and libsxg's build_deb
to have a non-default option to build with debug symbols and safety analysis. Propagate those settings down from end-to-end-test.yml
Candidates:
Hello .
I'm new with nginx and accept my apologies if the question is obvious.
i have SXG certificate from DigiCert. but in the SXG config i should give cert.cbor and resource.msg path.
how can i generate these files? all i have is pem file.
thank you in advance.
sxg on;
sxg_certificate /path/to/sxg/mySxg.pem;
sxg_certificate_key /path/to/sxg/mySxg.key;
sxg_cert_url https://website.test/certs/cert.cbor;
sxg_validity_url https://website.test/validity/resource.msg;
sxg_cert_path /certs/cert.cbor;
When nginx is misconfigured, finding startup message is good clue to debug.
There should some informational log message in the initializing sequence.
Hi! I'm leaving this issue here as there's a potential to improve the behavior of the module, however most importantly I wanted to leave somewhere a solution I found, so feel free to close if you don't see it relevant to fix. Other developers should be able to find it in closed issues as well.
A common way of implementing "clean URLs" (no trailing slashes) is to use try_files
directive:
try_files $uri $uri/index.html =404;
The URL looks like this: /foo/bar
, but the actual file lives in /foo/bar/index.html
. When try_files
is used alongside this module, the file path is in the signature instead of the requested URL: /foo/bar/index.html
.
As a workaround, I'm now using location
with a regular expression + alias
:
location ~ ^/(.*) {
rewrite ^/index(?:\.html|/)?$ / permanent;
rewrite ^/(.*)/index(?:\.html|/)?$ /$1 permanent;
rewrite ^/(.*)(?:\.html|/)$ /$1 permanent;
alias /var/www/$1/index.html;
}
Result is the same when it comes to regular requests, however signatures contain a correct URL: /foo/bar
with no /index.html
.
I wasn't fully able to produce a CBOR file with the module yet (like mentioned in a previous issue), but in the current phase of my testing I want to mention two important topics.
First, the ngx_http_cert_chain_handler
:
sxg_cert_url
, nothing touches that file. But I understand it, and it is logical. if (slc->cert_path.len <= 0 || req->uri.len != slc->cert_path.len || ngx_memcmp(req->uri.data, slc->cert_path.data, req->uri.len) != 0) { return NGX_OK; }
sxg_cert_path
and request URI
(if these strings are the same, it is considered a request to the CBOR file).Something like this:
The absolute path of the file:
sxg_cert_path /mnt/server_documents/certificates/customer_id_346512/sxg/document.cbor
The way this module asks for it to be delivered.
www.website.com/mnt/server_documents/certificates/customer_id_346512/sxg/document.cbor
In other words, you have to put the entire directory structure in the link, and serve it that way. Although not impossible, I find this approach a bit forcing. It was also undocumented, and had to go through the code to understand why anything wasn't refreshed(or tried to be refreshed) and my SXG's were invalid after some time. I was just serving the CBOR file in a different URI. And the string comparison returning False, prevented the code from being executed. There could be many possible ways to improve this, so it's an open debate. Even without an improvement, documenting the current behavior would certainly clarify things.
Secondly, logging of refreshing:
bool refreshed = refresh_if_needed(&slc->cert_chain); if (refreshed) { ngx_log_error( NGX_LOG_INFO, req->connection->log, 0, "nginx-sxg-module: OCSP Response in Certificate-Chain is refreshed"); }
It is logged only when CBOR refresh is needed and successful. A failure log is just as needed, with explaining the failure reason (OCSP error, file permission error etc...). Without fail log, the CBOR could be expiring without an alert.
The reason I primarily inspected sxg_cert_path
is that it is such a fundamental feature of the module. SXG simply doesn't work without it. Even though there is an alternative for it (gen-certurl), it could be more appropriate for people to do without any 3rd party tool, which brings something other than nginx to maintain/monitor.
It also takes like seven days for the CBOR to expire, making the tests very slow, and failure unapparent (You don't know if it's failing). So at least the documentation is paramount.
Thanks for your time, and your good work. Have a nice day.
I follow the steps on this doc
https://web.dev/how-to-set-up-signed-http-exchanges/#step-3-option-2
When I run make
after config nginx-sxg-module
it returns below error
-o objs/addon/nginx-sxg-module/ngx_sxg_utils.o \ ../nginx-sxg-module/ngx_sxg_utils.c In file included from src/os/unix/ngx_process.h:12, from src/core/ngx_core.h:55, from ../nginx-sxg-module/ngx_sxg_utils.c:20: src/os/unix/ngx_setaffinity.h:16:9: **error: unknown type name 'cpu_set_t'** 16 | typedef cpu_set_t ngx_cpuset_t; | ^~~~~~~~~ make[1]: *** [objs/Makefile:1234: objs/addon/nginx-sxg-module/ngx_sxg_utils.o] Error 1
I am using alpine:3.12 and nginx 1.19.2
nginx's subrequest has a limit (NGX_HTTP_MAX_SUBREQUESTS=50).
What should ngx_http_sxg_filter_module behave when it exceeds the limit of nginx's subrequest?
nginx-sxg-module/ngx_http_sxg_filter_module.c
Lines 533 to 535 in 0ffe86c
As spec requires
When served over HTTP, a response containing an "application/signed-
exchange" payload MUST include at least the following response header
fields, to reduce content sniffing vulnerabilities (Section 6.8):
o Content-Type: application/signed-exchange;v=_version_
o X-Content-Type-Options: nosniff
When this plugin is enabled, the nginx will return SXG to the qvalue
of the request header.
If there is any possibility of requested URL may returns SXG, Vary
response header should always be appended.
param_is_as
function in ngx_sxg_utils.c is not tested by fuzzer.
Every parser which accepts wild strings from internet should heavily tested to prevent attacks.
Currently, lifespan of SXG files created by this module are limited just 1 day.
In spec, SXG allows lifespan 7 days in maximum, the lifespan should be configurable in nginx config files.
From discussion at https://github.com/google/nginx-sxg-module/pull/37/files#r430855638
nginx running in threaded mode, handling subresource may happen simultaneously. We need mutex to support module context structure.
The module has the feature that automatically includes signatures of subresources if the upstream response has the link
header with rel=preload
. However, subrequest isn't happened if the header is Link
because the current implementation seems to be case-sensitive.
nginx-sxg-module/ngx_http_sxg_filter_module.c
Lines 401 to 411 in 6112087
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html
Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.
This is relatively minor bug, but it should be fixed I think.
Chromium will recursively prefetch any link
s from their origin unless allowed-alt-sxg
and alternate
are also specified. Thus, such links impede the privacy-preserving prefetch use case.
invoke_subrequests:
nginx-sxg-module/ngx_http_sxg_filter_module.c
Lines 334 to 335 in 56e7556
link
header to exclude any rel=preload
s for which integrity cannot be obtained, per subresource_fetch_handler: nginx-sxg-module/ngx_http_sxg_filter_module.c
Line 274 in 56e7556
(I'm not sure the best way to do that. I assume if the modification happens inside subresource_fetch_handler, it'll need to be protected by a mutex.)
I'm not sure this is a bug or intentional behavior though, currently the module doesn't allow duplicated sxg
directives and causes a startup error. So far we can't enable SXG only on a specific upstream, that makes A/B testing difficult to see how SXG performs better for loading performance.
location = /bar/ {
sxg on;
proxy_pass http://bar;
}
location = /foo/ {
sxg off;
proxy_pass http://foo;
}
Result
nginx: [emerg] "sxg" directive is duplicate in /etc/nginx/sites-enabled/default:38
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.