I was playing with PowerDNS and Lightningstream in Kubernetes and ran into a very weird issue. It is a StatefulSet and storage comes from Longhorn.
PowerDNS auth and Lightningstream are both the latest versions and it is running on ARM64.

The issue is that a query to pdns (dns or rest api) hangs and returns nothing. Executing the pdnsutil tool on the pdns container hangs as well. Like pdnsutil list-all-zones hangs and the prompt never returns.
Even with debug logging, there is no error. It shows the incoming request and that it is processing, but no answer is returned.

First I thought it is related to the pid clashing issue for containers that is mentioned in the docs. But it does not matter what I use for --minimum-pid, the issue persisted.

The issue goes away, if I send a dns query to PDNS for a valid record/zone before lightningstream is started. So the command in my lightingstream container is a shell script, which first sends a DNS query to the pdns container and then starts lightning stream.

It still looks somewhat like a locking issue, but the question is what is different internally how pdns auth server accesses the LMDB backend when there was a dns request to pdns before lightningstream is started vs when there was no request?

It runs stable with no issues so far with the workaround.

Do you have any input on what could be going on here?

Configurations listed below.

My pdns.conf looks like this.

pdns@pdns-ss-0:/$ cat /etc/powerdns/pdns.conf
primary=yes
allow-notify-from=0.0.0.0
allow-axfr-ips=127.0.0.1
api=yes
api-key=secret
config-dir=/etc/powerdns
default-soa-content=a.misconfigured.dns.server.invalid hostmaster.@ 0 10800 3600 604800 3600
default-ttl=3600
default-ksk-algorithm=ed25519
default-zsk-algorithm=ed25519
include-dir=/etc/powerdns/pdns.d
load-modules=liblmdbbackend.so
launch=lmdb
lmdb-filename=/var/lib/powerdns/pdns.lmdb
lmdb-shards=1
lmdb-sync-mode=nometasync
lmdb-schema-version=5
lmdb-random-ids=yes
lmdb-map-size=1000
lmdb-flag-deleted=yes
lmdb-lightning-stream=yes
local-address=0.0.0.0,::
log-dns-details=yes
log-dns-queries=yes
log-timestamp=yes
loglevel=7
loglevel-show=yes
query-logging=yes
resolver=1.1.1.1
# server-id
version-string=anonymous
webserver=yes
webserver-allow-from=127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,192.0.0.0/24
webserver-hash-plaintext-credentials=yes
webserver-loglevel=detailed
webserver-address=0.0.0.0
webserver-port=8081
webserver-password=secret2
zone-cache-refresh-interval=0
zone-metadata-cache-ttl=0

My lightningstream.yaml looks like this.

/app $ cat /lightningstream.yaml
instance: lmdbsync
storage_poll_interval: 10s
lmdb_poll_interval: 10s
storage_force_snapshot_interval: 4h

lmdbs:
  main:
    path: /var/lib/powerdns/pdns.lmdb
    schema_tracks_changes: true
    options:
      no_subdir: true
      create: false
  shard:
    path: /var/lib/powerdns/pdns.lmdb-0
    schema_tracks_changes: true
    options:
      no_subdir: true
      create: false

storage:
  type: s3
  options:
    access_key: pdns
    secret_key: pdns
    bucket: lightningstream
    endpoint_url: http://10.0.3.194:9000
    create_bucket: true
  cleanup:
    enabled: true
    interval: 15m
    must_keep_interval: 24h
    remove_old_instances_interval: 168h

http:
  address: ":8500"

log:
  level: info
  format: human
  timestamp: short

My shell script with the workaround:

/app $ cat start.sh
#!/bin/sh

if [ ${PDNS_LSTREAM_SLEEP}_ == _ ]; then
    echo "no sleep time set"
else
    sleep ${PDNS_LSTREAM_SLEEP}
fi

if [ ${PDNS_LSTREAM_DNS_SERVER}_ == _ ]; then
    echo "no DNS server set"
else
    eval DNS=\$$PDNS_LSTREAM_DNS_SERVER
    echo "ServiceIP: $DNS"
    echo "PodIP: $PDNS_POD_IP"
    dig @$PDNS_POD_IP $PDNS_LSTREAM_DOMAIN $PDNS_QUERY_TYPE
fi

/app/lightningstream --config /app/lightningstream.yaml --minimum-pid 200 --instance ${HOSTNAME}-lstream sync

Currently we do not run tests against Minio in CI. We should add it.

It would be very helpful if the stats command can also output how many entries per DBI are marked as deleted.

Since this requires a full scan of the LMDB, this should be disabled by default and require a flag like --full.

As a workaround, you can find the number of deleted entries with something like:

lightningstream snapshots dump ... | grep flags=01 | wc -l

Change the powerdns.com/platform/lightningstream import path to github.com/PowerDNS/lightningstream so that go install and https://pkg.go.dev/github.com/PowerDNS/lightningstream work correctly.

The alternative would be to add these url endpoints with special html meta tags on our website and make sure it always works, which is probably not worth the trouble.

The docs talk of use_update_marker to reduce S3 bills, but the obvious question here is what is the frequency of the LIST calls in the first place. The docs are silent on that ?

The different commercial S3 providers provide different levels of "free" LIST calls and so it would be nice to have an accurate guess at at which point people would exceed "free" and have a good idea of likely S3 bills.

When starting up a fresh docker compose setup, LS hangs on this error:

lightningstream-sync1-1  | level=info msg="[main          ] Waiting for initial receiver listing" 
                           db=main error="file does not exist" instance=instance-1

I suspect that Simpleblob is returning the "file does not exist" instead of an empty listing.

From a cursory glance at the docs there currently appears to be no encryption support.

If lightningstream is used in a cloud scenario (vs self-hosted MiniIO) then it could be attractive to encrypt contents prior to upload to the cloud.

With modern algorithms I can't imagine this would add too much overhead ?

As an enhancement to CI, I suggest adding govulncheck: https://github.com/marketplace/actions/golang-govulncheck-action

The simpleblob S3 backend leaks minio client reader objects over time.

See PowerDNS/simpleblob#44 (originally reported by @franklouwers, credit in release notes).
See PowerDNS/simpleblob#45 for the fix in simpleblob. LS needs to be upgraded to use it.

Likely introduced in v0.2.0 (first public release).

Cc @ahouene @nvaatstra

Please consider enhancing the docs by adding a section where you describe the process for migrating between S3 stores.

For example, say you start on MinIO and you wish to migrate to AWS S3, or vice-versa or any other number of examples.

It is not immediately clear how this would be safely done ?

Upgrade simpleblob when PowerDNS/simpleblob#31 is merged and released, to get clear errors when buckets do not exists, and in other failure scenarios.

Hi,

we love this. But providing a geo-redundant s3 bucket in our infrastructure would pretty much replace the complexity we tried to remove.

And hosting our data in a public cloud isn't really an option.

Therefore I would like to throw out the idea to implement some client side encryption for the data stored in s3. Just a shared key symmetric encryption would be plenty goo.

Looking through the codebase, it looks like lightningstream does not use the Content-MD5 header in S3-compatible APIs in order to ensure end-to-end integrity of blobs uploaded.

It would be nice to have a couple of paragraphs in the docs as to how lightningstream approaches this.

Please be aware that the use_update_marker functionality is not working, likely due to PowerDNS/simpleblob#40

When testing lightningstream with my own project written in JS (See kriszyp/lmdb-js#267) I noticed that the files generated are not readable by lightningstream.

Apparently the LMDB libs were updated and new binary formats introduced. And lmdb-go is still using the old one. (kriszyp/lmdb-js#273)
Thankfully lmdb-js can fall-back to v1, but I wonder if it would be good to go with the new format here.

I guess this is more of a lmdb-go issue. Perhaps I should raise it there?

PS: In the first issue I posted I posted some code. Do you see any serious issues with it? It seems to work, but I don't have the feeling that lightningstream is yet much used outside the pdns project...

Hi,
I'm trying to build a 1master - multiple read only slaves setup,

for security reasons, I would like to set the s3 access to readonly on the slaves nodes,but it don't seem to work,
I got a "access denied".

Setting write permissions is working, as they are creating shards on the s3
Is is expected for receiver only nodes ?