Comments (8)
reefland
commented on August 25, 2024
Was just going to open an issue on this...
$ sudo k3s etcd-snapshot ls | grep -c "s3://"
463
Even if I force prune, doesn't help:
$ sudo k3s etcd-snapshot prune --etcd-snapshot-retention 10
$ sudo k3s etcd-snapshot ls | grep -c "s3://"
463
$ sudo k3s --version
k3s version v1.29.5+k3s1 (4e53a323)
go version go1.21.9
from k3s.
brandond
commented on August 25, 2024
@reefland on-demand and scheduled snapshots have different base names, and pruning on-demand snapshots does not affect the scheduled ones. You'll need to specify the correct snapshot name to prune scheduled snapshots:
sudo k3s etcd-snapshot prune --etcd-snapshot-retention 10 --etcd-snapshot-name etcd-snapshot
If you've set a different default name for scheduled snapshots via the server's etcd-snapshot-name cli flag or config file key, use that instead of etcd-snapshot. The default name for on-demand snapshots is on-demand, and these are the only ones that are pruned by default via the CLI.
from k3s.
reefland
commented on August 25, 2024
I'm using the default snapshot names:
Name Location Size Created
etcd-snapshot-k3s01-1714233603 s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714233603 87822368 2024-04-27T12:00:03-04:00
etcd-snapshot-k3s02-1714255201 s3://k3s/etcd-snapshots/etcd-snapshot-k3s02-1714255201 84918304 2024-04-27T18:00:01-04:00
etcd-snapshot-k3s01-1714255202 s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714255202 88690720 2024-04-27T18:00:02-04:00
...
$ sudo k3s etcd-snapshot ls | grep -c "s3://k3s/etcd-snapshots/etcd-snapshot-"
466
Tried to use your suggestion with --etcd-snapshot-name but it didn't like it:
$ sudo k3s etcd-snapshot prune --etcd-snapshot-retention 10 --etcd-snapshot-name etcd-snapshot
Incorrect Usage: flag provided but not defined: -etcd-snapshot-name
...
FATA[0000] flag provided but not defined: -etcd-snapshot-name
But using --name etcd-snapshot instead seems to work, long list scrolled by:
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717538403 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1717538402 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717516805 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s02-1717516804 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1717516802 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717495205 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s02-1717495204 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1717495201 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1717473604 deleted.
....
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1714276802 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s03-1714255204 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1714255202 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s02-1714255201 deleted.
INFO[0002] Snapshot etcd-snapshots/etcd-snapshot-k3s01-1714233603 deleted.
However, the count did not drop:
$ sudo k3s etcd-snapshot ls | grep -c "s3://k3s/etcd-snapshots/etcd-snapshot-"
466
If I search for the 1st one it said was deleted, still shows up:
$ sudo k3s etcd-snapshot ls |grep "k3s01-1717538403"
etcd-snapshot-k3s01-1717538403 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-k3s01-1717538403 96759840 2024-06-04T18:00:03-04:00
etcd-snapshot-k3s01-1717538403 s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1717538403 96759840 2024-06-04T18:00:03-04:00
If I run the prune command again, gives me the same long list. And the k3s logs don't show any errors...
Jun 5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Checking if S3 bucket k3s exists"
Jun 5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="S3 bucket k3s exists"
Jun 5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Applying snapshot retention=10 to local snapshots with prefix etcd-snapshot in /var/lib/rancher/k3s/server/db/snapshots"
Jun 5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Applying snapshot retention=10 to snapshots stored in s3://k3s/etcd-snapshots/etcd-snapshot"
Jun 5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1717538403"
Jun 5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s03-1717538402"
Jun 5 12:48:50 k3s01 k3s[128858]: time="2024-06-05T12:48:50-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1717516805"
...
Jun 5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s03-1714255204"
Jun 5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714255202"
Jun 5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s02-1714255201"
Jun 5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Removing S3 snapshot: s3://k3s/etcd-snapshots/etcd-snapshot-k3s01-1714233603"
Jun 5 12:48:51 k3s01 k3s[128858]: time="2024-06-05T12:48:51-04:00" level=info msg="Reconciling ETCDSnapshotFile resources"
Jun 5 12:48:52 k3s01 k3s[128858]: time="2024-06-05T12:48:52-04:00" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"
Suggestion?
from k3s.
brandond
commented on August 25, 2024
Incorrect Usage: flag provided but not defined: -etcd-snapshot-name
Hmm, that's a bug. The other commands have aliases, I'll have to see why we missed it for that flag
--name value (db) Set the base name of the etcd on-demand snapshot (appended with UNIX timestamp). (default: "on-demand") --snapshot-compress, --etcd-snapshot-compress (db) Compress etcd snapshot --snapshot-retention value, --etcd-snapshot-retention value (db) Number of snapshots to retain. (default: 5) --s3, --etcd-s3 (db) Enable backup to S3
What do you have in the config.yaml? I am assuming that the rest of the etcd-s3 config options are in there; does that include etcd-s3-folder?
from k3s.
reefland
commented on August 25, 2024
etcd-s3: true
snapshot-compress: true
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 10
etcd-s3-bucket: "k3s"
etcd-s3-access-key: "<redacted>"
etcd-s3-secret-key: "<redacted>"
etcd-s3-endpoint: "truenas.<redacted>:9000"
etcd-s3-region: "minio"
etcd-s3-folder: "etcd-snapshots"
BTW - The docs https://docs.k3s.io/cli/etcd-snapshot reference --etcd-snapshot-schedule-cron but the k3s command gives me a warning:
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
Not sure why it has a problem with the entry above.
from k3s.
brandond
commented on August 25, 2024
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
Yeah, that flag is only valid on the server command-line to configure the snapshot schedule. It is not valid to pass that on the etcd-snapshot command line because it is inherently a one-shot option, not scheduled via cron. The warning on the CLI is not great but is an artifact of how the config parser converts the config file to CLI args.
Look at this a little more closely, I guess even on-demand prune is broken :/ The best you can probably do at the moment is to manually run etcd-snapshot delete with a list of snapshots to delete from S3. That should do the correct thing.
from k3s.
aganesh-suse
commented on August 25, 2024
Validated on master branch with version v1.30.2-rc3+k3s1
Environment Details
Infrastructure
- Cloud
- Hosted
Node(s) CPU architecture, OS, and Version:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
$ uname -m
x86_64
Cluster Configuration:
HA: 3 server/ 1 agent
Config.yaml:
token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
etcd-snapshot-retention: 2
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: <access-key>
etcd-s3-secret-key: <secret-key>
etcd-s3-bucket: <s3-bucket>
etcd-s3-folder: <s3-folder>
etcd-s3-region: <region>
debug: true
Testing Steps
- Copy config.yaml
$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
- Install k3s
curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.30.2-rc3+k3s1' sh -s - server
- Verify Cluster Status:
kubectl get nodes -o wide
kubectl get pods -A
- Perform:
$ sudo k3s etcd-snapshot prune
$ sudo k3s etcd-snapshot list
pruned snapshot should not be listed after being deleted.
Replication Results:
- k3s version used for replication:
$ k3s -v
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2
$ sudo /usr/local/bin/k3s etcd-snapshot prune
time="2024-06-21T01:09:48Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2024-06-21T01:09:48Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2024-06-21T01:09:48Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2024-06-21T01:09:48Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2024-06-21T01:09:48Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2024-06-21T01:09:48Z" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
time="2024-06-21T01:09:49Z" level=info msg="Snapshot on-demand-ip-172-31-30-182-1718932088 deleted."
time="2024-06-21T01:09:49Z" level=info msg="Snapshot <s3-folder>/on-demand-ip-172-31-30-182-1718932088 deleted."
$ sudo /usr/local/bin/k3s etcd-snapshot list 2>/dev/null On SERVER1: 18.188.91.198
Name Location Size Created
etcd-snapshot-ip-172-31-30-182-1718931604 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718931604 770080 2024-06-21T01:00:04Z
etcd-snapshot-ip-172-31-30-182-1718931661 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718931661 3207200 2024-06-21T01:01:01Z
etcd-snapshot-ip-172-31-30-182-1718931721 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718931721 3473440 2024-06-21T01:02:01Z
etcd-snapshot-ip-172-31-30-182-1718931784 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718931784 3948576 2024-06-21T01:03:04Z
etcd-snapshot-ip-172-31-30-182-1718931841 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718931841 4218912 2024-06-21T01:04:01Z
etcd-snapshot-ip-172-31-30-182-1718931901 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718931901 4378656 2024-06-21T01:05:01Z
etcd-snapshot-ip-172-31-30-182-1718931961 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718931961 5148704 2024-06-21T01:06:01Z
etcd-snapshot-ip-172-31-30-182-1718932024 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718932024 5378080 2024-06-21T01:07:04Z
etcd-snapshot-ip-172-31-30-182-1718932082 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718932082 5546016 2024-06-21T01:08:02Z
on-demand-ip-172-31-30-182-1718932088 s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-30-182-1718932088 5591072 2024-06-21T01:08:08Z
on-demand-ip-172-31-30-182-1718932096 s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-30-182-1718932096 5636128 2024-06-21T01:08:16Z
on-demand-ip-172-31-30-182-1718932096 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-30-182-1718932096 5636128 2024-06-21T01:08:16Z
on-demand-ip-172-31-30-182-1718932104 s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-30-182-1718932104 5677088 2024-06-21T01:08:24Z
on-demand-ip-172-31-30-182-1718932104 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-30-182-1718932104 5677088 2024-06-21T01:08:24Z
etcd-snapshot-ip-172-31-30-182-1718932144 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718932144 5791776 2024-06-21T01:09:04Z
etcd-snapshot-ip-172-31-30-182-1718932144 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-30-182-1718932144 5791776 2024-06-21T01:09:04Z
etcd-snapshot-ip-172-31-30-182-1718932200 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-30-182-1718932200 6778912 2024-06-21T01:10:00Z
etcd-snapshot-ip-172-31-30-182-1718932200 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-30-182-1718932200 6778912 2024-06-21T01:10:00Z
on-demand-ip-172-31-30-182-1718932088 is still listed above - after it was supposed to be deleted.
Validation Results:
- k3s version used for validation:
$ k3s -v
k3s version v1.30.2-rc3+k3s1 (aa4794b3)
go version go1.22.4
$ sudo /usr/local/bin/k3s etcd-snapshot prune
time="2024-06-21T01:08:06Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2024-06-21T01:08:06Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2024-06-21T01:08:06Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2024-06-21T01:08:06Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2024-06-21T01:08:06Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2024-06-21T01:08:06Z" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
time="2024-06-21T01:08:06Z" level=info msg="Snapshot on-demand-ip-172-31-21-205-1718931997 deleted."
time="2024-06-21T01:08:06Z" level=info msg="Snapshot on-demand-ip-172-31-21-205-1718931997 deleted."
$ sudo /usr/local/bin/k3s etcd-snapshot list
Name Location Size Created
on-demand-ip-172-31-21-205-1718932004 s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-21-205-1718932004 5562400 2024-06-21T01:06:44Z
on-demand-ip-172-31-21-205-1718932004 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-21-205-1718932004 5562400 2024-06-21T01:06:44Z
on-demand-ip-172-31-21-205-1718932014 s3://<s3-bucket>/<s3-folder>/on-demand-ip-172-31-21-205-1718932014 5607456 2024-06-21T01:06:54Z
on-demand-ip-172-31-21-205-1718932014 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-172-31-21-205-1718932014 5607456 2024-06-21T01:06:54Z
etcd-snapshot-ip-172-31-21-205-1718932020 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-21-205-1718932020 5644320 2024-06-21T01:07:00Z
etcd-snapshot-ip-172-31-21-205-1718932020 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-21-205-1718932020 5644320 2024-06-21T01:07:00Z
etcd-snapshot-ip-172-31-21-205-1718932084 s3://<s3-bucket>/<s3-folder>/etcd-snapshot-ip-172-31-21-205-1718932084 5824544 2024-06-21T01:08:04Z
etcd-snapshot-ip-172-31-21-205-1718932084 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-172-31-21-205-1718932084 5824544 2024-06-21T01:08:04Z
on-demand-ip-172-31-21-205-1718931997 is not on the list.
P.S: the --etcd-snapshot-name is still not recognized. --name worked.
If we dont want to handle this fix in this bug, we could create another issue for the same to track.
from k3s.
aganesh-suse
commented on August 25, 2024
$ k3s -v
k3s version v1.30.2-rc3+k3s1 (aa4794b3)
go version go1.22.4
sudo k3s etcd-snapshot prune --name on-demand
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
WARN[0000] Unknown flag --node-label found in config.yaml, skipping
WARN[0000] Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation.
INFO[0002] Snapshot on-demand-ip-172-31-21-205-1718933571 deleted.
INFO[0002] Snapshot on-demand-ip-172-31-21-205-1718933490 deleted.
INFO[0002] Snapshot on-demand-ip-172-31-21-205-1718932014 deleted.
INFO[0002] Snapshot on-demand-ip-172-31-21-205-1718933571 deleted.
INFO[0002] Snapshot on-demand-ip-172-31-21-205-1718933490 deleted.
INFO[0002] Snapshot on-demand-ip-172-31-21-205-1718932014 deleted.
$ sudo k3s etcd-snapshot prune --etcd-snapshot-name on-demand
Incorrect Usage: flag provided but not defined: -etcd-snapshot-name
NAME:
k3s etcd-snapshot prune - Remove snapshots that match the name prefix that exceed the configured retention count
USAGE:
k3s etcd-snapshot prune [command options] [arguments...]
OPTIONS:
--debug (logging) Turn on debug logs [$K3S_DEBUG]
--config FILE, -c FILE (config) Load configuration from FILE (default: "/etc/rancher/k3s/config.yaml") [$K3S_CONFIG_FILE]
--log value, -l value (logging) Log to file
--alsologtostderr (logging) Log to standard error as well as file (if set)
--node-name value (agent/node) Node name [$K3S_NODE_NAME]
--data-dir value, -d value (data) Folder to hold state default /var/lib/rancher/k3s or ${HOME}/.rancher/k3s if not root
--token value, -t value (cluster) Shared secret used to join a server or agent to a cluster [$K3S_TOKEN]
--server value, -s value (cluster) Server to connect to (default: "https://127.0.0.1:6443") [$K3S_URL]
--dir value, --etcd-snapshot-dir value (db) Directory to save etcd on-demand snapshot. (default: ${data-dir}/db/snapshots)
--name value (db) Set the base name of the etcd on-demand snapshot (appended with UNIX timestamp). (default: "on-demand")
--snapshot-compress, --etcd-snapshot-compress (db) Compress etcd snapshot
--snapshot-retention value, --etcd-snapshot-retention value (db) Number of snapshots to retain. (default: 5)
--s3, --etcd-s3 (db) Enable backup to S3
--s3-endpoint value, --etcd-s3-endpoint value (db) S3 endpoint url (default: "s3.amazonaws.com")
--s3-endpoint-ca value, --etcd-s3-endpoint-ca value (db) S3 custom CA cert to connect to S3 endpoint
--s3-skip-ssl-verify, --etcd-s3-skip-ssl-verify (db) Disables S3 SSL certificate validation
--s3-access-key value, --etcd-s3-access-key value (db) S3 access key [$AWS_ACCESS_KEY_ID]
--s3-secret-key value, --etcd-s3-secret-key value (db) S3 secret key [$AWS_SECRET_ACCESS_KEY]
--s3-bucket value, --etcd-s3-bucket value (db) S3 bucket name
--s3-region value, --etcd-s3-region value (db) S3 region / bucket location (optional) (default: "us-east-1")
--s3-folder value, --etcd-s3-folder value (db) S3 folder
--s3-insecure, --etcd-s3-insecure (db) Disables S3 over HTTPS
--s3-timeout value, --etcd-s3-timeout value (db) S3 timeout (default: 5m0s)
FATA[0000] flag provided but not defined: -etcd-snapshot-name
from k3s.
Related Issues (20)
- service ip is unreachable from pod HOT 3
- How do I know which container of this path belongs to HOT 1
- Node cannot get it´s IP - host IP unknown; known addresses: [] HOT 2
- etcd issues when removing and re-adding nodes HOT 7
- Validate SLES 15 SP6 HOT 1
- Validate Amazon Linux 2023 HOT 2
- Validate RHEL/Rocky/Oracle 8.10 HOT 2
- Validate RHEL/Rocky/Oracle 9.4 HOT 3
- k3s server crashes abruptly when approximately 300 requests are sent in a span of 60 seconds (client QPS is 5 and bursty limit is 10) HOT 1
- panic: runtime error: invalid memory address or nil pointer dereference HOT 8
- Configuration problem with static CPU manager HOT 1
- Migrate Docker tests from Drone to GHA
- [Release 1.29] Bump flannel v0.25.4 and fix issue with IPv6Masq HOT 1
- [Release 1.27] Bump flannel v0.25.4 and fix issue with IPv6Masq HOT 1
- [Release 1.28] Bump flannel v0.25.4 and fix issue with IPv6Masq HOT 1
- Install systemd unit to run cleanup on shutdown or reboot
- Ingress is going on all the nodes HOT 4
- k3s-agent not starting on ec2 error "failed to get CA certs" HOT 2
- Ingress is going on all the nodes HOT 1
- It looks like the Spegel is not working.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.