The fact that you need to load a path in the kctf-conf/... directory is a bit unintuitive.
Some ideas:

• remember all loaded configs and allow switching between them
• allow loading a challenge directory. If there's multiple configs, ask which one to load

Since we now remotely cache the most common parts of the Docker images, perhaps we can now parallelize the build.

This seems to imply finding a way of running make in parallel:

Note that since make start might also touch things outside it's own directory (../kctf-conf/base), we currently can't just add a &.

We probably need to somehow separate the Makefile so we isolate all the task-specific build steps.

W0409 16:00:12.138478 156274 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.

For example, we can have:

kctf-chal-ls

that lists all challenges that are deployed, or available, exposed, etc.

And then we can have:

kctf-chal-batch

that executes the same Make command on all provided challenges.

so, for example, you can do:

kctf-chal-ls deployed | xargs kctf-chal-bach stop

kctf-config-create requires user interaction, so it's hard to automate cluster configuration generation (we need to set some environment variables or generate the config manually).

Would be nice if we had a way to do this in a scriptable form.

docker run --privileged -it --rm -p 8000:1337 kctf-chal-apache-php

+ mount -t proc none /kctf/.fullproc/proc
 * Starting Apache httpd web server apache2                                                                                                                                     

 * 
--2020-04-30 23:08:58--  http://127.0.0.1:1337/?apache-is-running
Connecting to 127.0.0.1:1337... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2020-04-30 23:08:58 ERROR 500: Internal Server Error.

==> /var/log/apache2/access.log <==

==> /var/log/apache2/error.log <==
[Thu Apr 30 23:08:58.622500 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [W][2020-04-30T23:08:58+0200][1] bool mnt::mountPt(mount_t*, const char*, const char*)():202 mount(''/kctf/pow' -> '/.kctf/pow' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:false mandatory:false') src:'/kctf/pow' dstpath:'/tmp/nsjail.1000.root//.kctf/pow' failed: No such file or directory: /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.622588 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [D][2020-04-30T23:08:58+0200][1] bool mnt::mountPt(mount_t*, const char*, const char*)():126 Mounting ''/tmp' flags: type:'tmpfs' options:'' dir:true': /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.622709 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [D][2020-04-30T23:08:58+0200][1] bool mnt::mountPt(mount_t*, const char*, const char*)():126 Mounting ''/mnt/disks/sessions' -> '/mnt/disks/sessions' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:false': /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.622854 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [W][2020-04-30T23:08:58+0200][1] bool mnt::mountPt(mount_t*, const char*, const char*)():166 open('/tmp/nsjail.1000.root//mnt/disks/sessions', O_CREAT|O_RDONLY|O_CLOEXEC, 0644): Is a directory: /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.623050 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [W][2020-04-30T23:08:58+0200][1] bool mnt::mountPt(mount_t*, const char*, const char*)():202 mount(''/mnt/disks/sessions' -> '/mnt/disks/sessions' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:false') src:'/mnt/disks/sessions' dstpath:'/tmp/nsjail.1000.root//mnt/disks/sessions' failed: No such file or directory: /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.623123 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [F][2020-04-30T23:08:58+0200][1] bool subproc::runChild(nsjconf_t*, int, int, int, int)():448 Launching child process failed: /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.623221 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [W][2020-04-30T23:08:58+0200][95] bool subproc::runChild(nsjconf_t*, int, int, int, int)():478 Received error message from the child process before it has been executed: /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.623288 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [E][2020-04-30T23:08:58+0200][95] int nsjail::standaloneMode(nsjconf_t*)():256 Couldn't launch the child process: /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.623339 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] AH01215: [D][2020-04-30T23:08:58+0200][95] int main(int, char**)():343 Returning with 255: /usr/lib/cgi-bin/nsjail-php-cgi
[Thu Apr 30 23:08:58.623368 2020] [cgi:error] [pid 35:tid 139991861860096] [client 127.0.0.1:36272] End of script output before headers: nsjail-php-cgi

==> /var/log/apache2/other_vhosts_access.log <==
kctf-nsjail-php:1337 127.0.0.1 - - [30/Apr/2020:23:08:58 +0200] "GET /?apache-is-running HTTP/1.1" 500 800 "-" "Wget/1.20.3 (linux-gnu)"

https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc#creating_a_cluster_in_your_first_service_project

make docker is probably not needed for example, there are several other kctf- scripts in bin/ that probably don't need the symlink (we can leave them in scripts/ if needed but removing them from bin makes it simpler)

According to https://cloud.google.com/load-balancing/docs/ssl-certificates/troubleshooting#certificate-managed-status Provisioning a Google-managed certificate might take up to 60 minutes. This makes it unusable during a CTF in practice.

Note that a possibly worse problem is leaking challenge names before the CTF starts - see for example https://crt.sh/?q=kctf.dev

google/nsjail#132

The old trick

def stdin_is_localhost_socket():
    try:
        sock = socket.socket(fileno=sys.stdin.fileno())
        peername = sock.getpeername()
        sock.detach()
        return peername[0] == '::ffff:127.0.0.1'
    except OSError:
        return False

Doesn't work anymore
google/nsjail@273ce6b

it's a bit confusing we have two
https://www.npmjs.com/package/proof-of-work

How to contribute to kCTF - like contributors.md

gsutil acl ch -u "${GSA_EMAIL}:O" "gs://${BUCKET_NAME}" from scripts/cluster/start.sh is only working with Fine-grained access-control. If Uniform access-control enabled for bucket, this needs to be changed for:

gsutil iam ch serviceAccount:${GSA_EMAIL}:roles/storage.legacyBucketOwner gs://${BUCKET_NAME}/

Probably need to add check before choosing the command.

Let's say you have a working deployment.
You want to update it, but the new one doesn't start.
Now for debugging, you run make ssh or make healthcheck-ssh, but that will ssh to the working version instead of the failing one.

HTTP/S load balancers support mapping through the hostname. We can map different hostnames to different HTTP ports. #101 (comment)

Currently web tasks are http, but some tasks will require https. Setting this up atm is kindof complicated.

We need to somehow pass down to the task its own hostname, so it can request a certificate at runtime, or we need to do this at config time so we can request a certificate through k8s.

Background on k8s CRD: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/

We can add challenge object with it's yaml config looking smth like:

apiVersion: "kctf.dev/v1"
kind: Challenge
metadata:
  name: mychal
spec:
  public: true
  healthcheck: false
  https: true
  dns: true
  autoscaling: {}
  # any other arbitrary fields

On the k8s cluster size there will be a controller (needs to be implemented): it will just watch for creation/update of challenge resource in the cluster, and set it up upon creation/update. E.g. if you change public: true to public: false it will delete service for that challenge.

Some advantages of having this:

1. challenge will become a 1st-class k8s API citizen, allowing to use kubectl list challenges or kubectl describe challenge mychal
2. it allows us to use standard config (k8s yaml) and standard commands like kubectl create challenge mychal or kubectl delete challenge mychal
3. we can take away some steps from being executed locally to k8s controller, e.g. building an image for challenge: the controller would create a Job for building an image and then creates a Deployment for challenge
4. this can also help with versioning: we can specify it with apiVersion: "kctf.dev/v1", this is again a standard way of k8s API versioning

Either at the k8s side:
https://github.com/containernetworking/plugins/tree/master/plugins/meta/bandwidth
https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/#support-traffic-shaping
https://github.com/minrk/tc-init/blob/master/tc-init

or with tc/iptables:
https://www.frozentux.net/iptables-tutorial/iptables-tutorial.html#OWNERMATCH
https://lartc.org/howto/lartc.cookbook.fullnat.intro.html
http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm

this would limit the risk of DoS

as evidenced by #96 our logic wasn't very robust

Takes a while to install, we probably can move them too (like pwntools, chroot, nsjail, etc)

We currently have an apache+php example. this is fine for many challenges, but it's likely authors will want to write tasks in python, ruby, nodejs, etc

As such, one easy way to implement this would be by:

1. Having nsjail in listening mode on the port
2. Running the web server inside nsjail listening on port 8080
3. Have socat redirect STDIN/STDOUT to that port socat - TCP:localhost:8080 (an alternative solution would be to have nsjail support port forwarding by default google/nsjail#134)

This has the downside that every http request, including to static resources, would trigger startup costs for nsjail and the web server (which is worsened by the fact there's no pow for web tasks (see #32)).

This can be ameliorated by telling authors to split their application, so static resources are served by apache, and "dynamic" paths are proxied to the other server listening locally.

Looks like they now support python3
https://github.com/Gallopsled/pwntools

Adding a new cached docker image is currently very finicky since you have to touch multiple files.
This should be easy to avoid by automating the push etc to automatically handle all subdirectories of config/docker.

I.e. ideally I would just create a new file in config/docker/my-image/Dockerfile, and the tooling would create the image gcr.io/../kctf-my-image for me.
This should work with a for dir in config/docker/* loop.

here:
https://github.com/google/kctf/blob/master/.github/workflows/test.yaml

we don't actually check that the healthcheck passed properly, just that it deployed. we should check that the deployment actually was succesful

right now they only get rebuilt when we make a change, but we probably should have something rebuilding them constantly

If you setup the wrong umask everything breaks. I wonder if we can detect this early on and fix it for the user somehow (or warn at least).

for now, here's what I'm doing to fix a 027 umask to 022

find . | xargs stat -c '%a %n' | grep -v './.git' | awk '{print substr($1,0,2) substr($1,2,1) " " $2}' | xargs -I{} bash -c "chmod {}"

Perhaps we can upload it as a package (for users with fast internet).

currently it is harcoded

an idea: it would be cool if the infrastructure could take care of the proof of work automatically. I.e. enabling it if it gets too much traffic and increasing the difficulty.

Possibly just run make healtcheck-test-docker for local, and make healthcheck-port-forward for remote

you can have a GSA for a workload. might be useful for limiting access

We should provide better monitoring for reliability and security

It's highly likely that someone running dirbuster on a .php file will cause issues, so we should add pow to web challenges too.

this can be done by wrapping the call to nsjail:

so that it checks:

1. If pow is enabled
2. if enabled, then check KCTF_POW cookie has a valid signed timestamp within tolerance
3. if cookie invalid, redirect to /.__kctf__/pow?continue=$path

and add /.__kctf__/pow that:

1. asks the user for a pow challenge
2. if the challenge is valid, set KCTF_POW cookie with a valid signed current timestamp
3. redirect user to ?continue param

kctf-config-load will delete the old config from kubectl. This breaks if the newly loaded config is the same as the old

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

With the operator, the way that we do it with a ConfigMap might be too much and unnecessary.
#128 (review)

kctf-upgrade or similar

We should support / document how testers can try out a challenge without exposing it to the public.

The approach that works right now is to give the tester k8s access to the gcp project and have them run make port-forward.
However, this is not ideal since it gives access to all challenges. Maybe the access could be limited per namespace?

Alternatively, maybe we could support opening the challenges to certain IPs only?

• investigate best approach to make challenges available to testers only
• implement
• document

kctf-chal-create $CHALLENGE_NAME -t $BASE

by default it copies "pwn" template, but can also copy "web" or "xss"

we then rename challenge-skeleton to pwn, apache-php to web and xss-bot to xss

The apache-php example didn't build on my machine since gcr.io/kctf-docker/kctf-nsjail-chroot:latest was outdated.
We should always try to fetch the newest images from gcr.io during the build process, at least when creating the challenge directory.
Or alternatively hardcode the versions and update them when they change.

To make the local kubernetes tutorial, you need to have gcloud installed, because a configuration is necessary for it to work (kctf config create). As it's a local walk-through, gcloud shouldn't be necessary.

• #13 replace makefiles to use source ../Makefile
• Remove the src directory
• Rename k8s to advanced
• Make config secret, move secrets to config