1. Use this branch
   git checkout hard-coded-certs

2. Build it
   GIT_TERMINAL_PROMPT=1 SNAPCRAFT_BUILD_ENVIRONMENT=host snapcraft --debug

3. Install docker
   sudo snap install docker

4. Edit /etc/hosts. Add these lines

   146.148.90.233 kaas-edge-nodes.arm.com
   146.148.90.233 kaas-edge-admin.arm.com
   146.148.90.233 fog-proxy.arm.com
   

5. Install pelion-edge
   sudo snap install --dangerous --classic pelion-edge_1.2_amd64.snap

6. Save this to test-pod.yaml

   apiVersion: v1
   kind: Pod
   metadata:
     name: test-pod-016c6de7e4f7000000000001001003c1
   spec:
     automountServiceAccountToken: false
     hostname: test-pod
     nodeName: 016c6de7e4f7000000000001001003c1
     containers:
     - name: client
       image: alpine:3.9
       command: ["/bin/sh"]
       args: ["-c","echo 'hello'; sleep 6000000"]
       imagePullPolicy: IfNotPresent
       resources:
         limits:
           cpu: 200m
           memory: 100Mi
         requests:
           cpu: 200m
           memory: 10Mi
     restartPolicy: Always

7. Install kubectl

8. Make sure the node 016c6de7e4f7000000000001001003c1 is Ready

   $ kubectl get nodes
   NAME                               STATUS     ROLES    AGE     VERSION
   016c674b365b00000000000100100071   NotReady   <none>   8d      v0.0.0-argus+$Format:%h$
   016c6d0bfb6d00000000000100100233   NotReady   <none>   6d22h   v0.0.0-argus+$Format:%h$
   016c6d813206000000000001001001c4   NotReady   <none>   6d20h   v0.0.0-argus+$Format:%h$
   016c6daad42300000000000100100276   NotReady   <none>   6d20h   v0.0.0-argus+$Format:%h$
   016c6de7e4f7000000000001001003c1   Ready      <none>   6d19h   v1.13.2-argus

9. Deploy the Pod
   kubectl create -f test-pod.yaml

10. Make sure it's in the ContainerCreating state

    $ kubectl get pods
    NAME                                        READY   STATUS              RESTARTS   AGE
    test-pod-016c6de7e4f7000000000001001003c1   0/1     ContainerCreating   0          1s

11. Follow the kubelet logs
    journalctl -f -u snap.pelion-edge.kubelet.service

When kubelet attempts to create the container for this pod it fails repeatedly due to this error.

container "0cec1b8f0bec5f2c88bed2103e66825dd78f5c555ec6cbf766710ed2ec13ad9d" network for pod "test-pod-016c6de7e4f7000000000001001003c1": NetworkPlugin cni failed to set up pod "test-pod-016c6de7e4f7000000000001001003c1_default" network: failed to Statfs "/proc/2841/ns/net": permission denied
Aug 14 11:14:59 desktop pelion-edge.kubelet[29727]: E0814 11:14:59.111301   29900 kuberuntime_sandbox.go:65] CreatePodSandbox for pod "test-pod-016c6de7e4f7000000000001001003c1_default(a615cca7-beae-11e9-9621-263fe8375b9d)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "0cec1b8f0bec5f2c88bed2103e66825dd78f5c555ec6cbf766710ed2ec13ad9d" network for pod "test-pod-016c6de7e4f7000000000001001003c1": NetworkPlugin cni failed to set up pod "test-pod-016c6de7e4f7000000000001001003c1_default" network: failed to Statfs "/proc/2841/ns/net": permission denied
Aug 14 11:14:59 desktop pelion-edge.kubelet[29727]: E0814 11:14:59.111330   29900 kuberuntime_manager.go:662] createPodSandbox for pod "test-pod-016c6de7e4f7000000000001001003c1_default(a615cca7-beae-11e9-9621-263fe8375b9d)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "0cec1b8f0bec5f2c88bed2103e66825dd78f5c555ec6cbf766710ed2ec13ad9d" network for pod "test-pod-016c6de7e4f7000000000001001003c1": NetworkPlugin cni failed to set up pod "test-pod-016c6de7e4f7000000000001001003c1_default" network: failed to Statfs "/proc/2841/ns/net": permission denied
Aug 14 11:14:59 desktop pelion-edge.kubelet[29727]: E0814 11:14:59.111406   29900 pod_workers.go:190] Error syncing pod a615cca7-beae-11e9-9621-263fe8375b9d ("test-pod-016c6de7e4f7000000000001001003c1_default(a615cca7-beae-11e9-9621-263fe8375b9d)"), skipping: failed to "CreatePodSandbox" for "test-pod-016c6de7e4f7000000000001001003c1_default(a615cca7-beae-11e9-9621-263fe8375b9d)" with CreatePodSandboxError: "CreatePodSandbox for pod \"test-pod-016c6de7e4f7000000000001001003c1_default(a615cca7-beae-11e9-9621-263fe8375b9d)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"0cec1b8f0bec5f2c88bed2103e66825dd78f5c555ec6cbf766710ed2ec13ad9d\" network for pod \"test-pod-016c6de7e4f7000000000001001003c1\": NetworkPlugin cni failed to set up pod \"test-pod-016c6de7e4f7000000000001001003c1_default\" network: failed to Statfs \"/proc/2841/ns/net\": permission denied"

These errors stem from a phase in the container creation where kubelet invokes CNI plugins to set up the container's networking interfaces. This error specifically occurs when kubelet executes the loopback binary stored in $SNAP/wigwag/system/opt/cni/bin/loopback. This binary hits a permission error when it attempts to run the statfs system call on /proc/2841/ns/net. This path changes on each attempt as the process ID of the container changes. When running outside of snap this error should not occur and the Pod state should read as 'Readywhen runningkubectl get pods`