Intermittently, the response latency for a request will be disproportionately long (> 10s). Volterra dashboards suggests the latency is in the app (as opposed to ADN hops).

Investigate why this is happening.

• Already reviewed container logs and didn't find anything
• Already tweaked workload flavors but this doesn't appear to have any effect

We can make things a little more robust with proxy and frontend health check (former on the HTTP LB and Latter on readiness/liveliness on the pod). For the latter, look at the loadgen examples in the google microservices respository.

With the mid-Jan release, the platform now supports Integrated Bot Defense in tenants where it's enabled. Support for this feature should be added to the HTTP Loadbalancer.

Move the loadgen workload to a separate namespace. This will:

1. Make RBAC easier
2. Save vk8s resources in the application NS
3. Make vk8s observability more intuitive

Once the tor loadgen container is in place (which will greatly vary source IP of the traffic), enable malicious user detection by IP address.

• Test running NGINX unit in an RE
• Host this behind a URI route ({{app}}/unit)
• Use standard 'hello world' apps for now -- add functionality later

• Write a customized block page
• craft a unique request that is filterable under security events (that gets blocked)
• generate a QR code per tenant for that request. Host the QR code in this repo so it can be linked.

This will allow people giving the demo to meaningfully show filtering capabilities.

The Volterra error pages are jarring at this point.

Label-based or endpoint-based policies help prevent unauthorized traffic from reaching the endpoints. We should enable network policies for demo-app to highlight the policies capabilities.
Have a generator send requests to various endpoints to demo the blocking nature of policies

Make test traffic appear to come from more geographically diverse sources.

Dashboard data for API endpoint discovery features are not being populated.

• update app type to collect relevant metrics
• verify appropriate tag against relevant volterra object

It appears the 'microservices-demo' project has pulled some of the older image tags from the gcr.io repos. We can no longer pull the older 'adservice' from the repo.

This is a good time to update to the new "boutique" app (moving away from the hipster shop branding).

The Security Events only shows the last 500 events. 500 events takes about 12 minutes to fill up.

Update k6 loadgen so that TLS version is varied per session.

See here for more details.

Investigate if this Is this something that could easily be integrated into the demo application.

Make the traffic look more realistic within the our constraints (ASNs and clients will always be limited to REs).

refactor resulting in some stray b64 encoding.

Terraform is not aware that the vk8s credentials have expired, so volterra_api_credential.cred and local_file.kubeconfig are not updated. Results in the following error:

│ Error: failed to create kubernetes rest client for read of resource: Get "https://tenant.console.ves.volterra.io/api/vk8s/namespaces/demo-app/demo-app-vk8s/api?timeout=32s": remote error: tls: expired certificate │ │ with module.kubectl.kubectl_manifest.documents[6], │ on modules/kubectl/main.tf line 28, in resource "kubectl_manifest" "documents": │ 28: resource "kubectl_manifest" "documents" {

Thanks

Generate a list of ~10 unique attach events that will trigger security monitoring.

The Azure AD authentication is still happening even though the wingman status comes back "wingman not ready". Refactor logic so that authentication will not happen until wingman is ready.

There is little need to log excessively for this application.

1. Proxy container -- no need to have access logs
2. Frontend container -- same access logs there

Don't rely on local TF execution.

• Remote State
• Cover all relevant tenants

AI/ML should be enabled for demo-app ( demo-app NS) to discover API endpoints and create OAS files. It also aids in malicious user detection

As deployed, the redis based cart service runs on each RE. This causes state inconsistencies with 'Add to Cart' functionality.

Adding stickiness to RE based loadbalancers are yielding 503s. The more correct way to solve for state would be to expose a single redis pod to all instances of the app.

For one reason or another, pods will fail (and the scheduler will recreate these). They are not being cleaned up and this is counting against storage quotas.

Fix this.

Test this to see if vk8s auth flakiness is due to the current provider being used.

We're bumping up against quota usage limitations within a single vk8s. We need to implement more granular control of container resource limits within our pods.

In deployment.spec

    resources:
      limits:
        memory: "200Mi"
      requests:
        memory: "100Mi"

With Workload flavors

See limits here.
We could create our own flavor (under 'shared' namespace) and assign it. Note this endpoint is not covered in the TF provider.

The current TF didn't properly anticipate the app would be deployed over several workspaces/tenants.

The kubeconfig file should be tied to an individual workspaces state and it's lifecycle tied to the vk8s cluster/creation.

Dump these in a public slack which SEs could join as needed.