Short
When using a netcat client and server setup in Oakestra flakiness occurs, i.e. if you send 20 messages several of them get dropped with no clear pattern or explanation. - This was my assumption a week ago, when I now replicated this it no longer occurred ...
Deeper description of the bug
Concrete Example
Let's take service C (client) and S (server).
(Note: There are multiple different netcat implementations. I used "netcat-traditional".)
We run this script on S
#!/bin/bash
while true
do
# Make sure to use -p otherwise no message can get propagated.
# -w teminates the server session after 5 s. This is used to avoid getting stuck due to a broken/stale connection.
nc -l -p 99 -w 5
doneAnd this cmd on C:
# -q is used to terminate the client after sending the message to avoid getting the client stuck.
for i in {1..20}; do echo $i | nc 10.30.27.3 99 -q 1; doneThis screenshot shows outputs of flaky behavior. On my local setup (non Oakestra) there is a smooth flow from 1-20.
Update
I have pushed a custom image that has that exact netcat version installed as well as provided both scripts there for easier testing.
Here is the SLA:
{
"sla_version": "v2.0",
"customerID": "Admin",
"applications": [
{
"applicationID": "",
"application_name": "app",
"application_namespace": "test",
"application_desc": "",
"microservices": [
{
"microserviceID": "",
"microservice_name": "server",
"microservice_namespace": "test",
"virtualization": "container",
"cmd": ["bash","server.sh"],
"memory": 100,
"vcpus": 1,
"storage": 0,
"code": "ghcr.io/malyuk-a/netcat:testing",
"addresses": {
"rr_ip": "10.30.27.3"
}
},
{
"microserviceID": "",
"microservice_name": "client",
"microservice_namespace": "test",
"virtualization": "container",
"cmd": ["bash","client.sh"],
"memory": 100,
"vcpus": 1,
"storage": 0,
"code": "ghcr.io/malyuk-a/netcat:testing",
"one_shot": true
}
]
}
]
}Interestingly enough, now I no longer can replicate that flakiness ...
Example output:
...
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
...
I have no idea why or what is going on differently now all of a sudden. When experiencing the bug mentioned above I was able to replicate that behavior multiple times in the span of ~2-3 days.
I am able to see rather strange behavior when it comes to redeploying these services, I am not sure if this is related or due to the CLI tool I use that handles these deployments/creations in a very quick way.
Solution
We (@giobart, @smnzlnsk, @Malyuk-A) had a look and we couldn't spot errors in the NetManager logs, thus this needs deep analysis.
Update
Right now I simply want to know what others can observe. When you run the same SLA do you see a nice flow from 1-20 or do you see gaps? If multiple people do not see any gaps we can close this Issue I guess.
Status
Replicated and discussed. Needs deep analysis to figure out what is going wrong.
Why might this be critical: If a classic tool like netcat is not properly working who knows how other tools behave? This can very much interfere with proper practical/scientific work.
Update
Let's see, maybe this was a very strange anomaly on my local system's side.
Checklist
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent