Container tests fail with EADDRINUSE about entropy HOT 11 CLOSED

cc @halter73

from entropy.

I have no idea what's going on with this one. The client simply failed to establish a TCP connection. Maybe Kestrel just took too long to start this one time? That's why there's a retry.

Even though TC marked it as flaky, if you look at the test history it looks like this is the only time this test has ever failed: http://aspnetci/project.html?tab=testDetails&projectId=XPlat&testNameId=8180339492066270673&page=1

from entropy.

ConfigWalkingValuesWebTests.KestrelX64CoreCLR
Failure details: http://aspnetci/viewLog.html?buildId=191998&tab=buildResultsDiv&buildTypeId=XPlat_Linux_UbuntuUniverseCoreCLR.

@halter73 This one has the same error. Fails to connect and retry is canceled.

@muratg Are these rare failures acceptable?

from entropy.

@halter73 we want to abolish all flaky tests. We have too many flaky tests across the board, so it's a death-by-a-thousand-paper-cuts situation.

from entropy.

@natemcmaster could you take a look? (@halter73 is out this week.)

from entropy.

ContainerFallbackWebTests failed with

Unhandled Exception: System.IO.IOException: Failed to bind to address http://localhost:6000 on the IPv4 loopback interface: port already in use. ---> System.AggregateException: One or more errors occurred. (Error -98 EADDRINUSE address already in use) ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -98 EADDRINUSE address already in use

ConfigWalkingValuesWebTests failed with the same error, but ports 5900. I couldn't find any other tests that explicitly use ports 5900 or 6000. Could this be an issue in Kestrel?

from entropy.

Didn't we have some automatic port selector thingy that we used somewhere for Kestrel tests? Or some other tests?

@mikeharder @Tratcher maybe you guys remember?

from entropy.

If we do that would help. To help debug situations like this, I'm make a pull request to make SelfHostDeployer log the status of requested port before it attempts to start the web server. See aspnet/Hosting#993

from entropy.

Most Kestrel tests specify port 0 and let the OS dynamically assign a free port. A few Kestrel tests do bind to specific ports, and we use a tiny helper method to choose a free port:

private static int GetNextPort()
{
    using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
    {
        // Let the OS assign the next available port. Unless we cycle through all ports
        // on a test run, the OS will always increment the port number when making these calls.
        // This prevents races in parallel test runs where a test is already bound to
        // a given port, and a new test is able to bind to the same port due to port
        // reuse being enabled by default by the OS.
        socket.Bind(new IPEndPoint(IPAddress.Loopback, 0));
        return ((IPEndPoint)socket.LocalEndPoint).Port;
    }
}

https://github.com/aspnet/KestrelHttpServer/blob/dev/test/Microsoft.AspNetCore.Server.Kestrel.FunctionalTests/AddressRegistrationTests.cs#L508

This code could be copied where needed since it's so small and simple, or we could move it into a (source) shared class.

from entropy.

Thanks @mikeharder. I've used this approach in #216 and removed all hard-coded port numbers from tests.

from entropy.

#216

from entropy.