Jetty 12.0.6: NPE in error handling leading to 100% cpu usage by Jetty threads about jetty.project HOT 5 CLOSED

We believe we have a fix for #11326 that will be in the 12.0.7 release at the end of the month.

The ISE is also something that we have been addressing, but it has mostly been a harmless second call to callback.failed() after a callback has already been succeeded or failed.
The NPE is also a problem in the error handling, which also looks like something has already cleared the headers as we are trying to write out an error.

So it may be that 12.0.7 will have fixed these, but I'll have a bit more of a look at this area before the release and at the very least put in a bit better handling of errors whilst handling errors. For both of those exceptions it would be good to see the actual error that was being handled.

from jetty.project.

@gregw what about the NPE in org.eclipse.jetty.http.HttpField.getHeader() during a ServletContextResponse.resetContent call?

from jetty.project.

@joakime yeah, that NPE should not happen with normal handling of a Mutable HttpFields class. So it is plausible that this is again double handling of an error, so two threads are modifying the HttpFields at once. The null kind of suggests one thread is adding a field, whilst the other is trying to clear it.

So I do suspect the error handling that we have already cleaned up somewhat, but I can't yet say for sure this is fixed.

@konstantin-mikheev can you tell us more about the actual failure. I.e. details about what you mean with "infinite stream of messages". Plus next time it happens, can you take a full thread dump.

from jetty.project.

Ugh! The NPE is an exception thrown whilst handling an exception thrown whilst handling an exception! It is turtles all the way down!

from jetty.project.

"infinite stream of messages" meant we got 100k identical NPE messages in logs per minute for the next few minutes. Node had died and was restarted so no thread dump unfortunately.

During this timeframe besides those 100k per minute NPEs we got 6 IllegalStateExceptions from org.eclipse.jetty.ee10.servlet.ServletChannelState. Besides those there are only 34 other messages all of those not really interesting: couple of timeouts towards other locations and retries while trying to update some settings. Usually those logs mean packetloss.

I have looked into ploss statistics by location and found out at the last case there was no changes in ploss rate when incident started. Out of about 30 locations there was one with about 20% ploss which was already going for approximately 50 minutes. There were couple of ploss spikes to other locations within 30 minutes before NPEs happened but they were not happening at the time when NPEs started to appear.

For now I have ported project back to Jetty 11. I will probably left one instance with Jetty 12 to try to obtain thread dump but could not promise anything unfortunately.

What also could I add: I have looked into cpu usage distribution and there is an increase from 5 to 20% of the "kernel mode" cpu usage for couple of minutes right when those issues started. I will investigate more what could it be.

from jetty.project.