[BUG] kernel oops that happens during provison are marked as panic in the last task about beaker HOT 8 OPEN

Panic detection can be configured by PANIC_REGEX, it's set on LC basis

You can disable panic detection all together by including <watchdog panic="None"/> in your recipe.

from beaker.

Note, I don't want to ignore Oops in general. My only problem are with Oops that happened on provision that causes the last task in the recipe to be marked as Panic, this task shouldn't be marked as panic as there was no Oops when the task executed.

from beaker.

@bgoncalv is right. I'm aware of this behavior. The problem is that we can't really mark Panic during installation at this moment, therefore most of the time it lands on the first task. But, it may happen that we failed to detect it at first therefore it will land on another (the one which was running when we found Panic).

Yeah. This needs to remain open and we will need to find a better way how to manage this.

from beaker.

Thanks @StykMartin, do you have any suggestion how can we workaround this? For example, we could create a dummy task to trigger this panic detection and run it as first task. Is there a way for a task force to run the detection?

from beaker.

Hello @bgoncalv. My answer never ended on GitHub. I'm sorry about that. So the situation is quite a bit more complicated, but I feel like we can make some compromise if you still need it.

So panic detection is running on background in each region. Then it is proxied to the main server to process. The main server will check tasks assigned to given recipe and iterate over. If iteration is exhausted we will mark last item - because that's what will end up in variable.

If you have any idea how we can help you @bgoncalv to make it saner feel free to shoot.

from beaker.

@StykMartin thanks for the reply. Indeed that feature would be helpful for us. If the panic detection will assign panic detected during provision always in the last task of the recipe that would help us indeed.

from beaker.

This is happening at this moment. All panics are assigned to the last task if there is no running task. So basically if we didn't register any start (for example provisioning) we will always mark last.
However, there is a catch. If restraint will report n-2 task as finished and n-1 is not reported as started then we may report panic in this window to last as well. But changes are quite small that restraint will crash in moments like this.

from beaker.

I would suggest you put dummy tasks at the end and then collect panics from there.

I will try to redesign this feature so we can report panics during provisioning to dedicated space and not marking tasks.

from beaker.