[bug, cosmetic] unhandled message in error state: after Farmware exits normally about farmbot_os HOT 8 CLOSED

Hello @RickCarlino

This one is like a fly racing room to room in a house . . it won't go away :-)

FBOS v10.1.0-rc1

from farmbot_os.

Interesting find @jsimmonds2 thanks for bringing it to our attention. I will take a look next time I'm in there.

from farmbot_os.

@RickCarlino
This warning has quietly vanished in v9.2.2 . . will close as I can't reproduce it by either running a Farmware from a Sequence or just using the RUN button on the Farmware web app page.

from farmbot_os.

@jsimmonds2 That sounds right. I made some fixes for the users that were hitting "Can't perform X in state Y" errors, and it very well could be that the fix in the firmware handler corrected unnoticed problems elsewhere.

from farmbot_os.

Hmmm .. it's back . . I'll try and hunt down more relevant detail / evidence !

Using v9.2.2 in dev env plus custom farmbot_system_rpi3 which supports crash.dump file creation.

from farmbot_os.

@jsimmonds2 Thanks for investigating this.

RE: Nerves System updates - It's still on my TODO list (See! I even took a screenshot of the Trello card! 😅) but I've been putting it off until things are a bit more stable. Right now ~10% of customers are hitting failed OTAs and we still have not isolated the root cause, making me hesitant to change the platform until a root cause is isolated.

I will keep you posted.

from farmbot_os.

• making me hesitant to change the platform until a root cause is isolated.

Absolutely ! For me there's no rush at all to have crash dumps officially enabled, since I managed to build a local farmbot_system_rpi3 with that flag.

What I'm doing with that is trying to come up with a change that permits more FarmEvents without using so much heap.

The crashes I'm getting with 8_000 or more FarmEvents are usually due to the FarmbotCeleryScript.Scheduler process Garbing . . likely due to the huge amount of heap churn caused by the large amount of stuff changing in state.
I'm thinking an ordered_set inside an ets table is the way to store these things (?)

from farmbot_os.

Ah interesting. I am not too familiar with those internals yet, but your theory makes sense. Scheduling FarmEvents has historically been very resource intensive.

from farmbot_os.