[Fleet] Improve status reporting for Agents about elastic-agent HOT 8 CLOSED

Pinging @elastic/ingest-management (Team:Ingest Management)

from elastic-agent.

I think with the new Fleet Server architecture we should have the Elastic Agent send the status information directly to Fleet Server and from their Fleet Server should append the status information to an index. This removes the need to Elastic Agent to have to write directly to elasticsearch, and it ensures that the Fleet Server has the updated information to perform required adjustments based on the coordinator work that the Fleet Server will need to perform in the future.

from elastic-agent.

I think with the new Fleet Server architecture we should have the Elastic Agent send the status information directly to Fleet Server and from their Fleet Server should append the status information to an index

I agree that we should send the data to Fleet server, so the coordinator can take advantage of that, I am just wondering if the agent should also persist is status (maybe in the logs) for agent observability purpose it will allow to have agent status for standalone agents too.

If we go to the direction where Fleet Server write the agent status, we should probably implement the same in Kibana, I think we can have an index similar to uptime synthetics-agent-status-default, (this could be defined in an elastic agent package ?) Right now is kind of tricky to write from the Kibana checkin route because we do not have the permission to write to ES unless we use the fleet_enroll admin user.

from elastic-agent.

The status should be reported the Fleet Server, but I still think Elastic Agent should persist it to ES directly as log or similar. This means, status reporting also works in standalone case for example.

from elastic-agent.

Agent can fail/be degraded in these cases (* meaning overall agent failure not component related):

* - metadata loading	 				failed to load metadata	

* - recover previous state 				could not recover state, error %+v, skipping... 

* - prepare FS 						filesystem error from os 

gateway - dispatch failed				failed to dispatch actions, error: %s", err

gateway - encode request				fail to encode the checkin request

gateway - decode response				fail to decode checkin response

gateway - request failed(auth,network)			fail to checkin to fleet (more detail in a message)


process - download - req fail				fetching package failed

process - download - !200 respose 			call to '%s' returned unsuccessful status code: %d

process - download - FS error				{details from os}

process - verify - hash not found			hash for '%s' not found

process - verify - hash mismatch			hash does not match

process - verify - asc fetch req fail 			fetching asc file from %s {details}
	
process - verify - asc armored key fail 		read armored key ring

process - verify - asc signature check 			check detached signature


process - install - os error				e.g "TarInstaller: error writing to %s: %v" or "tar contained invalid filename: %q"


process - start						failed to start '{details from go runtime}'

process - config					e.g application stopping

agent server 						e.g "failed to marshal connection information", "failed to write connection information"

upgrade - non upgradeable agent				cannot be upgraded; must be installed with install sub-command and running under control of the systems supervisor

upgrade - copy actions store 				failed to copy action store

upgrade - fs errors					error details from FS

from elastic-agent.

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

from elastic-agent.

@michalpristas do you think that it's still relevant?

from elastic-agent.

Closing this as done, we implement a more detailed status reporting already.
cc @ycombinator

from elastic-agent.