Elastic Agent non-Windows host - Agent doesn't finish installing until after re-starting the service about elastic-agent HOT 25 CLOSED

@mdelapenya do you know if we can select the OS language in the e2e testing framework?

No, we cannot, as we have a Windows 2019 AMI pre-baked with Packer to be used as the target instance for the E2E. I've just commented in elastic/e2e-testing#2805 (comment) that I'd suggest having manual tests on the language support

from elastic-agent.

As @blakerouse is out maybe @michalpristas could give us pointers about non-English Windows host configuration.

from elastic-agent.

@amolnater-qasource looking at the logs it seems that Elasticsearch is unreachable on its 9200 port.
Moreover another issue related to this particular filesystem not being ready: D:: GetDiskFreeSpaceEx

Does it ring a bell?

from elastic-agent.

Pinging @elastic/agent (Team:Agent)

from elastic-agent.

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

from elastic-agent.

There were some fixes related to i18n. Can this be tested again?

from elastic-agent.

@dikshachauhan-qasource @amolnater-qasource thanks for the help, can you re-test and post back?

from elastic-agent.

Hi @EricDavisX

Due to Vsphere issue today, we were unable to validate this. However, we will reattempt it again and will share observation accordingly.

Thanks
QAS

from elastic-agent.

Hi @EricDavisX

We have re-attempted to validate the current behavior of agent with endpoint on non-english VM and found same observations as shared above.

• Agent is stuck in updating state until user manually restarts the agent service.

Build details:
Version : 7.15.1 GA
Artifact link: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.15.1-windows-x86_64.zip
BUILD 44185
COMMIT c1aa1ef8dc38b511ce4a647fe92ca0348aadd834

Screenshot:

Please let me know if info is required from our end.

Thanks
QAS

from elastic-agent.

I realize now the testing we did was after the expected fixes - I am not surprised here. @andresrc @jlind23 if you think this is expected world-wide usage, we could prioritize the research? At least Michal has Endgame cluster vSphere access to a non-English host we can spin up for him if desired / needed. and we can get more folks access as needed to spread the work-load, let me know in email.

from elastic-agent.

@EricDavisX did you check that the build number was with the related fixes? If yes, then it seems that we missed some things. @andresrc would you mind giving more context here? English hosts works fine?

from elastic-agent.

@jlind23 hi - glad to confirm, the 7.15.1 build is intended to have the fixes. Perhaps if we review the backport we'd find a problem? original pr is elastic/beats#26665

This is a testing hole we were lucky to find (Alvaro found it for us!) and after much review, we found code that needed to reference a translated-to-different-language word, in this case the folder on the host is called 'administradores' instead of 'administrators' in Windows so the code was updated. It was believed to be working, but QA reported this immediately after the fixes went into full testing and so we can follow up again. There is a pair of hosts in the Endgame vSphere cluster that are prepped for testing if we want them. IT helpdesk can grant access to you or others as needed.

from elastic-agent.

@EricDavisX saw another thread in which blake was saying that maybe the version was not downloaded properly. May it be related to this? #148

from elastic-agent.

Thanks. Assigning to @amolnater-qasource to follow up and see if we can reproduce, and if so if we can isolate the downloading failure or prove it is something else.

from elastic-agent.

Hi @EricDavisX

We have revalidated this issue by installing elastic-agent on French VM and found this issue still reproducible.

• Agent is stuck in updating state until user manually restarts the agent service.

Kibana/Build details:

7.15.1 Kibana cloud environment.
Build: 44185
Commit: c1aa1ef8dc38b511ce4a647fe92ca0348aadd834

We observed below [error] log line for metricbeat under logs tab:
13:52:33.532 elastic_agent.metricbeat [elastic_agent.metricbeat][error] Error dialing dial tcp 127.0.0.1:9200: connectex: Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée.
(English translation: No connection could be made because the target computer expressly refused it.)

Debug level logs:
logs.zip

Note:
Elastic-agent artifact was downloaded completely without any errors.

cc: @jlind23
Please let me know if anything else is required.
Thanks

from elastic-agent.

Port 9200 is used for elasticsearch communication isn't it? @amolnater-qasource is your elasticsearch pod running?

from elastic-agent.

Looks like a configuration issue as metricbeat cannot communicate with elasticsearch. Did you configure the output settings correctly in the Fleet UI?

from elastic-agent.

In the description it is cited as a 7.14 BC4 cloud stack, I'm betting the test didn't include changing anything in the fleet output settings. I'm unassigning Amol, seems like it needs an engineering owner

from elastic-agent.

Hi @jlind23

@amolnater-qasource is your elasticsearch pod running?

We were testing upon 7.15.1 cloud production environment and even agents installed Healthy on other OS's other than non-English Windows host.
So yes it should be running as the issue is only observed for non-english Windows host.

@blakerouse

Looks like a configuration issue as metricbeat cannot communicate with elasticsearch. Did you configure the output settings correctly in the Fleet UI?

Do we need to do any separate settings to install agent on non-English Windows host?
As we followed the steps we usually follow to install the elastic-agent as per guide: Install a Fleet-managed Elastic Agent

• Downloaded and extracted the elastic-agent artifact.
• From the extracted folder we executed Install command available on Fleet UI.

Please let us know if anything else is required from our end.
Thanks

from elastic-agent.

@amolnater-qasource i'm digging hold issues. Is this something still relevant?

from elastic-agent.

Hi @jlind23
We have revalidated this issue on latest 8.0 Snapshot Kibana cloud environment.
We have attempted this on 03 different languages Windows VM consisting:

• French Windows 10
• Spanish Windows 10
• Korean Windows 10

No issues are observed on Spanish and Korean VM

• Agent installed directly Healthy and no need of restarting service is observed.

However this issue is still reproducible on French VM.

• French VM elastic-agent stuck in updating state and required a manual restart of service.

Screenshot:

Build details:
BUILD: 49040
COMMIT: 155e06787e48de9a8de4345d86a826e95edf32ec
Artifact Link: https://snapshots.elastic.co/8.0.0-129ef708/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-windows-x86_64.zip

Please let us know if anything else is required from our end.
Thanks

from elastic-agent.

Hi @jlind23
We have revalidated this issue on latest 8.4 Snapshot Kibana cloud environment and found it still reproducible.

• French VM elastic-agent stuck in updating state and required a manual service restart to get Healthy.

Screenshots:

Build details:
BUILD: 54160
COMMIT: b509d2466e88bf6c4386d8dd5fe89b5c8a54a97f
Artifact Link: https://snapshots.elastic.co/8.4.0-0384b1d2/downloads/beats/elastic-agent/elastic-agent-8.4.0-SNAPSHOT-windows-x86_64.zip

Thanks

from elastic-agent.

@amolnater-qasource do we have logs saying why the agent is stuck?

@mdelapenya do you know if we can select the OS language in the e2e testing framework?

from elastic-agent.

Hi @jlind23
Please find below attached logs for agent installed on French Windows10 VM.

Logs:
elastic-agent-diagnostics-2022-07-07T10-16-38Z-00.zip

Further we have observed below error logs for elastic-agent.metricbeat dataset.

15:37:50.574
elastic_agent.metricbeat
[elastic_agent.metricbeat][error] Error fetching data for metricset system.filesystem: error getting filesystem usage for D:\: GetDiskFreeSpaceEx failed: Le périphérique n’est pas prêt.
15:37:52.198
elastic_agent.metricbeat
[elastic_agent.metricbeat][error] Failed to connect to backoff(elasticsearch(http://127.0.0.1:9200)): Get "http://127.0.0.1:9200": dial tcp 127.0.0.1:9200: connectex: Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée.
15:37:54.269
elastic_agent.metricbeat
[elastic_agent.metricbeat][error] Error dialing dial tcp 127.0.0.1:9200: connectex: Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée.

Screenshot:

Internet on this VM is working fine.

Please let us know if anything else is required from our end.
Thanks

from elastic-agent.

Hi @jlind23

We have revalidated this issue on 8.6.0 release kibana cloud-production environment and found it fixed now.

Host:
French Windows 10

Observations:

• French VM elastic-agent gets installed Healthy and required no manual restart.

Build details:
BUILD: 58852
COMMIT: d3a625ef4a6e611a5b3233a1ce5cbe8ef429eb47
Artifact Link: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.6.0-windows-x86_64.zip

Screen Recording:

Hence we are closing this issue and marking it as QA:Validated.

Thanks

from elastic-agent.