After upgrade of Opensearch to 2.14 Graylog starts throwing exceptions - Unable to perform search query: OpenSearch exception [type=concurrent_modification_exception, reason=null]. about graylog2-server HOT 15 CLOSED

Investigation showed, that IMHO this is a bug in OpenSearch. opensearch-project/OpenSearch#14032

from graylog2-server.

@janheise thank you for the fast investigation and for excellent reporting of this to the respective project

from graylog2-server.

Hello, thanks for raising this

Can I ask how busy the cluster is (how often events are running, typically)?

Any possibility you could attach graylog's server.log file?

from graylog2-server.

@clickbg Looking at the failing code, we should be raising a related system notification event.
Do you see one on the system / alerts & events tab?

from graylog2-server.

Hi, thanks for the fast reply.

Sure I am attaching the server.log and the relevant Opensearch logs - I just have removed any personally identifying domains or IPs the rest should be as it was logged. I am also attaching screenshot of the Event Definitions and the errors in System / Alerts & Events - @patrickmann Yes there are a lot of them - basically one for each alert definition that was ran after the upgrade.

In terms of busy, the system (its a single system) isn't busy at all - log ingestion is below 100MB per day on average - around 50MB per day. There are 8 event definitions in total, 6 of them are running every 5 minutes with a backlog search of 6 minutes, 1 is running every 30 minutes with a backlog search of 31 minutes and the last one is running every 2 days (which was configured by mistake, it should be daily) but I left it as it for now. The misconfigured one hasn't ran since the upgrade which was done on 2024-06-01 14:33 EEST / 11:33 UTC.

Thanks again!

server.log
opensearch-logs.tar.gz

from graylog2-server.

@clickbg Great - can you share the (redacted) details view of one of those System Notification Events?

from graylog2-server.

Sure, I am attaching the details of most recent event that has generated an error. Most of them similar and search for a simple pattern and group by either source or an Grok extracted field - IP (%{IP})

from graylog2-server.

@clickbg I meant details of a System Notification event instance, not the definition itself. I'm hoping it will contain the actual query error.
Here's an example (of a different even type):

from graylog2-server.

Ah sorry my mistake, I am attaching the details for the related Events and Alerts

from graylog2-server.

Just noting that https://go2docs.graylog.org/current/downloading_and_installing_graylog/installing_graylog.html says the max OpenSearch version supported with Graylog 6.0.x is currently 2.13.x (and the integration tests only seem to be testing up to 2.12.x).

from graylog2-server.

@coffee-squirrel yes unfortunately OpenSearch doesn't support downgrading and they treat 2.13 -> 2.14 as a minor upgrade, at least from package management perspective. Unlike Graylog where you have to purposefully change the repo, OpenSearch just upgrades automatically...

root@apollo:~# apt policy opensearch
opensearch:
  Installed: 2.14.0
  Candidate: 2.14.0
  Version table:
 *** 2.14.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
        100 /var/lib/dpkg/status
     2.13.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.12.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.11.1 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.11.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.10.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.9.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.8.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.7.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.6.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
     2.5.0 500
        500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages

So anyone who does regular Ubuntu/Debian/RH/SLES upgrades will inevitably end up with 2.14 without a way to revert without having to delete everything and start from scratch. One way to avoid this is to bundle the correct version of OpenSearch in the Graylog repo - that way you control which version we get but it will add extra work in maintaining an extra package. Another way is to upgrade the docs and advise users to put a hold on the OpenSearch package (apt-mark hold) but that risks the reverse problem - people running too old of a version of OpenSearch which isn't compatible with Graylog anymore. External dependencies are always a pain.

from graylog2-server.

Same here on Graylog 5.2.7 / OpenSearch 2.14.0.

from graylog2-server.

fixed for 2.15, see opensearch-project/opensearch-build#4681

from graylog2-server.

I can confirm that the error is gone after upgrading to 2.15.0

from graylog2-server.

I can also confirm that after upgrading to 2.15 on 25th this month no new alerts for this bug have been generated.

from graylog2-server.