Comments (3)
Do you have some deployment that might share/change the kong_prefix directory across different data_plane instances?
No, there's no configuration changes between deployments.
And do you have steps to help us to reproduce this issue locally?
So, the scenario is like I told on the bug report. I have one control plane, and two data planes in my production environment running on ECS. The autoscaling configuration(based on cpu and memory consumption) is very sensitive, so it scales out very drastically(3-to-5).
And when this happens, some or all of the new containers have one or more errors like the ones that I added, and does not load the configuration from the control plane. The containers appear to be in a deadlock.
I don't know if this can be reproduced locally. I might throw a hand, and maybe you can try using some Kubernetes stuff like k6, adding very little resources to the dataplane node, and forcing it to scale in the cluster and see if these issues happen. The problem happens on trying to add new data plane nodes to an existent environment.
One thing that might stabilize my services, is changing the health check from the /status
to /status/ready
route, what would avoid the data planes without control plane communication to serve traffic. I'm going to deploy this change today, and hope that it stabilizes everything.
from kong.
@lays147 Thanks for your report. Seems the file of LMDB is corrupted. Now Kong is using LMDB to store entities on the data_plane. Do you have some deployment that might share/change the kong_prefix
directory across different data_plane instances?
from kong.
And do you have steps to help us to reproduce this issue locally?
from kong.
Related Issues (20)
- TLS SNI Route not work HOT 7
- Kong info, notice, inspect logs are all getting logged as error in GCP(google cloud platform) HOT 3
- Cannot use kong.db HOT 1
- http-log plugin: Host header not including port HOT 3
- Every time request localhost:8001/metrics, kong-cp-kong-pod will prompt a license-related error HOT 3
- go plugin error, worker-events: event callback failed; source=plugin_server, event=reset_instance HOT 1
- Error: attempt to index local 'ssl' (a nil value) for Kong >= 3.6.0 when KONG_PG_SSL=on HOT 6
- Handling two JWT signing keys for same iss HOT 1
- Documentation Improvement - Clarify Plugin Compatibility with Kong Versions](https://github.com/Kong/kong/issues/9999 HOT 1
- Error "error loading plugin schemas: on plugin <my-plugin>: <my-plugin> plugin is enabled but not installed" with Python plugin for kong HOT 6
- Where is kong 3.7.0 container image ? Can't find it on Dockerhub HOT 2
- Hola
- Inconsistent Rate Limiting Behavior Under High Load HOT 2
- No ability in plugin schema to have a record with dynamic fields HOT 2
- Changelog md no longer maintained? HOT 2
- Odd Kong startup errors post 2.8.x to 3.7.0 upgrade HOT 5
- Kong can't match routes correctly with the same hosts, paths, and different protocols in the route. HOT 4
- AWS Lambda plugin in Gateway 2.8.4 missing aws_imds_protocol_version support HOT 1
- https over mtls is not working for upstream HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kong.