Comments (6)
Issue-Label Bot is automatically applying the label question
to this issue, with a confidence of 0.90. Please mark this comment with
Links: app homepage, dashboard and code for this bot.
from worker.
The only reporting API ifs he /reports API - you can filter it by the worker hostname (note this will be the worker container hostname not the server hostname), device_group & by unix timestamp like described in https://nebula.readthedocs.io/en/latest/api/general/#list-a-filtered-paginated-view-of-the-optional-reports-system
There are two things in the report that will help ensure that the status updated successfully:
- The
apps
field under `current_device_group_config' will give you the current app ID of each app, when it matches the one you get from the /app/app_name you know that the device has pulled the latest needed config from Nebula. - The
apps_containers
are the current container on said device as given at thereport_creation_time
, theid
of each is uniuqe per container so if it changes you know the container crashed and a new one started in it's place
So basically if the current_device_group_config
includes the correct info and the apps_containers
shows a container up of said app you know it's running.
If there's a specific bit of info you think will be helpful to add that can be gleaned off the worker feel free to open a ticket suggesting it to be added (aside from the worker logs, there are many preexisting ways to centralize those already that can be combined with Nebula, like an ELK stack with an app running filebeat as nebula app for example)
from worker.
Hello, @naorlivne Thanks for the Response.
Scenario =>
I have configured the MongoDB on the Host ( For persisting the Database whenever there is a change in Manager or worker ) and I have configured the Manager and reporter connection with the Host MongoDB. All the connections and Updates are working properly.
Now once the update is completed I get a new app id and report creation time also in the http://<my_vps_url>/api/v2/reports
API.
Problem =>
But I saw a behavior that the records
from nebula_reports
section of the nebula
database get deleted automatically after some time. ( Approximately after 2 to 3 hours )
After that Whenever I send a request on this API http://<my_vps_url>/api/v2/reports
I get the following response =>
{
"data": null,
"last_id": null
}
Is it the Intended behavior that records from nebula_reports
section will clean automatically after some time or am I doing something wrong?
Expected Behavior =>
I was expecting the records to be persisted in the database unless cleared or deleted manually.
References =>
I have attached both the docker-composes for the reference purpose =>
docker-compose.yml
for Manager, zookeeper, kafka and reporter
version: '3'
services:
manager:
container_name: manager
hostname: manager
image: nebulaorchestrator/manager
ports:
- "80:80"
restart: unless-stopped
network_mode: host
environment:
MONGO_URL: mongodb://localhost:27017/nebula?authSource=admin
SCHEMA_NAME: nebula
BASIC_AUTH_PASSWORD: nebula
BASIC_AUTH_USER: nebula
AUTH_TOKEN: nebula
zookeeper:
container_name: zookeeper
hostname: zookeeper
image: zookeeper:3.4.13
ports:
- 2181:2181
restart: unless-stopped
environment:
ZOO_MY_ID: 1
kafka:
container_name: kafka
hostname: kafka
image: confluentinc/cp-kafka:5.1.2
ports:
- 9092:9092
restart: unless-stopped
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://<my_vps_url>:9092
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
reporter:
container_name: reporter
hostname: reporter
depends_on:
- kafka
image: nebulaorchestrator/reporter
restart: unless-stopped
network_mode: host
environment:
MONGO_URL: mongodb://localhost:27017/nebula?authSource=admin
SCHEMA_NAME: nebula
BASIC_AUTH_PASSWORD: nebula
BASIC_AUTH_USER: nebula
KAFKA_BOOTSTRAP_SERVERS: localhost:9092
KAFKA_TOPIC: nebula-reports
docker-compose.yml
for worker =>
version: '3'
services:
worker:
container_name: worker
build:
context: .
dockerfile: Dockerfile
volumes:
- /var/run/docker.sock:/var/run/docker.sock
restart: unless-stopped
hostname: worker
environment:
REGISTRY_HOST: <my_registry_url>
REGISTRY_AUTH_USER: <my_registry_user>
REGISTRY_AUTH_PASSWORD: <my_registry_password>
MAX_RESTART_WAIT_IN_SECONDS: 0
NEBULA_MANAGER_AUTH_USER: nebula
NEBULA_MANAGER_AUTH_PASSWORD: nebula
NEBULA_MANAGER_HOST: <my_vsp_url>
NEBULA_MANAGER_PORT: <my_manager_port>
NEBULA_MANAGER_PROTOCOL: http
NEBULA_MANAGER_CHECK_IN_TIME: 5
DEVICE_GROUP: example
KAFKA_BOOTSTRAP_SERVERS: <my_vps_url>:9092
KAFKA_TOPIC: nebula-reports
from worker.
You are correct about the reporting data being purged after some time has passed, by default this is a value of 3600 seconds (1 hour) but can be changed by configuring mongo_report_ttl
on your reporter as described in https://nebula.readthedocs.io/en/latest/config/reporter/
The logic behind it being the default is that Nebula was created to allow a large volume of workers which can quickly create a very large volume of data for the DB to handle & that most people won't care what happened to their workers an hour ago - as new data is still being sent and only data older then an hour is being purged out.
I'm guessing that the reason why your seeing null
data on the DB is that either the worker is powered off so it's not sending new data or that your filtering the report by timestamp so reports newer then a given timestamp aren't shown (or both)?
from worker.
I'm guessing that the reason why your seeing null data on the DB is that either the worker is powered off so it's not sending new data or that your filtering the report by timestamp so reports newer then a given timestamp aren't shown (or both)?
Yes, you're correct I had turned off the device as I wanted to confirm if my understanding related to the purge was correct. As the device was continuously reporting at NEBULA_MANAGER_CHECK_IN_TIME
( i.e. In my case it was reporting in around 5 seconds )
You are correct about the reporting data being purged after some time has passed, by default this is a value of 3600 seconds (1 hour)
Did you mean a report which has a report_creation_time
older than 1 hour then, in that case, that device report will purge?
The logic behind it being the default is that Nebula was created to allow a large volume of workers which can quickly create a very large volume of data for the DB to handle & that most people won't care what happened to their workers an hour ago - as new data is still being sent and only data older then an hour is being purged out.
Yes, you are right due to the continuously reporting there becomes a large volume of the data for the DB to Handle.
I have a doubt What's the intention behind reporting the data continuously to the reporter from the worker?
I was expecting the scenario in which the data will be reported only when the device is updated successfully or update failed. This can help to maintain the whole update history of the device and a large volume of data won't be accumulated in the database. ( This data can optionally be purged after 6 months or a year. )
from worker.
You are correct about the reporting data being purged after some time has passed, by default this is a value of 3600 seconds (1 hour)
Did you mean a report which has a
report_creation_time
older than 1 hour then, in that case, that device report will purge?
No, the purge happens based on the report_insert_date
timestamp so each report will be deleted after mongo_report_ttl
time has passed since it was inserted into MongoDB.
The logic behind it being the default is that Nebula was created to allow a large volume of workers which can quickly create a very large volume of data for the DB to handle & that most people won't care what happened to their workers an hour ago - as new data is still being sent and only data older then an hour is being purged out.
Yes, you are right due to the continuously reporting there becomes a large volume of the data for the DB to Handle.
I have a doubt What's the intention behind reporting the data continuously to the reporter from the worker?
This is mostly done to allow users to have a constant report about the memory\CPU\etc status of the workers
I was expecting the scenario in which the data will be reported only when the device is updated successfully or update failed. This can help to maintain the whole update history of the device and a large volume of data won't be accumulated in the database. ( This data can optionally be purged after 6 months or a year. )
It might be possible to add a filter to the /reports endpoint to return only changes of ID's\etc to assist in that, please open a ticket about that feature request with your exact needs\suggestion.
from worker.
Related Issues (20)
- Connecting to a nebula managr hosted at a non root path HOT 3
- Query about a container running standalone script !! HOT 3
- Dependabot couldn't authenticate with https://pypi.python.org/simple/ HOT 1
- Prune image API not working HOT 3
- Cron jobs do not work per schedule if its frequency is shorter than the check-in time of worker HOT 2
- Build breaking due to unavailable package `freeze` HOT 3
- Add automated unit tests HOT 2
- Move automatic Docker imags build from Docker Hub to Travis-CI HOT 1
- have worker have the option to connect to the managers with a UUID token instead of basic auth HOT 1
- Self update worker container on deployed remote devices HOT 11
- Add cron jobs management support HOT 1
- Auto match version to branch on deployment and have it part of the report generated for the optional reporting system HOT 1
- fix creating new branch from last push not starting travis run due to auto added changelog having the [skip travis] flag on the commit message HOT 1
- Facing issue in running worker on remote device. HOT 6
- Facing issue in creating reporting kafka connection object HOT 5
- Feature Request: Get Update status at the reporter from worker only when update is performed ( failed or successful ). HOT 11
- Facing issue in configuring AWS ECR as a registry using credential helper for Nebula worker. HOT 7
- Consider moving ARM CI/CD build to drone.io HOT 1
- Create containers multiarch manifest file to allow single container multiarch support HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from worker.