Analytics for Task not Working about cvat HOT 7 CLOSED

Steps to reproduce? Environment? Logs?

Please, fill the issue template completely, otherwise it will be closed.

from cvat.

hi @bsekachev

Logs:

cvat_server | 2024-06-03 06:49:42,322 DEBG 'uvicorn-0' stdout output:
cvat_server | INFO: 192.168.80.4:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified
cvat_server |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,392 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,391] DEBUG rq.worker: Dequeued job analytics:calculate-report-task-8 from analytics_reports
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,392 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,392] INFO rq.worker: analytics_reports: _check_analytics_report(cvat_job_id=None, cvat_project_id=None, cvat_task_id=8) (analytics:calculate-report-task-8)
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,393 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,393] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.
cvat_worker_analytics_reports |
cvat_server | 2024-06-03 06:49:42,396 DEBG 'uvicorn-0' stdout output:
cvat_server | INFO: 172.31.3.211:0 - "POST /api/analytics/reports?org=qXRLN HTTP/1.0" 202 Accepted
cvat_server |
traefik | {"ClientAddr":"172.31.3.211:47872","ClientHost":"172.31.3.211","ClientPort":"47872","ClientUsername":"-","DownstreamContentSize":45,"DownstreamStatus":202,"Duration":177796073,"OriginContentSize":45,"OriginDuration":177747592,"OriginStatus":202,"Overhead":48481,"RequestAddr":"lcannotation.qure.ai","RequestContentSize":27,"RequestCount":28,"RequestHost":"lcannotation.qure.ai","RequestMethod":"POST","RequestPath":"/api/analytics/reports?org=qXRLN","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"cvat@docker","ServiceAddr":"192.168.80.14:8080","ServiceName":"cvat@docker","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"192.168.80.14:8080","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-06-03T06:49:42.219504813Z","StartUTC":"2024-06-03T06:49:42.219504813Z","entryPointName":"web","level":"info","msg":"","time":"2024-06-03T06:49:42Z"}
cvat_worker_analytics_reports | 2024-06-03 06:49:42,410 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,409] DEBUG rq.worker: Started Job Registry set.
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,410 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,410] DEBUG rq.worker: Preparing for execution of Job ID analytics:calculate-report-task-8
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,411 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,410] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 90 seconds.
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,414 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,414] DEBUG rq.worker: Job preparation finished.
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,414 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,414] DEBUG rq.worker: Performing Job...
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,533 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,533] DEBUG rq.worker: Job analytics:calculate-report-task-8 raised an exception.
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,537 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,536] DEBUG rq.worker: Handling failed execution of job analytics:calculate-report-task-8
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,548 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,548] DEBUG rq.worker: Handling exception for analytics:calculate-report-task-8.
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,549 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,548] ERROR rq.worker: [Job analytics:calculate-report-task-8]: exception raised while executing (_check_analytics_report)
cvat_worker_analytics_reports | Traceback (most recent call last):
cvat_worker_analytics_reports | File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 1431, in perform_job
cvat_worker_analytics_reports | rv = job.perform()
cvat_worker_analytics_reports | File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 1280, in perform
cvat_worker_analytics_reports | self._result = self._execute()
cvat_worker_analytics_reports | File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 1317, in _execute
cvat_worker_analytics_reports | result = self.func(*self.args, **self.kwargs)
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/create.py", line 202, in _check_analytics_report
cvat_worker_analytics_reports | db_report, job_reports = cls()._compute_report_for_task(
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/create.py", line 399, in _compute_report_for_task
cvat_worker_analytics_reports | self._compute_report_for_job(db_job, job_report, data_extractors)
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/create.py", line 362, in _compute_report_for_job
cvat_worker_analytics_reports | primary_statistics = {
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/create.py", line 363, in
cvat_worker_analytics_reports | pm.key(): self._get_statistics_entry(pm) for pm in primary_metrics
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/create.py", line 330, in _get_statistics_entry
cvat_worker_analytics_reports | **{"data_series": statistics_object.calculate()},
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/primary_metrics/objects.py", line 63, in calculate
cvat_worker_analytics_reports | rows = self._data_extractor.extract_for_job(self._db_obj.id, {"scopes": scopes})
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/primary_metrics/base.py", line 38, in extract_for_job
cvat_worker_analytics_reports | self._rows = self._make_clickhouse_query(
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/primary_metrics/base.py", line 34, in _make_clickhouse_query
cvat_worker_analytics_reports | return make_clickhouse_query(query=self._query, parameters=parameters)
cvat_worker_analytics_reports | File "/home/django/cvat/apps/analytics_report/report/primary_metrics/utils.py", line 19, in make_clickhouse_query
cvat_worker_analytics_reports | result = client.query(query, parameters=parameters)
cvat_worker_analytics_reports | File "/opt/venv/lib/python3.10/site-packages/clickhouse_connect/driver/client.py", line 193, in query
cvat_worker_analytics_reports | return self._query_with_context(query_context)
cvat_worker_analytics_reports | File "/opt/venv/lib/python3.10/site-packages/clickhouse_connect/driver/httpclient.py", line 209, in _query_with_context
cvat_worker_analytics_reports | response = self._raw_request(body,
cvat_worker_analytics_reports | File "/opt/venv/lib/python3.10/site-packages/clickhouse_connect/driver/httpclient.py", line 428, in _raw_request
cvat_worker_analytics_reports | self._error_handler(response)
cvat_worker_analytics_reports | File "/opt/venv/lib/python3.10/site-packages/clickhouse_connect/driver/httpclient.py", line 353, in _error_handler
cvat_worker_analytics_reports | raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
cvat_worker_analytics_reports | clickhouse_connect.driver.exceptions.DatabaseError: :HTTPDriver for http://clickhouse:8123 returned response code 404)
cvat_worker_analytics_reports | Code: 60. DB::Exception: Table cvat.events does not exist. (UNKNOWN_TABLE) (version 23.11.5.29 (official build))
cvat_worker_analytics_reports |
cvat_worker_analytics_reports |
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,550 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,549] DEBUG rq.worker: Invoking exception handler <function rq_exception_handler at 0x7f2fde0b45e0>
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,550 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,550] DEBUG rq.worker: Invoking exception handler <function handle_rq_exception at 0x7f2fe1af9000>
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,583 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,583] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,584 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,583] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,585 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,585] DEBUG rq.worker: *** Listening on analytics_reports...
cvat_worker_analytics_reports |
cvat_worker_analytics_reports | 2024-06-03 06:49:42,588 DEBG 'rqworker-analytics-reports-0' stderr output:
cvat_worker_analytics_reports | [2024-06-03 06:49:42,586] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.
cvat_worker_analytics_reports | [2024-06-03 06:49:42,587] DEBUG rq.worker: Dequeueing jobs on queues analytics_reports and timeout 405
cvat_worker_analytics_reports |
traefik | {"ClientAddr":"172.31.45.39:28708","ClientHost":"172.31.45.39","ClientPort":"28708","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":29150,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":29150,"RequestAddr":"172.31.8.69:8082","RequestContentSize":0,"RequestCount":30,"RequestHost":"172.31.8.69","RequestMethod":"GET","RequestPath":"/auth/login","RequestPort":"8082","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"StartLocal":"2024-06-03T06:49:43.500304894Z","StartUTC":"2024-06-03T06:49:43.500304894Z","entryPointName":"web","level":"info","msg":"","time":"2024-06-03T06:49:43Z"}
cvat_vector | 2024-06-03T06:49:43.563241Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=2}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] has been rate limited 1 times.
cvat_vector | 2024-06-03T06:49:43.563268Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=2}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 404 Not Found" internal_log_rate_limit=true
cvat_vector | 2024-06-03T06:49:43.563285Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=2}: vector::sinks::util::sink: Response failed. response=Response { status: 404, version: HTTP/1.1, headers: {"date": "Mon, 03 Jun 2024 06:49:43 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "x-clickhouse-server-display-name": "4130b44cb6bf", "transfer-encoding": "chunked", "x-clickhouse-query-id": "435d5279-4dcc-496d-86e2-b6cbba05625b", "x-clickhouse-timezone": "UTC", "x-clickhouse-exception-code": "60", "keep-alive": "timeout=10", "x-clickhouse-summary": "{"read_rows":"0","read_bytes":"0","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"0","result_bytes":"0","elapsed_ns":"894085"}"}, body: b"Code: 60. DB::Exception: Table cvat.events does not exist. (UNKNOWN_TABLE) (version 23.11.5.29 (official build))\n" }
cvat_vector | 2024-06-03T06:49:43.563317Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=2}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] has been rate limited 1 times.
cvat_vector | 2024-06-03T06:49:43.563322Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=2}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error="Response failed." request_id=2 error_type="request_failed" stage="sending" internal_log_rate_limit=true
cvat_vector | 2024-06-03T06:49:43.563339Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been rate limited 1 times.
cvat_vector | 2024-06-03T06:49:43.563344Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
cvat_server | 2024-06-03 06:49:45,332 DEBG 'uvicorn-0' stderr output:
cvat_server | [2024-06-03 06:49:45,332] WARNING django.request: Bad Request: /api/analytics/reports
cvat_server |
cvat_server | 2024-06-03 06:49:45,333 DEBG 'uvicorn-0' stdout output:
cvat_server | INFO: 172.31.3.211:0 - "POST /api/analytics/reports?org=qXRLN&rq_id=analytics:calculate-report-task-8 HTTP/1.0" 400 Bad Request
cvat_server |
traefik | {"ClientAddr":"172.31.3.211:47872","ClientHost":"172.31.3.211","ClientPort":"47872","ClientUsername":"-","DownstreamContentSize":933,"DownstreamStatus":400,"Duration":163061168,"OriginContentSize":933,"OriginDuration":162803564,"OriginStatus":400,"Overhead":257604,"RequestAddr":"lcannotation.qure.ai","RequestContentSize":27,"RequestCount":31,"RequestHost":"lcannotation.qure.ai","RequestMethod":"POST","RequestPath":"/api/analytics/reports?org=qXRLN\u0026rq_id=analytics:calculate-report-task-8","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"cvat@docker","ServiceAddr":"192.168.80.14:8080","ServiceName":"cvat@docker","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"192.168.80.14:8080","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-06-03T06:49:45.170784966Z","StartUTC":"2024-06-03T06:49:45.170784966Z","entryPointName":"web","level":"info","msg":"","time":"2024-06-03T06:49:45Z"}
traefik | {"ClientAddr":"172.31.31.205:60994","ClientHost":"172.31.31.205","ClientPort":"60994","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":31441,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":31441,"RequestAddr":"172.31.8.69:8082","RequestContentSize":0,"RequestCount":32,"RequestHost":"172.31.8.69","RequestMethod":"GET","RequestPath":"/auth/login","RequestPort":"8082","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"StartLocal":"2024-06-03T06:49:45.445720766Z","StartUTC":"2024-06-03T06:49:45.445720766Z","entryPointName":"web","level":"info","msg":"","time":"2024-06-03T06:49:45Z"}
cvat_vector | 2024-06-03T06:49:46.816343Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=3}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] is being rate limited.
cvat_vector | 2024-06-03T06:49:46.816383Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=3}: vector::sinks::util::sink: Response failed. response=Response { status: 404, version: HTTP/1.1, headers: {"date": "Mon, 03 Jun 2024 06:49:46 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "x-clickhouse-server-display-name": "4130b44cb6bf", "transfer-encoding": "chunked", "x-clickhouse-query-id": "bed62b82-90de-4d89-b46e-ba0d60b0e0ae", "x-clickhouse-timezone": "UTC", "x-clickhouse-exception-code": "60", "keep-alive": "timeout=10", "x-clickhouse-summary": "{"read_rows":"0","read_bytes":"0","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"0","result_bytes":"0","elapsed_ns":"1183206"}"}, body: b"Code: 60. DB::Exception: Table cvat.events does not exist. (UNKNOWN_TABLE) (version 23.11.5.29 (official build))\n" }
cvat_vector | 2024-06-03T06:49:46.816415Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=3}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being rate limited.
cvat_vector | 2024-06-03T06:49:46.816428Z ERROR sink{component_kind="sink" component_id=clickhouse component_type=clickhouse component_name=clickhouse}:request{request_id=3}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being rate limited.
cvat_server | 2024-06-03 06:49:53,084 DEBG 'uvicorn-1' stdout output:
cvat_server | INFO: 192.168.80.4:0 - "GET /api/auth/rules HTTP/1.0" 200 OK

from cvat.

@bsekachev I have added the steps and env + error logs from the server.

from cvat.

There is a problem with your setup. Clickhouse is not available.

File "/opt/venv/lib/python3.10/site-packages/clickhouse_connect/driver/httpclient.py", line 353, in _error_handler
cvat_worker_analytics_reports | raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
cvat_worker_analytics_reports | clickhouse_connect.driver.exceptions.DatabaseError: :HTTPDriver for http://clickhouse:8123/ returned response code 404)
cvat_worker_analytics_reports | Code: 60. DB::Exception: Table cvat.events does not exist. (UNKNOWN_TABLE) (version 23.11.5.29 (official build))

from cvat.

services:
  cvat_redis_inmem:
    container_name: cvat_redis_inmem
    image: redis:7.2.3-alpine
    restart: always
    command: [
      "redis-server",
      "--save", "60", "100",
      "--appendonly", "yes",
    ]
    volumes:
      - cvat_inmem_db:/data
    networks:
      - cvat

  cvat_redis_ondisk:
    container_name: cvat_redis_ondisk
    image: apache/kvrocks:2.7.0
    restart: always
    command: [
      "--dir", "/var/lib/kvrocks/data"
    ]
    init: true
    volumes:
      - cvat_cache_db:/var/lib/kvrocks/data
    networks:
      - cvat

  cvat_server:
    container_name: cvat_server
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      <<: *backend-deps
      cvat_opa:
        condition:
          service_started
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      DJANGO_MODWSGI_EXTRA_ARGS: ''
      ALLOWED_HOSTS: '*'
      ADAPTIVE_AUTO_ANNOTATION: 'false'
      NUMPROCS: 2
      CVAT_ANALYTICS: 1
      CVAT_BASE_URL: lcannotation.qure.ai
      ONE_RUNNING_JOB_IN_QUEUE_PER_USER:
    command: init run server
    labels:
      - traefik.enable=true
      - traefik.http.services.cvat.loadbalancer.server.port=8080
      - traefik.http.routers.cvat.rule=Host(`${CVAT_HOST:-localhost}`) &&
        PathPrefix(`/api/`, `/static/`, `/admin`, `/documentation/`, `/django-rq`)
      - traefik.http.routers.cvat.entrypoints=web
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      cvat:
        aliases:
          - cvat-server

  cvat_utils:
    container_name: cvat_utils
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on: *backend-deps
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      CVAT_REDIS_INMEM_PASSWORD: ''
      NUMPROCS: 1
    command: run utils
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_import:
    container_name: cvat_worker_import
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on: *backend-deps
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      NUMPROCS: 2
    command: run worker.import
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_export:
    container_name: cvat_worker_export
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on: *backend-deps
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      NUMPROCS: 2
    command: run worker.export
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_annotation:
    container_name: cvat_worker_annotation
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on: *backend-deps
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      NUMPROCS: 1
    command: run worker.annotation
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_webhooks:
    container_name: cvat_worker_webhooks
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on: *backend-deps
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      NUMPROCS: 1
    command: run worker.webhooks
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_quality_reports:
    container_name: cvat_worker_quality_reports
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on: *backend-deps
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      NUMPROCS: 1
    command: run worker.quality_reports
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_analytics_reports:
    container_name: cvat_worker_analytics_reports
    image: qureai/cvat_server:${CVAT_VERSION:-dev}
    restart: always
    depends_on: *backend-deps
    env_file:
      - ./cvat.env
    environment:
      <<: *backend-env
      NUMPROCS: 2
    command: run worker.analytics_reports
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_ui:
    container_name: cvat_ui
    image: cvat/ui:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_server
    labels:
      - traefik.enable=true
      - traefik.http.services.cvat-ui.loadbalancer.server.port=80
      - traefik.http.routers.cvat-ui.rule=Host(`${CVAT_HOST:-localhost}`)
      - traefik.http.routers.cvat-ui.entrypoints=web
    networks:
      - cvat

  traefik:
    image: traefik:v2.10
    container_name: traefik
    restart: always
    ports:
      - 8082:8082
      - 8090:8090
    environment:
      CVAT_HOST: ${CVAT_HOST:-localhost}
      DJANGO_LOG_VIEWER_HOST: grafana
      DJANGO_LOG_VIEWER_PORT: 3000

      TRAEFIK_ACCESSLOG_FORMAT: json
      TRAEFIK_LOG_FORMAT: json
      TRAEFIK_ENTRYPOINTS_web_ADDRESS: :8082
      TRAEFIK_PROVIDERS_DOCKER_EXPOSEDBYDEFAULT: "false"
      TRAEFIK_PROVIDERS_DOCKER_NETWORK: cvat
      TRAEFIK_PROVIDERS_FILE_DIRECTORY: /etc/traefik/rules
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./grafana_conf.yml:/etc/traefik/rules/grafana_conf.yml:ro
    networks:
      - cvat
    logging:
      driver: "json-file"
      options:
        max-size: 100m
        max-file: "10"

  cvat_opa:
    container_name: cvat_opa
    image: openpolicyagent/opa:0.63.0
    restart: always
    networks:
      cvat:
        aliases:
          - opa
    command:
      - run
      - --server
      - --log-level=error
      - --set=services.cvat.url=http://cvat-server:8080
      - --set=bundles.cvat.service=cvat
      - --set=bundles.cvat.resource=/api/auth/rules
      - --set=bundles.cvat.polling.min_delay_seconds=5
      - --set=bundles.cvat.polling.max_delay_seconds=15

  cvat_clickhouse:
    container_name: cvat_clickhouse
    image: clickhouse/clickhouse-server:23.11-alpine
    restart: always
    environment:
      - CLICKHOUSE_DB=cvat
      - CLICKHOUSE_USER=user
      - CLICKHOUSE_PASSWORD=user
    networks:
      cvat:
        aliases:
          - clickhouse
    volumes:
      - ./init.sh:/docker-entrypoint-initdb.d/init.sh:ro
      - cvat_events_db:/var/lib/clickhouse/

  cvat_vector:
    container_name: cvat_vector
    image: timberio/vector:0.26.0-alpine
    restart: always
    depends_on:
      - cvat_clickhouse
    environment:
      - CLICKHOUSE_DB=cvat
      - CLICKHOUSE_USER=user
      - CLICKHOUSE_PASSWORD=user
      - CLICKHOUSE_HOST=clickhouse
    networks:
      cvat:
        aliases:
          - vector
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro

  cvat_grafana:
    image: grafana/grafana-oss:10.1.2
    container_name: cvat_grafana
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_BASIC_ENABLED=false
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
      - GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=grafana-clickhouse-datasource
      - GF_SERVER_ROOT_URL=http://${CVAT_HOST:-localhost}/analytics
      - GF_INSTALL_PLUGINS=https://github.com/grafana/clickhouse-datasource/releases/download/v2.0.7/grafana-clickhouse-datasource-2.0.7.linux_amd64.zip;grafana-clickhouse-datasource
      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/all_events.json
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
          - name: 'ClickHouse'
            type: 'grafana-clickhouse-datasource'
            isDefault: true
            jsonData:
              defaultDatabase: cvat
              port: 9000
              server: clickhouse
              username: user
              tlsSkipVerify: false
            secureJsonData:
              password: user
            editable: true
        EOF
        mkdir -p /etc/grafana/provisioning/dashboards
        cat <<EOF > /etc/grafana/provisioning/dashboards/dashboard.yaml
        apiVersion: 1
        providers:
          - name: cvat-logs
            type: file
            updateIntervalSeconds: 30
            options:
              path:  /var/lib/grafana/dashboards
              foldersFromFilesStructure: true
        EOF
        exec /run.sh
    networks:
      cvat:
        aliases:
          - grafana

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_inmem_db:
  cvat_events_db:
  cvat_cache_db:

networks:
  cvat:

this is my compose file. I see cvat clickhouse running and below are its logs:

ClickHouse Database directory appears to contain a database; Skipping initialization
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
Processing configuration file '/etc/clickhouse-server/users.xml'.
Merging configuration file '/etc/clickhouse-server/users.d/default-user.xml'.
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.```

from cvat.

@bsekachev this is screenshot for the services running and on which ports they are running

from cvat.

Closing this issue. I was able to solve it by regenerating the clickhouse DB.

from cvat.