taskforcesh / taskforce-connector Goto Github PK

View Code? Open in Web Editor NEW

35.0 35.0 15.0 1.1 MB

Service to allow local queues be managed in taskforce.sh

License: MIT License

JavaScript 12.77% TypeScript 85.98% Dockerfile 1.25%

taskforce-connector's People

Contributors

Stargazers

Watchers

Forkers

ritter ex7r3me kyen99 dubiousdavid naphta niek fairhopeweb suryatmodulus create-inc pvp-com seaskydl step-finance dcoffey-zengenti ftntming hernas

taskforce-connector's Issues

Promoting a non delayed job crashes the connector

It seems we are not properly handling an exception:

/usr/local/lib/node_modules/taskforce-connector/node_modules/bull/lib/job.js:356
throw new Error(‘Job ’ + jobId + ' is not in a delayed state’);
^
Error: Job 760534a3-bbc6-484b-b288-5559f83326f21 is not in a delayed state
at /usr/local/lib/node_modules/taskforce-connector/node_modules/bull/lib/job.js:356:15

Moderate vulnerabilities - dependabot

Hi,
We're using BullMQ with Taskforce and I've noticed we're continuously getting vulnerability alerts coming from this package. Checked you're PRs and looks like they match with the vulnerabilities reported by node/npm.
Is this still maintained and is there any chance to fix them in a future version?
Thanks!

--nodes stopped working in newest version

Seems like if passing list of nodes with comma delimited list of nodes, it is not working anymore except when used as ENV.

I've added this patch to make it work for my use case:

diff --git a/node_modules/taskforce-connector/dist/queue-factory.js b/node_modules/taskforce-connector/dist/queue-factory.js
index 9052c4b..4ef1623 100644
--- a/node_modules/taskforce-connector/dist/queue-factory.js
+++ b/node_modules/taskforce-connector/dist/queue-factory.js
@@ -75,6 +75,17 @@ exports.getRedisInfo = getRedisInfo;
 function getRedisClient(redisOpts, type, clusterNodes) {
     if (!redisClients[type]) {
         if (clusterNodes && clusterNodes.length) {
+            process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
+            Object.assign(redisOpts, {
+                redisOptions: {
+                    tls: {
+                        cert: Buffer.from(process.env.REDIS_CLUSTER_TLS ?? '', 'base64').toString('ascii')
+                    },
+                }
+            });
+            if (clusterNodes.constructor === String) {
+                clusterNodes = clusterNodes.split(",")
+            }
             redisClients[type] = new ioredis_1.Redis.Cluster(clusterNodes, redisOpts);
         }
         else {

How do we set the user used to connect to redis?

All the other parameters are editable, but we can't specify the username. We need to do that in order to connect to heroku redis as username can change (and we can't decide its value). Is there a way to achieve that?

Authorization failed

When using in library mode I get the following error:

nodejs                 | WebSocket: Authorization failed, please check that you are using the correct token from your account page
nodejs                 | WebSocket:  retry in 5000ms
nodejs                 | WebSocket: closed [Arguments] { '0': 1006 }

in docker mode I get:

taskforce-connector_1  | Redis: redis connection error  getaddrinfo ENOTFOUND redis
taskforce-connector_1  | Redis: Reconnecting in 256 ms

In either case I know that the token and the redis address are correct.

able to connect to only specified queues

During the connection, the connector automatically connects to all the queues in the redis db unlike the other monitoring libraries like bull-board where you need to specify the Queue names. This has resulted the older queues (not required to be shown in the UI dashboard) to be included in the dashboard.
If there is a way to fix this can you please tell me the way?

Docs on using a docker container gives error

The docs says to have a docker-compose.yml file like so

  taskforce-connector:
    build: https://github.com/taskforcesh/taskforce-connector.git#master
    ...

I found that format gave me the following error:

❯ docker compose up taskforce-connector
[+] Building 0.0s (1/1) FINISHED
 => CACHED [internal] load git source https://github.com/taskforcesh/taskforce-connector.git#master                                                                          0.0s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount784378217/https:/github.com/taskforcesh/taskforce-connector.git#master/Dockerfile: no such file or directory

I found that changing the format of docker-compose.yml to following fixed the issue:

  taskforce-connector:
    build: https://raw.githubusercontent.com/taskforcesh/taskforce-connector/master/Dockerfile
    ...

Avoid a forced upgrade

In order to avoid production problems after new releases on production we should not force the user to upgrade: taskforcesh/issues#11

Official docker image

Hello. Where I can find the official docker image for taskforce-connector? I don't want to build it by myself every time this repo updates.

Select database in Redis connection

I'm using a single Redis server for multiple applications that use bull and Taskforce, so I've needed the option to select the database in Redis connection object.
I'll submit a pull request on this issue

Multiple shards issue

Hello, I'm trying to use the connector for AWS elasticache Redis (cluster mode enabled) with 3 shards/primary nodes and 3 replicas for every shard (9 nodes), but I cannot see all my queues in the dashboard connection,
When I refresh the connector, I see different queues every time, it seems like the connector is getting the queues only for 1 shard at the same time

doesn't this connector support multiple shards?

Unable to view connection on taskforce.sh/connections

I am running the following command

taskforce -n "myconnection" -t "xxx" -h "yyy" --passwd "zzz"

The CLI logs are as follows

Taskforce Connector v1.15.0 - (c) 2017-2021 Taskforce.sh Inc.
WebSocket: opening connection to Taskforce.sh
WebSocket: opened connection to Taskforce.sh
WebSocket: received {"id":"xxxx","data":{"res":"connections","cmd":"getConnection"}}
Redis: connected to redis server
WebSocket:  retry in 5000ms
WebSocket: closed [Arguments] { '0': 1006 }
WebSocket: reconnecting...
WebSocket: Unexpected server response: 502
WebSocket:  retry in 5000ms
WebSocket: closed [Arguments] { '0': 1006 }
WebSocket: reconnecting...
WebSocket: opened connection to Taskforce.sh
WebSocket: received {"id":"xxxxxx","data":{"res":"connections","cmd":"getConnection"}}
Redis: connected to redis server
WebSocket:  retry in 5000ms
WebSocket: closed [Arguments] { '0': 1006 }
WebSocket: reconnecting...
WebSocket: Unexpected server response: 502
WebSocket:  retry in 5000ms
WebSocket: closed [Arguments] { '0': 1006 }
WebSocket: reconnecting...
WebSocket: Unexpected server response: 502
WebSocket:  retry in 5000ms
WebSocket: closed [Arguments] { '0': 1006 }
WebSocket: reconnecting...

Add a test suite

Keeps making Redis connections

The number of connections to Redis keeps growing over time and the connections are not closed.

add support for specifying redis uris

Sometimes is useful to specify an URI instead of a combination or host/port/password.

Redis DB doesn't appear in dashboard after connecting via connector.

When I run

taskforce -n "My Redis" -t my-token --port 6555 --passwd my-password

the connector outputs

Taskforce Connector v1.24.5 - (c) 2017-2023 Taskforce.sh Inc.
WebSocket: opening connection to Taskforce.sh (wss://api.taskforce.sh)
WebSocket: opened connection to Taskforce.sh
WebSocket: received {"id":"100a738e-9206-46b0-ab79-423d713e07c0","data":{"res":"connections","cmd":"getConnection"}}
Redis: connected to redis server

however the connection never appears in my dashboard [see pic]

If I regenerate my token, I get a connection error so I know I'm using the correct account.

[Feature Request] Levelled Log Methods

Amazing product guys! The Docker configuration has made this such an easy integration ❤️.

One thing that would be super useful from the Docker perspective is swapping out the console.log calls with a levelled log package like winston or pino etc, to allow users to pass a LOG_LEVEL environment variable in with the docker-compose.yml config to mute debug level noise, or set to warn / error events only, etc...

Obviously not a critical bug or anything, so feel free to just close if that's not on the roadmap.😁

A restart of the connector sometimes requires refreshing connection in Taskforce

In some situations, for example if the FE is not running and the connector is restarted, the connector will stop working until a refresh is performed manually from the FE.

Docker image broken

The docker image is not working, because of PM2 issues:

PM2 error: TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (23)

I worked around by ditching PM2 completely and using this in my docker compose file:

command: taskforce --name "${TASKFORCE_CONNECTION}" --passwd "${REDIS_PASSWORD}"

When using a non existing team no error is given.

It looks as if the connector is working but no connection will be shown on the dashboard.

Metrics not working

Bull version: 4.8.1
Taskforce-connector version: 1.17.0

Instance information:

docker compose with the following:

  taskforce-connector:
    build: https://github.com/taskforcesh/taskforce-connector.git#master
    container_name: taskforce-connector
    env_file:
      - .env.docker.development
    environment:
      TASKFORCE_CONNECTION: "local-dev-redis"
      REDIS_HOST: '127.0.0.1'
    command: taskforce --name local-dev-redis
    depends_on:
      - redis

Observed issue: no metrics available

Jobs correctly show in other tabs

Redis has metrics data

127.0.0.1:6379> lrange "bull:SensorQueue:metrics:failed:data" 0 -1
 1) "12"
 2) "11"
 3) "14"
 4) "10"
 5) "12"
 6) "12"
 7) "11"
 8) "12"
 9) "12"
10) "11"
11) "13"
12) "10"
13) "12"
14) "11"

127.0.0.1:6379> keys bull:SensorQueue:metrics:*
1) "bull:SensorQueue:metrics:failed"
2) "bull:SensorQueue:metrics:failed:data"

What further steps should I take to diagnose this, and any ideas about a fix?