About crawlab.json about crawlab-sdk HOT 6 OPEN

Multi-spider support is on the way. Please follow this issue crawlab-team/crawlab#1190

from crawlab-sdk.

Multi-spider support is on the way. Please follow this issue crawlab-team/crawlab#1190

Will schedules deployments also be included?

from crawlab-sdk.

Would you elaborate a bit?

from crawlab-sdk.

Would you elaborate a bit?

In practice, I need to create dozens of new cronjobs along with a new crawler spider,
crawler spider upload can be done from the command line, so can cronjobs be done too?
then I can write these commands to CICD.

So I would like to add a new param schedules to the crawlab.json to publish and manage cronjobs, like this

    {
      "path": "packages/py_spider",
      "exclude_path": ".venv",
      "name": "py spiders",
      "description": "py spiders",
      "cmd": "python",
      "schedules": [
        {
          "name": "py spider 1 cron",
          "cron": "* 1 * * *",
          "command": "python py_spider_1/main.py",
          "param": "",
          "mode": "random",
          "description": "py spider 1 cron",
          "enabled": true
        },
        {
          "name": "py spider 2 cron",
          "cron": "* 2 * * *",
          "command": "python py_spider_2/main.py",
          "param": "",
          "mode": "random",
          "description": "py spider 2 cron",
          "enabled": true
        },
       ...
      ]
    }

what do you think of these ideas, or do you have any other better suggestions?

from crawlab-sdk.

Would you elaborate a bit?

@tikazyq What do you think about the above

from crawlab-sdk.

I think that's a good idea but it might take some time to implement it. Let's create a new enhancement issue in the main repo https://github.com/crawlab-team/crawlab

from crawlab-sdk.