This repository enhances task management in Python by facilitating the execution of interconnected operations across various configurations.

Traditional methods, such as executing scripts sequentially within a single bash file, often fail to optimize computational resources.

This repository introduces a more practical approach to managing computational resources across operations.

We offer a queue management system for handling each stage of linked operations. Each stage has its queue, which is automatically updated every N seconds to reflect task completions. Completed tasks move to the next stage's queue.

Using a queue for each operation rather than each configuration, this scheme allows an efficient flow of operations and optimizes resource use by reducing idle time.

While Apache Airflow is an excellent option, our direct queue management for each operation offers a more straightforward, more intuitive solution that independently optimizes each process flow. Benefits include:

• Resource Utilization Optimization: Independent queues for each stage reduce idle time, enhancing computational efficiency.
• Flexible Task Management: Easier adjustment of task priorities, dynamic resource allocation, and quick retries.
• Simplicity and Intuitiveness: I think it is designed to be straightforward and user-friendly.

FIRST, try to run python task_demo.py to see how the features work.

To customize and run your tasks using this framework, follow these steps:

1. Customization Examples: To understand how to customize tasks for your specific needs, refer to the DemoScripter and DemoDetector classes in the task_demo.py file. These examples demonstrate how to execute and monitor custom operations.

2. Define Your Tasks: Implement your tasks using the Scripter and Detector classes to define how each task is executed and how to detect its completion.

3. Configure Task Execution: Utilize the Info class to set up your task list, the maximum number of parallel operations per task, and other execution parameters.

4. Run Your Tasks: Execute your tasks by running the python task_demo.py. Monitor the execution through the logs to ensure tasks are progressing as expected.

5. Troubleshooting: In case of errors or unexpected behavior, consult the log files generated by the execution manager. Adjust your Scripter or Detector implementations to resolve issues.

6. Further Customization: Explore advanced features by modifying the ExecutionManager and QueueManager for more sophisticated task scheduling and error handling strategies.

Contributions to enhance the functionality or documentation of this framework are welcome. For suggestions or bug reports, please submit pull requests or raise issues on our GitHub repository.

This project is licensed under the MIT License - see the LICENSE file for details.