Git Product home page Git Product logo

asyncflow's Introduction

AsyncFlow Library

Integrate asynchronous job flows with ease in your .NET applications. AsyncFlow is designed for web applications that use a pattern where users call an invoke API, receive a job ID, and can then track and retrieve the job's results using separate endpoints.

Features

  • Asynchronous Job Invocation: Easily handle long-running jobs by providing an immediate response with a job ID.
  • Status Tracking: Allow users to track the status of their job.
  • Flexible Result Retrieval: Once the job is complete, users can retrieve the result.
  • Integration with Hangfire: Make use of the robust background processing library, Hangfire, to handle job execution.
  • Extensible Cache Options: Store job results in-memory or use a distributed cache.

Getting Started

Installation

Use the package manager NuGet to install AsyncFlow packages:

dotnet add package AsyncFlow
dotnet add package AsyncFlow.Queues.Generator

Configuration

  1. Setup Hangfire (or any supported background job processor):

    builder.Services.AddHangfire(x => x.UseMemoryStorage());
    builder.Services.AddHangfireServer(options => options.Queues = Flows.All));
  2. Add AsyncFlow with Desired Cache:

    For in-memory cache:

    builder.Services.AddAsyncFlow(options => options.UseMemoryCache());

    For distributed cache:

    builder.Services.AddAsyncFlow(options => options.UseDistributedCache(yourDistributedCacheInstance));
  3. Map Endpoints:

    app.MapFlow<YourJobClass, YourRequestType, YourResponseType>("endpointName");

Certainly! Let's replace the C# record descriptions with JSON examples for the responses:

The Auto created API Endpoints

When you integrate the AsyncFlow library into your application, the following API endpoints are provided for you:

  1. Enqueue Endpoint:

    • Path: /[flowName]
    • HTTP Method: POST
    • Purpose: To initiate the async flow process.
    • Response:
      {
        "RequestId": "12345-abcd",
        "DateTime": "2023-08-14T15:30:45Z"
      }
  2. Status Endpoint:

    • Path: /[flowName]/{jobId}/status
    • HTTP Method: GET
    • Purpose: To check the status of a previously enqueued request.
      • Response:
        {
          "RequestId": "12345-abcd",
          "Status": "Processing",
          "CreatedAt": "2023-08-14T15:30:45Z",
          "progressData": {
              "progress": "Generating Data",
              "percentage": 90
          }
        }
  3. Result Endpoint:

    • Path: /[flowName]/{jobId}/result
    • HTTP Method: GET
    • Purpose: To retrieve the result of the job.
    • Response: JSON that represents the result object.
  4. Delete Endpoint:

    • Path: /[flowName]/{jobId}
    • HTTP Method: DELETE
    • Purpose: To delete the result of the job.
    • Response: No content (empty response).
  5. Error Endpoint:

  • Path: /[flowName]/{jobId}/error
  • HTTP Method: GET
  • Purpose: To retrieve error details if a job fails.
  • Response:
  "JobId": "unique-job-id",
  "Error": {
  "Type": "ExceptionType",
  "Message": "Detailed error message",
  "StackTrace": "Stack trace details..."
     }
  }

Remember to replace the placeholders like "Details about the Result endpoint, including a JSON example." with the actual details for those endpoints if they provide responses similar to the ones you've described. If not, describe them as needed.

Also, make sure to guide your users on how to replace the [flowName] placeholder appropriately.

Custom Endpoint Configuration

You can customize the behavior of the AsyncFlow endpoints using the AsyncFlowEndpointConfigurator:

var configurator = new YourConfiguratorSubClass();
app.MapFlow<YourJobClass, YourRequestType, YourResponseType>("endpointName", configurator);

Usage Example

Define a job:

public class GenerateDataJob : IAsyncFlow<GenerateDataRequest, GenerateDataResponse>
{
    public async Task<GenerateDataResponse> ProcessAsync(GenerateDataRequest request ,IProgress<ProgressData> progress , CancellationToken cancellationToken)
    {
        // Your logic here
    }
}

Invoke it:

app.MapFlow<GenerateDataJob, GenerateDataRequest, GenerateDataResponse>("data");

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the MIT License.

asyncflow's People

Contributors

thefo2sh avatar

Stargazers

Gabriel Schmith avatar John Katsiotis avatar Eric Loveland avatar Hoseini Peyrov avatar  avatar Tsung-Wei Huang avatar Giorgos Kal avatar  avatar Darren Kattan avatar  avatar  avatar  avatar O0oo0O avatar RHQYZ avatar  avatar Jason avatar  avatar  avatar Davron avatar Mattia M. avatar fikrihakim avatar Sanam avatar Anurag Mishra avatar  avatar Jo avatar Christopher Neff avatar  avatar Jatin Patel avatar Dillan Cagnetta avatar Cl0uds1 avatar Abdulsamad Osunlana avatar  avatar 龚杰 avatar Manrike Villalobos Báez avatar John Xiong avatar Ali Gholipour avatar  avatar  avatar Dan Shoubridge avatar Türhan Yıldırım avatar Isaac Ojeda avatar  avatar send2vinnie avatar  avatar Phillip Amir Esguerra avatar Tim Kersey avatar  avatar

Watchers

send2vinnie avatar John Katsiotis avatar  avatar  avatar  avatar

asyncflow's Issues

Enhancement Request: Default Hangfire Queue Name as Class Name for Unattributed Flows*

Description:
Building upon the proposal to use separate Hangfire queues per flow type, we've further refined the concept to provide even more flexibility and convenience. In scenarios where users do not specify a queue using the proposed QueueAttribute, we suggest implementing a default behavior. Specifically, when no QueueAttribute is applied to a flow class, the name of the class itself will be used as the Hangfire queue name.

Feature Details:

  • Default Queue Behavior: If a flow class is not adorned with the QueueAttribute, the name of the class will be utilized as the Hangfire queue name.

  • Use Cases:

    • Prioritization: By using the class name as the queue name, users can implicitly prioritize certain flows by manipulating the order of class names.
    • Logical Grouping: Flows that are not assigned a specific queue can still be logically grouped based on their class names.
    • Simplified Configuration: Users can leverage this behavior without explicitly applying the QueueAttribute, simplifying the configuration process.
  • Example:
    If a flow class named CriticalFlow is not attributed with QueueAttribute, the default queue name for this flow will be CriticalFlow.

  • Implementation:

    • Modify the MapFlow method to automatically determine the Hangfire queue name based on the class name if no QueueAttribute is applied.
    • Ensure that this behavior aligns with the broader goal of improving job queue management.
  • Benefits:

    • Streamlined Configuration: Users who do not need to explicitly set queue names can rely on the default behavior without additional steps.
    • Logical Grouping: Even without attribute configuration, flows can still be logically grouped by queue based on their class names.
    • Compatibility: This behavior complements both attributed and non-attributed flows, providing a unified approach.

Your feedback on this refined enhancement proposal is greatly appreciated. Please share your thoughts and considerations regarding the suggested default behavior. If there are any potential concerns or considerations that should be addressed during implementation, kindly let us know. Your input will play a crucial role in shaping this feature for the benefit of AsyncFlow library users.

.NET Standard 2.0 Support

Is this something that seems feasible and you'd be interested in? Some of our software is still on .NET Framework and your library looks like it would be incredibly helpful! I'd be happy to open a PR for this if so!

Support for CancellationToken in IAsyncFlow

Title:

Support for CancellationToken in IAsyncFlow

Description:

Problem:

Currently, our IAsyncFlow interface does not support CancellationToken. This means that if a delete request comes in while ProcessAsync is still running, the process won't be cancelled immediately.

Detailed Explanation:

When processing long-running jobs, it's crucial to have the ability to cancel them if needed. Not having cancellation support can lead to wasted resources, especially if the result of the job is no longer required (e.g., if a delete request has been received).

Proposed Solution:

Introduce a CancellationToken parameter to the ProcessAsync method in the IAsyncFlow interface. This would enable any implementing class to respect cancellation requests.

Task<TResult> ProcessAsync(TRequest request, CancellationToken cancellationToken);

Additionally, when handling the Delete request on the API side, signal the cancellation token to attempt to halt the ongoing operation.

Impact:

Existing implementations of the IAsyncFlow interface would need to be updated to handle the new parameter. Documentation would need to be updated as well to guide users on how to use the cancellation token properly.

Errors Endpoint

New Endpoint Proposal: Fetching Job Errors

Issue Type: Feature Request

Description:
As we continue to evolve the AsyncFlow library, it's become apparent that there's a potential gap in our error-handling capabilities. Currently, while users can track the status of their jobs, there isn't a straightforward way to fetch detailed error information in case a job fails. This can be invaluable for both developers and end-users to understand what might have gone wrong.

Proposal:
Introduce a new endpoint:
/{flowName}/{jobId}/error

This endpoint will return detailed error information if the corresponding job encounters a failure.

Response Structure (proposed):

{
    "JobId": "unique-job-id",
    "Error": {
        "Type": "ExceptionType",
        "Message": "Detailed error message",
        "StackTrace": "Stack trace details..."
    }
}

Customization:
In line with the ethos of flexibility that AsyncFlow offers, we should also provide users with the ability to customize the error response. This can be achieved by incorporating a Func<Exception, IResult> within the AsyncFlowEndpointConfigurator, allowing users to modify the default error output or even introduce additional logging mechanisms.

Benefits:

  • Provides clearer insights into job failures.
  • Enhances user experience by offering detailed feedback.
  • Allows users to tailor error outputs, making it adaptable for various applications.

Implementation Steps (tentative):

  1. Design the endpoint structure.
  2. Integrate it with the job execution flow to capture and relay errors.
  3. Implement the customization mechanism through AsyncFlowEndpointConfigurator.
  4. Update documentation and provide examples.

Introduce IProgress<ProgressData> Support in IAsyncFlow

Title:

Introduce IProgress Support in IAsyncFlow

Description:

Problem:

For long-running tasks executed by IAsyncFlow, there's currently no mechanism to report and track progress during the ProcessAsync method execution. This means users can't gauge how much of the job has been completed and might be left waiting without any indication of progress.

Detailed Explanation:

To provide users with more feedback on the progress of their jobs, we should introduce support for progress reporting within the IAsyncFlow interface. This will allow the job to provide periodic updates about how much of the task has been accomplished.

Proposed Solution:

  1. Modify the IAsyncFlow interface to include an IProgress<ProgressData> parameter for the ProcessAsync method.
Task<TResult> ProcessAsync(TRequest request, IProgress<ProgressData> progress, CancellationToken cancellationToken);
  1. The ProgressData should be defined as:
public record ProgressData(string Progress, double Value);
  1. In the API, when the status call is made for a job, the latest progress data should be returned along with the current job status.

Impact:

This enhancement will improve user experience by providing real-time feedback on job progress. Existing implementations of IAsyncFlow will need to be updated to cater to the new parameter. Additionally, we'll need to store progress data for each job, which might require changes to our storage mechanism.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.