Git Product home page Git Product logo

punchclock's Introduction

NuGet Stats Build Code Coverage #yourfirstpr


Punchclock: A library for managing concurrent operations

Punchclock is the low-level scheduling and prioritization library used by Fusillade to orchestrate pending concurrent operations.

What even does that mean?

Ok, so you've got a shiny mobile phone app and you've got async/await. Awesome! It's so easy to issue network requests, why not do it all the time? After your users one-:star2: you for your app being slow, you discover that you're issuing way too many requests at the same time.

Then, you try to manage issuing less requests by hand, and it becomes a spaghetti mess as different parts of your app reach into each other to try to figure out who's doing what. Let's figure out a better way.

So many words, gimme the examples

var wc = new WebClient();
var opQueue = new OperationQueue(2 /*at a time*/);

// Download a bunch of images
var foo = opQueue.Enqueue(1, 
    () => wc.DownloadFile("https://example.com/foo.jpg", "foo.jpg"));
var bar = opQueue.Enqueue(1, 
    () => wc.DownloadFile("https://example.com/bar.jpg", "bar.jpg"));
var baz = opQueue.Enqueue(1, 
    () => wc.DownloadFile("https://example.com/baz.jpg", "baz.jpg"));
var bamf = opQueue.Enqueue(1, 
    () => wc.DownloadFile("https://example.com/bamf.jpg", "bamf.jpg"));

// We'll be downloading the images two at a time, even though we started 
// them all at once
await Task.WaitAll(foo, bar, baz, bamf);

Now, in a completely different part of your app, if you need something right away, you can specify it via the priority:

// This file is super important, we don't care if it cuts in line in front
// of some images or other stuff
var wc = new WebClient();
await opQueue.Enqueue(10 /* It's Important */, 
    () => wc.DownloadFileTaskAsync("http://example.com/cool.txt", "./cool.txt"));

What else can this library do

  • Cancellation via CancellationTokens or via Observables
  • Ensure certain operations don't run concurrently via a key
  • Queue pause / resume

Contribute

Punchclock is developed under an OSI-approved open source license, making it freely usable and distributable, even for commercial use. Because of our Open Collective model for funding and transparency, we are able to funnel support and funds through to our contributors and community. We ❤ the people who are involved in this project, and we’d love to have you on board, especially if you are just getting started or have never contributed to open-source before.

So here's to you, lovely person who wants to join us — this is how you can support us:

punchclock's People

Contributors

anaisbetts avatar arhell avatar chrispulman avatar dependabot-preview[bot] avatar dependabot[bot] avatar elipriaulx avatar elonon avatar flagbug avatar ghuntley avatar glennawatson avatar idisposable avatar pedroreys avatar prasannavl avatar reactiveuibot avatar renovate[bot] avatar rlittlesii avatar theangrybyrd avatar worldbeater avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

punchclock's Issues

Resume Queue Implemented

Has ResumeQueue been implemented yet?

readme says it supports Pause/Resume:

  • I am able to see and call the .PauseQueue();
  • but don't see how to "un pause"
  • I tried calling .PauseQueue(); twice just to see what would happen and that doesn't "toggle" it :)

(Thank you for all the awesome libraries btw)

Single Operation

First off, Simple, yet very elegant library!

The string keys preventing concurrent operations of the same key, is a very nice feature. I'd very much like to see a feature where there can be a restriction to also ignore operations with the same key if one already exists in the queue, equal to, or higher than the given priority.

This could be immensely useful in repeated operations, where you don't keep adding similar ops to the queue if one already exists.

Thoughts?

Performance niggle

In PriorityQueue.DequeueAll, it calls down to Dequeue in a loop which means that RemoveAt(0) gets called repeatedly. This means that RemoveAt logic that reallocates the _items array fires repeatedly every time the current _size drops below the 1/4 of then current capacity. If we're doing this against a really large queue, it could end up copying the _items array elements repeatedly (and newing the items array a few times). It might be awesome to pass a emptying flag down to a private overload ofDequeue, which could forward that flag to RemoveAt (either an private overload or just defaulted false value on the current one). When the emptying flag is true , then we skip the _items array reallocation and copy until the _size falls to below the default capacity (currently 16).

https://github.com/paulcbetts/punchclock/blob/master/Punchclock%2FPriorityQueue.cs

I would be happy to submit a PR, but don't want to waste your time if this does seem to be an issue for you.

Allow maximumConcurrent value to be changed on an OperationQueue instance

Hi Paul,

I have a scenario where I want to change the maximumConcurrent value.

For example, when I know I'm going to have an intense sync happening I want to increase the amount of concurrent http requests. And, after I'm done, I want to lower it back.

Right now, as a consumer of punchclock, the only place where I can change that value is in the constructor of the queue instance.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Pending Status Checks

These updates await pending status checks. To force their creation now, click the checkbox below.

  • chore(deps): update dependency roslynator.analyzers to v4.12.0

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

github-actions
.github/workflows/ci-build.yml
.github/workflows/ci-buildAndRelease.yml
.github/workflows/lock.yml
  • dessant/lock-threads v5
nuget
src/Directory.build.props
  • Roslynator.Analyzers 4.11.0
  • stylecop.analyzers 1.2.0-beta.556
  • Nerdbank.GitVersioning 3.6.133
  • Microsoft.SourceLink.GitHub 8.0.0
  • coverlet.msbuild 6.0.2
  • PublicApiGenerator 11.1.0
  • DiffEngine 15.3.0
  • FluentAssertions 6.12.0
  • Xunit.StaFact 1.1.11
  • Xunit.SkippableFact 1.4.13
  • xunit.runner.visualstudio 2.5.4
  • xunit.runner.console 2.6.2
  • xunit 2.6.2
  • Microsoft.NET.Test.Sdk 17.9.0
src/Punchclock.Tests/Punchclock.Tests.csproj
  • Verify.Xunit 22.5.0
  • PublicApiGenerator 11.1.0
src/Punchclock/Punchclock.csproj
src/global.json
  • dotnet-sdk 8.0.10
  • MSBuild.Sdk.Extras 3.0.44

  • Check this box to trigger a request for Renovate to run again on this repository

feature: extend the OperationQueue by adding support for randomness

Is your feature request related to a problem? Please describe.
I want to add jittering (noise) to my http-request-queues for a specific endpoint.
Discovered Fusillade and it would be awesome to have this on OperationQueue.

Describe the solution you'd like
Trying to improve the performance of a system by adding randomness, as well as
trying to avoid limiting of some endpoint that requires noise.

Describe alternatives you've considered
Set maximumConcurrent to 1 and use:

Observable
    .Return(Unit.Default)
    .Delay(TimeSpan.FromMilliseconds(new Random().Next(min, max)))

Limitation to System.Reactive < 4.0.0 holds me from upgrading other packages (Akavache)

Note: for support questions, please ask on StackOverflow: https://stackoverflow.com/questions/tagged/punchclock . This repository's issues are reserved for feature requests and bug reports.

Do you want to request a feature or report a bug?
bug

What is the current behavior?
Limitation to System.Reactive < 4.0.0

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
Ugrade a project with System.Reactive, Fusillade, Punchclock & Akavache.
I need to upgrade Akavache, but this needs System.Reactive >= 4.0.0

What is the expected behavior?
Being able to upgrade my packages to the latest version.

What is the motivation / use case for changing the behavior?
I can't apply fixes in new versions of other packages, because PunchClock (& Fusillade) blocks this.

Which versions of PunchClock, and which platform / OS are affected by this issue? Did this work in previous versions of PunchClock?
v2.1.0. is affected. v2.0.0 does not mention the limitation, but I'm not sure wether it's safe to downgrade to this version in combination with System.Reactive > 4.0.0

Other information (e.g. stacktraces, related issues, suggestions how to fix)
No suggestions, because I don't know why the limitation is needed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.