API throttling versus bulk data reporting about node-zendesk HOT 12 CLOSED

A long time ago I thought about this, I think I chose to leave it up to the user how to deal with this, I wanted to solve for this but at the time i could rarely hit this limit... I think it'd be good to have a helper function like this in request all, I'm still figuring out how to go about satisfying both camps... I would request that in your PR you allow for an option to toggle 429 auto retry, and definitely some sort of UI indicator that its waiting (although I'm concerned that may add some level of indirection as console output will not be solely output) I suppose no UI indication would work but I worry some people might think their program is hanging. I'll lean on you for how you want to approach, but definitely add option to turn off 413 auto read if your going to make a PR.

from node-zendesk.

For the high-urgency report I needed to create, I just throttled outside your library using a ...customized? version of this library: https://github.com/brianloveswords/throttle-function

I'm guessing you've never had to query the API for all of an organization's tickets for a number of months, including comments. Our helpdesk is very high-volume, typically 200 new tickets a day, and merely running this report sends over 3500 requests to the API.

Any time there's a requestAll call, the request loop should be throttled. Since we're dealing with asynchronous stuff, throttling shouldn't be too difficult to inject, really, so long as we can maintain the state of the throttled request loop.

I'm not certain how enabling the 429 auto-retry via toggle option is beneficial to users of this library's requestAll methods. I would recommend, and may implement in my fork, to emit an event regarding 'continuing to next in requestAll' and 'throttling request' to allow implementors to monitor the progress of their larger requests. I haven't actually changed a line of this library, though.

from node-zendesk.

oops, that's my personal account. this is my corporate account. same person.

from node-zendesk.

@iconecd: I'd gladly accept an example if you could throw a simple one together using that throttle-function package, I think it'd be useful to other users. No need to change the library.

from node-zendesk.

I'll have to fork the throttle package and publish my version first. I'm still curious why requestAll shouldn't automatically manage API throttle calls. They're standardized across clients and methods, with specific temporary exceptions. Why shouldn't this be part of the library?

from node-zendesk.

Oh I'm sorry I misunderstood, You make a good case for it, If you want to modify requestAll I'm down for that, I just don't know what you know to implement this.

from node-zendesk.

Great, I'll work on it when I can. Thanks!

from node-zendesk.

@iconecd thanks for that feature!

from node-zendesk.

hey, thanks for this library!

from node-zendesk.

This is a great addition! I'm running into the same issue as well. I'm hitting 429s with every call since I'm trying to pull all of our tickets. I know this feature was implemented last spring. Is there any documentation for this yet?

from node-zendesk.

Sorry, I really should have added documentation with this feature. I'll try to submit a new PR. In the meantime, you can pass an object named 'throttle' to createClient, like so:

var zdapi = require( 'node-zendesk' );
var throttleconf = {
  window: 75,
  limit: 150
};

[...]

var zd = zdapi.createClient({
  username...
  token...
  ...
  throttle: throttleconf
});

from node-zendesk.

That's great! Thanks!

from node-zendesk.