Telegram notifications quickly run into rate limit about fredy HOT 10 CLOSED

Hi @sven-simonsen ,

the issue I see with this is how the architecture of Fredy is built.
Currently, an interval is triggering Fredy every x minutes (configurable). Then, all providers are being queried in a loop. The result of each provider is routed to the configured notification adapter. Each adapter is returning a promise which resolved when all notifications have been sent. Only then, the next provider is being queried.

Now when I delay sending messages to a specific provider like telegram, I run into multiple issues, as the whole pipeline is delayed. In the worst case, if somebody configures a crawl interval of let's say 2 minutes, the delays will add up until we eat up all of the system resources.

Because of this, I currently have no real idea, how to fix this. Any clues?

from fredy.

Sounds like there are actually 3 steps to what fredy does and they are currently happening in the same task loop.
Step 1 is collecting all the relevant Anzeigen with name and link.
Step 2 is filtering for duplicates.
Step 3 is sending out notifications on all adapter channels.

I think to make some of the improvements like filtering over all providers together and timing message sends to comply with rate limits, a structural change would be necessary where steps 1 and 3 no longer run in the same loop.

For example saving a bit more information about every Anzeige would allow filtering to happen for every batch of new Anzeigen found before adding to the db.
And then step 3 could run on its own loop depending on adapter rate limits and just work through the db entries and mark the ones where notifications have been sent.

from fredy.

Yeah, that was something I was playing around with in my mind for a little longer already.

As you might imagine, this is quite a big change in the architecture of Fredy. It requires not only structural changes, but also changes in the database which in turn means migrating existing data.

In the long run, I am planning to move away from lowdb to SQLite. Also I am planning to run the steps mentioned by you in sequences. Each of which returning a promise, resolving when done..

However, I am not sure when I will have the time to do those changes, as I just became a father ;)

from fredy.

@sven-simonsen rather than trying to fix telegram, I am thinking about ditching the integration in favor of pushover.net.

If has a great free Tier and it has almost no rate limit.

What is your take on it?

from fredy.

Seems like users would need to install an app exclusively for use with fredy. I would personally not use that adapter. I would rather go back to email then and change my workflow with my partner to send emails back and forth discussing the flats.
To me the advantage of Telegram is that I can send the flat announcements to a channel and then have a decent space and tools for collaboration.

from fredy.

Ok got it

from fredy.

Hey. Just a little status update;

Splitting the jobs is way harder than is seems. I think instead of dramatically changing the architecture of Fredy, I will try to change the telegram adapter...

from fredy.

@sven-simonsen do you have a link where i can read more about the telegram rate limit? All I find is the rate limit to not send more than 1 message per second

from fredy.

https://core.telegram.org/bots/faq#my-bot-is-hitting-limits-how-do-i-avoid-this
This is probably where I got my info.
Note the part that says: "Also note that your bot will not be able to send more than 20 messages per minute to the same group."
Some months ago I claimed 10 was the limit and looking back I have been running into the limit less, so maybe they changed it?
I also changed my use of fredy to have less parallel searches, that might also have helped.

from fredy.

I'm going to close this for now, as I cannot reproduce it. Maybe they changed their api, however I was able to send 50+ messages without any issue

from fredy.