Timeout errors from Browserless API interrupt runs about dust HOT 15 CLOSED

Yeah this one is annoying as it is handled as a run error and the whole run is interrupted preventing the rest of inputs to complete.

We can extract that error and return a JSON to represent it. That means apps would have to gracefully handle that state which requires more logic.

from dust.

Any way to adjust timeout length?

from dust.

I think these pages are just going to timeout whatever value you put. But to answer your question, not at the time, need to add that as a parameter to the browserless block.

from dust.

30s is already... a lot.

from dust.

Worth noting that it's 30s not just for downloading the page, but for downloading all the scripts, images, etc., and running all the JavaScript until the page settles down. I also suspect but am not sure that:

• Browserless uses pretty underpowered machines
• The browsers start with nothing in cache
• Low usage accounts may have low numbers of simultaneous browsers spun up until they hit timeouts

from dust.

Agreed. But some pages just fail repetitively and consistently. The first fix is really to return errors as a result instead of interrupting the run I think?

from dust.

I can confirm there is something badly implemented here. I ran my own code in python using browserless API as a test and never getting any errors from default parameters.

from dust.

Nice. Can you share some urls where you see discrepancies ?

from dust.

https://www.cnn.com/2020/02/21/business/wells-fargo-settlement-doj-sec/index.html

https://www.forbes.com/sites/eriksherman/2021/09/28/wells-fargo-gets-into-trouble-yet-again-over-alleged-fraud/

from dust.

This happens a lot

from dust.

Yes I had the same issue on CNN. Will investigate 👍

from dust.

Added 16s timeout instead of 30s and the networkidle2 option, should solve most issues, let me know if that's not the case 👍

from dust.

I still have this error for many webpages, is there any way to code around a solution to allow it to allow failures or have them null in the results array? In instances where you're navigating to multiple search result pages to parse there's still good data in the other pages

from dust.

@bcjordan we've added a configuration flag (only available behind API at this stage) to have the browser block return an error object instaed of erroring the run[0]:

You need to pass in your API call: config: { "BROWSER_BLOCK_NAME": { "error_as_output": true } }

[0] https://github.com/dust-tt/dust/blob/main/core/src/blocks/browser.rs#L91-L100

from dust.

@spolu oh that's perfect, thanks so much!

from dust.