Support If-Modified-Since headers in dumper about fuuka HOT 6 CLOSED

it requires storing even more state

It's one header string per page, how's that going to consume any RAM?

it doesn't strike me as something that brings any kind of bandwidth saving in normal usage.

I don't know how intensive the dumper is, but it will lower the cpu usage as it doesn't need to process unmodified pages.
It will be a huge improvement for closed stickies for example, instead of downloading MBs of HTML, it will download exactly 162 bytes of headers.
With that improvement, the dumper will be able to dramatically decrease its refresh rate without trouble.

4chan X benefits of it.

from fuuka.

It's one header string per page, how's that going to consume any RAM?

It should be completely negligible, yes. But I don't pretend to understand Perl threads and its model of sharing variables work, as I didn't write the dumper, and the dumper uses up wayyy too much RAM for something that apparently isn't storing that much.

It will be a huge improvement for closed stickies for example

The dumper doesn't enter those threads to begin with unless it sees new replies in the index page. Fuuka's dumper works by monitoring index pages (which are almost always changed, so the benefits for If-Modified-Since are null for any medium to fast board) and only enters threads when it needs to, so it should never enter a closed sticky or a thread that hasn't been updated under normal operation. In fact, the dumper doesn't even enter threads when it can get all the info it needs from the index page (ie: when the last post it saw for that thread is still on the main page, the new post doesn't have an image and the new post isn't cut off by "Comment too long. Click here to view the full text")

Like I said in the OP, while it can't harm anything, If-Modified-Since brings very little to the table for Fuuka's main mode of operation. It should help its secondary mode of operation by quite a lot, (the forced refreshes) but since they happen once every 12, 16 or even 120 minutes (these are example default values, it's the thread-refresh-rate setting in the config file), it doesn't seem like that much of a big deal. The advantage I'm seeing with If-Modified-Since is that one could most likely decrease the thread-refresh-rate by quite a lot, and that'd be a nice complement to the main mode, as it'd help with catching sages in threads in pages in the back and posts in permasaged threads.

from fuuka.

Does the dumper support gzipped content?
Not sure if LWP::UserAgent can do it automatically, but this might be the best way to save bandwidth.
The almost 6k posts long sticky on /b/ for example only takes a bit more than 600KB.

from fuuka.

I don't pretend to understand Perl threads and its model of sharing variables work

Oh, well me neither.

The dumper doesn't enter threads unless it deems it necessary.

I see.

Well the only advantage I'd see in keeping threads open would be to catch posts that got deleted or banned without changing and that are not visible on the board pages.

from fuuka.

Well the only advantage I'd see in keeping threads open would be to catch posts that got deleted or banned without changing and that are not visible on the board pages.

It does keep threads "open," but it only does forced refreshes on them with a pretty huge interval.

But yeah, you got me thinking about it. While doing I-M-S wouldn't affect normal operation, it would let people lower their thread-refresh-rate and that'd result in catching banned + deleted posts very quickly. Not to mention catching saged posts and posts in permasaged threads faster than the refresh rate for pages in the back, which can be set to an hour or more.

I am convinced now and will look at this.

Does the dumper support gzipped content?

This is a great idea, thanks.

from fuuka.

Cool.

from fuuka.