More "articles" scrapped than there are about zimit HOT 6 CLOSED

@Popolechien They will probably not appear after next release of zimit with the blacklist (today or tomorrow), you should re-run the scrape.

from zimit.

One of the problem is that we have this HTML redirects to secure suggestions work fine.

from zimit.

I don't think there's much we can do about this here but please share the ZIM so I can provide exact figures

from zimit.

I believe this would be the task https://farm.youzim.it/pipeline/d1c2f201514f3da67f887df5

from zimit.

Thanks ; ZIM was gone so I recreated one which gave me the following preview on kiwix 3.4.3

Drastically changed (about half) but still above the limit.

This ZIM has the following:

• 16,532 ZIM articles
• 7,696 ZIM articles in A/ namespace
• 1,147 HTML articles in A/ namespace. I believe that's what Kiwix android displays.

And here's the ventilation of those articles:

• 404.html: 1
• embed.ted.com: 1
• i0.wp.com: 4
• index.html: 1
• mesquartierschinois.wordpress.com: 475
• player.vimeo.com: 1
• public-api.wordpress.com: 2
• topFrame.html: 1
• twitter.com: 125
• vimeo.com: 1
• widgets.wp.com: 3
• www.benbest.com: 1
• www.facebook.com: 227
• www.google.com: 136
• www.pinterest.com: 124
• www.youtube.com: 44

We can clearly see the target website has 475 HTML articles ; well bellow the 1,000 limit. Other HTML articles are embed of some sort that Kiwix has no knowledge about and can not discriminate.

So nothing to be done here, expect feeding the articles' numbers discussion for kiwix-lib.

from zimit.

Interesting, thanks.

twitter.com: 125
www.benbest.com: 1
www.facebook.com: 227
www.google.com: 136
www.pinterest.com: 124

None of these are (willingly) used/mentioned/quoted in the blog - I suspect they could be ads or ad trackers of some sort that get counted as items.

from zimit.