just some ideas about photon HOT 5 CLOSED

Thanks for the suggestions, I really appreciate it but I have two questions to ask :')

1. How do you find out if a javascript file/code runs ads and how is that relevant?
2. Don't you think that falls out of the scope for a crawler?

from photon.

nah, just checking for ads.txt file, as in 'robots.txt' - it's an RTB standard (example: https://www.gazeta.ru/ads.txt)
checking for https cert is useful for some activities (antifraud etc.).

It's not hard to implement, maybe I should find time for a pull request

from photon.

It's not about the difficulty of implementation, it's about relevance. A crawler isn't supposed to check SSL certificates.

from photon.

but as a crawler it CAN check for their presence/grab them - as an important metadata [intel]

from photon.

Sorry but I am closing this issue, I don't think it's relevant at all.

1. ads.txt isn't that common
2. Grabbing SSL certificates is out of scope

Thanks for the suggestions tho, let me know if you have some more ^_^

from photon.