Comments (5)
sjdirect
commented on June 7, 2024
Have you tried creating your own IHyperLinkParser or Extending the AnglesharpHyperlinkParser to implement this logic. Wouldn't be hard to do. You would also need to change the following to make sure it would download the content of the sitemap url...
config.DownloadableContentTypes = "text/html, application/xml";
from abot.
JoshTango
commented on June 7, 2024
I might one day.
but the sitemap.xml is suck a generalized standard thing these days I thought you might want to build it in to Abot
from abot.
winzig
commented on June 7, 2024
Abot doesn't use sitemaps to help discover pages to crawl?
from abot.
sjdirect
commented on June 7, 2024
It's default behavior is to crawl the site based on real navigate-able links. The sitemap can be completely out of sync with the real site so was never part of the original design. However, you can implement your own IHyperLinkParser like mentioned above that will use the sitemap.
from abot.
winzig
commented on June 7, 2024
In my experience, we have used sitemaps extensively to help search engines index pages of our sites that they may otherwise have trouble finding. So yeah we'll have to implement this internally I guess.
from abot.
Related Issues (20)
- How to implement the decision maker function HOT 1
- User Agent config value appears to be getting split on spaces and sending requests with multiple user-agent headers HOT 2
- Update Readme file? HOT 1
- Collection of links found HOT 1
- Bot stop crawling without any error
- Handling pagination HOT 1
- Best way to do page-by-page crawling HOT 2
- Incorrect usage of ThreadManager means it does nothing HOT 4
- How do I direct it to crawl, specifc paged content in a site directory? HOT 2
- The PageRequester downloads the content by default HOT 1
- Exception is thrown when redirect URI is relative HOT 1
- Base URI logic of the HyperLinkParser breaks on Linux when faced with a root-relative base tag value HOT 2
- Base URI logic of the HyperLinkParser doesn't respect terminated relative base tag values HOT 2
- Question HOT 1
- Control maximum number of concurrent pages to be crawled HOT 1
- Are 304 responses properly handled? HOT 2
- Crawl .jsp sites
- Content.Text empty despite response code OK and Content stream contains data
- Cancellation is not working properly and integration test is wrong HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abot.