Is your feature request related to a problem? Please describe.

Incremental Crawler visits a specific site multiple times in a specific interval of time. The main focus of an incremental crawl is to keep on updating the hyperlink list of a site whose content is being changed frequently. At every iteration, after the crawler visits such a site, the site’s “freshness” keeps on decreasing as time passes. Hence in the next iteration, the crawler updates the hyperlinks, resetting the freshness.

Describe the solution you'd like
TODO

Describe alternatives you've considered
TODO

Additional context
TODO

I have installed requirements successfully but using Python 3.8 on both on Ubuntu 20.04 and Windows 10 I get the same error after trying to run darkspider.py:

Traceback (most recent call last):
  File "darkspider.py", line 58, in <module>
    from modules.helper import get_tor_proxies, setup_custom_logger
  File "/home/acwh0110/DarkSpider/modules/__init__.py", line 1, in <module>
    from modules.crawler import Crawler
  File "/home/acwh0110/DarkSpider/modules/crawler.py", line 15, in <module>
    from modules.checker import url_canon
  File "/home/acwh0110/DarkSpider/modules/checker.py", line 9, in <module>
    from modules.helper import TorProxyException, TorServiceException, get_requests_header
  File "/home/acwh0110/DarkSpider/modules/helper/__init__.py", line 1, in <module>
    from .helper import *
  File "/home/acwh0110/DarkSpider/modules/helper/helper.py", line 79, in <module>
    name: str, filename: str = "log.log", verbose_: bool = False, filelog: bool = True, argv: list[str] = None
TypeError: 'type' object is not subscriptable

Describe the solution you'd like
Track links generated in crawler module

Format proposal

{
  "link1": [
    "link2",
    "link3",
    "link4"
  ],
  "link2": [
    "link5",
    "link6",
    "link4"
  ],
  "link3": [
    "link7",
    "link2",
    "link9"
  ],
  "link4": [
    "link1"
  ]
}

Optimization

• Prevent infinite looping: link1 -> link2 -> link4 -> link1

TLD wise distribution of incoming links from dark web to the surface web.

Problem building wxpython. Already tried this but still get the below.

Checking for header Python.h                          : Distutils not installed? Broken python installation? Get python-config now!
      The configuration failed
      (complete log in /tmp/pip-install-8or1rxz3/wxpython_6001026db67149c2aed0382e7a140ba4/build/waf/3.8/gtk3/config.log)
      Command '"/usr/bin/python3.8" /tmp/pip-install-8or1rxz3/wxpython_6001026db67149c2aed0382e7a140ba4/bin/waf-2.0.24 --wx_config=/tmp/pip-install-8or1rxz3/wxpython_6001026db67149c2aed0382e7a140ba4/build/wxbld/gtk3/wx-config --gtk3 --python="/usr/bin/python3.8" --out=build/waf/3.8/gtk3 configure build ' failed with exit code 1.
      Finished command: build_py (0m2.111s)
      Finished command: build (19m2.947s)
      Command '"/usr/bin/python3.8" -u build.py build' failed with exit code 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for wxpython
  Running setup.py clean for wxpython
Failed to build wxpython
Installing collected packages: wxpython, pandas, contourpy, beautifulsoup4, matplotlib, Gooey, seaborn
  Running setup.py install for wxpython ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for wxpython did not run successfully.
  │ exit code: 1
  ╰─> [63 lines of output]

and

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> wxpython

Describe the solution you'd like
-t, --thread number : How many pages to visit (Threads) at the same time (Default: 16)

The distribution of the distance between connected pairs with the percentage of connected pairs.

Describe the solution you'd like
Generate unit test cases for each module respectively.

Is your feature request related to a problem? Please describe.
The Ultimate aim of the project is to detect illicit websites. As of now the algorithm uses graph knowledge to target suspicious links. Advance techniques are required to accurately classify links and reduce the computational complexity.

Describe the solution you'd like
Text-based classification using NLP which transforms the crawler into Context Focused Crawler from the traditional Naive-Best First Crawler. This will further help in crawling at greater depths.

Describe alternatives you've considered
Classification technique is yet to be decided.

Lay down the actual finding from all the graphs and insights. Must be logged in a file for further study.

Describe the solution you'd like
Make amends to crawl images as well as scripts

Is your feature request related to a problem? Please describe.

Extractor takes maximum file name length under consideration and creates sub-directories based on the url.

http://a.com/b.ext?x=&y=$%z2 -> a.com/b.extxyz2_.html (a.com folder with b.extxyz2_.html file in it)

This is good for storage purpose but does not act like a database.

Issues:

• File retrieval and merging of data for URL classification is complex.
• An URL can be very big but file names have length constraints.

Describe the solution you'd like
A linear architecture where a folder consists of files with file names as SHA1 hash of the respective URLs.

$ cat output/github.com/extracted/

00d1fbae77557ec45b3bfb3bdebfee49fd155cf9
b615c769e688dd83b2845ea0f32e2ee0c125c366
9b76fbceb3abd3423318ee37fd9ec1073961c14d

The links.txt file is renamed to links.json with the following content:

{
    "00d1fbae77557ec45b3bfb3bdebfee49fd155cf9": "http://github.com",
    "b615c769e688dd83b2845ea0f32e2ee0c125c366": "http://github.com/about/careers",
    "9b76fbceb3abd3423318ee37fd9ec1073961c14d": "http://github.com/sponsors"
}

Describe alternatives you've considered

Storing URLs in a big flat directories is a performance overhead as well (O(N) lookups).

Possible options:

• SQL DB
• Neo4j

Usage in README is still not detailed.
Based on current development as mentioned in #23 more functionality will be included in future so it'll be better to move usage to Wiki pages.

Dark web isn't made of just .onion links. Dark web consist of various darknets such as tor, I2P and freenet. These darknets also holds considerable amount of data. As they are P2P (peer to peer) networks, some adjustments are required to crawl them as well

Describe the solution you'd like

• Starting a I2P server on the local machine
• Using its proxy (127.0.0.1:4444) to traverse a I2P links.

I successfully implemented this method but need more ideas on integrating it with current crawler efficiently.

Describe the solution you'd like
*.com or general wildcards which will be excluded from crawler module

-z, --exclusions : Paths that you don't want to include

Describe the solution you'd like
Using NetworkX package plot a topological structure.

Derive indegree-outdegree based conclusion