Katti is in the development phase. We are continuously working on improving Katti and adding new features. Therefore we will release new versions in regular intervals.
The development of Katti started as a pure crawling framework. In the meantime, Katti not only offers the possibility to automatically load websites and store the data, but also provides the tools to analyze the obtained data. The goal is to develop a scalable platform that allows an easy automated analysis of websites and their infrastructure.
docker
and thedocker compose
plugin- Python >=3.10 (tested on
python==3.10
)
- Clone this repo to
~/katti
- Optionally: Create and activate a virtual environment
cd katti
pip install virtualenv
virtualenv venv
source venv/bin/activate
- Set
$PYTHONPATH
export PYTHONPATH=~/katti
- Install Katti and its dependencies
pip install .
- Start backbone services (Redis, MongoDB, RabbitMQ) with
docker compose
docker compose up -d
- Execute the Setup Script
python katti/SetupKatti
- Start the celery workers
cd katti/CeleryApps
celery -A celery -A KattiApp worker -Q scanning,default,fast_lane --concurrency 6 --loglevel debug
Youβre done! Execute the test script ~/katti/test.py
to see if it works.
See ~/katti/katti_config
database_configs.yml
,celery.yml
: Configure the way Katti talks to the backbone servicesenv.yml
: Manage API credentialsscanner.yml
: Configure default scannersYou can change the default config paths in
~/katti/katti/KattiUtils/Configs/Paths.py