View Code? Open in Web Editor
NEW
Scape firmware metadata from 18 vendors and download corresponding firmware images. Save in MySQL database for InfoSec research purposes.
License: MIT License
Python 74.70%
Jupyter Notebook 24.13%
Shell 0.63%
Dockerfile 0.54%
amos2022ws01-firmware-scraper's Issues
User story
As a user
I want to be able to setup a json configuration file
So that the polling interval for each of the vendor can be added and later when script is running - scheduled.
Acceptance criteria
It is possible to add and edit the json configuration file
For each vendor the scheduled time is saved correctly
The firmware is downloaded after earlier specified time
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a developer
When i commit code to github
Then i Want the code to be tested by the test suite
As a user
When i use the amos firmware scraper
Then i Want the software to be tested
Acceptance criteria
Common test suite is agreed between developers
Demo a test with 100% passing
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
I want to be able to delete and overwrite data in the database
So that it has updated information
Acceptance criteria
it is possible to check whether the data about specific firmware is already in the database
the data can be deleted from the database
the data can be changed in the database
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need to have the basic logger
So that it can be later called by other parts of the code
Acceptance criteria
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
I need a MySQL database to be set up
So that entries can be later added to it
Acceptance criteria
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen Asus product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
Correct firmware is downloaded
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen NOZ product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
Correct firmware is downloaded
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user when i am using the amos firmware scraper
I want the code to have a common decided structure that include:
Main file
Schedule file
Directories for the vendor modules
Logging module
So that it allows merging of future codes
Acceptance criteria
Core structure matches as per the decided structure and includes the above
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need the Zyxel specific module to be integrated into the core
So that the firmware from that vendor can be downloaded
Acceptance criteria
There is an Zyxel specific module integrated into the core
The firmware from Zyxel can be downloaded
And the meta data can be mapped and written into the database
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a developer
I want the AVM module when scraped
must contain the respective upload date
Acceptance criteria
Recent upload date available
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I want to have the logger in the vendor modules
So that specified events and errors are being logged
Acceptance criteria
There is a logger implemented in the vendor modules
The events to be logged are defined
The errors to be logged are defined
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need the Synology specific module to be integrated into the core
So that the firmware from that vendor can be downloaded
Acceptance criteria
There is a Synology specific module integrated into the core
The firmware from Synology can be downloaded
The mata data is mapped to the MySQL schema
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a {}
I want / need {}
So that {}
Acceptance criteria
Criterion 1
Criterion 2
...
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after typing in the vendor's URI
I want the software to search for the download link
So that the firmware can be downloaded
Acceptance criteria
Message of correct execution after having correctly downloaded the firmware
Error message in case the link isn't found
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having found the firmware on vendor's website
I want the software to download the metadata and save it to MySQL database
So that it can be stored in the database
Acceptance criteria
it is possible to give an input string to the program
data is saved in the database
the program returns successful execution or error message
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
I want the code to include the docker file
So that the application can strat in the docker container
Acceptance criteria
Docker container starts and execute automatically
Initiliases mySQL Database
Fullfills all software dependencies
Handles the download path
Exports all environment variable (python path, MySQL username)
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a {}
I want / need {}
So that {}
Acceptance criteria
Criterion 1
Criterion 2
...
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen random vendor's product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
Correct firmware is downloaded
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
when i access the firmware scraper
The Core should contain the logger module
Acceptance criteria
Core should contain the logger module
Logger module gets passed to the core funtions (ex download firmware, get product catalogs etc)
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen Synology product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
Correct firmware is downloaded
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen random vendor
I want the software to search the website
So that the download can be initiated
And meta data can be mapped
Acceptance criteria
Correct firmware is downloaded
The Meta data is mapped to the database
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need the Schneider Electric scraper to return all valid metadata whenever possible
So that the firmware from can be downloaded
Acceptance criteria
Multiple download links for the same firmware product can be handled
Metadata fields are extracted correctly, even for corner cases
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a developer
When i commit code to github
Then i Want the code to run through code checking
Acceptance criteria
Ex. pytest flake8 black isort (optioal : safety bandit) are passed
If tests are unsuccessfull the developer is responsible to correct and commit the correct one
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I want it to check if logging in is necessary in order to download firmware and if the logging credentials are already available
So that the firmware can be downloaded
Acceptance criteria
it is possible to determine whether logging in is necessary for downloading firmware
the software can check whether credentials for logging in already exist in the database
the software is able to download the firmware that requires logging in
in case there are no credentials the software sends a message to the user
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
When i run the core
Then the core should be able to compare the product catalog with all historised firmware data already available
Acceptance criteria
Find appropriate comaprision method
Should run as fast and effecient as possible
Write python test for finding new products
if no new products found skip downloading
Display information about the number of items being dowloaded and comapred (logging)
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
I want to be able to type in the time interval
So that the software will start the action at the set time point
Acceptance criteria
it is possible to input time
the software runs at the set time period
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen Zyxel product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
Correct firmware is downloaded
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need the AVM specific module to be integrated into the core
So that the firmware from that vendor can be downloaded
Acceptance criteria
Mapping and writing to database
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
I want the software to have honeypot traps avoider part
So that the human like behaviour can be imitated (through random delays, random view time) in order to not beeing detected by the vendor's scraper detection logic.
Acceptance criteria
There is a honeypot avoider part of code
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need the Schneider specific module to be integrated into the core
So that the firmware from that vendor can be downloaded
Acceptance criteria
There is a Schneider specific module integrated into the core
The firmware from Schneider can be downloaded
The mata data is mapped to the MySQL schema
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen AVM product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
Correct firmware is downloaded
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a {}
I want / need {}
So that {}
Acceptance criteria
Criterion 1
Criterion 2
...
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user when i am using the amos firmware scraper
I want the output must have the following mandatory fields/attributes:
Manufacturer (e.g., Siemens, D-Link, AVM, ABB, ...)
Model/Name (e.g., S7-1500, DIR300, Scalance-X, ...)
Version (e.g., 1.2.3)
Type (e.g., Router, Switch, Firewall, PLC, ...)
Release Date (if available)
Checksum (sha512)
EMBA tested (yes/no)
EMBA link to report (filesystem)
EMBArk link to report (http://...)
Firmware download link (vendor link)
Firmware filesystem link (to find it on the filesystem)
So that we have a standardised template
Acceptance criteria
All the above mentioned fields/attributes must be visible in the output
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen Schneider product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
Correct firmware is downloaded
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
I need the vendor modules to have a template structure
So that all the implemented modules are written in similar format
Acceptance criteria
There is a common structure for vendor modules
The structure has a dedicated space for logging in
The structure has a dedicated space for accessing the main page
The structure has a dedicated space for looping over the products
The structure has a dedicated space for accessing the product pages
The structure has a dedicated space for extracting meta data
The structure has a dedicated space for downloading meta data
The structure has a dedicated space for mapping meta data
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user
I want the the downloaded vendor specific meta data to be mapped into the mySQL schema
So that we have a all vendor information in a common structure
Acceptance criteria
Downloaded vendor meta data should be mapped into the schema
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need the AVM vendor data to be downloaded from web servers instead of FTP server
So that the firmware data is downloaded
Acceptance criteria
The AVM vendor firmware data is downloaded as per the schema
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user after having chosen Cisco product
I want the software to search the website
So that the download can be initiated
Acceptance criteria
There is a draft of Cisco module
The Cisco firmware can be downloaded
The metadata is mapped correctly to the MySQL schema
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a {}
I want / need {}
So that {}
Acceptance criteria
Criterion 1
Criterion 2
...
Definition of done (DoD)
Added only after week 5
The same for all features
Here goes the project specific part
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release
User story
As a user of the software
I need the Asus specific module to be integrated into the core
So that the firmware from that vendor can be downloaded
Acceptance criteria
There is an Asus specific module integrated into the core
The firmware from Asus can be downloaded
Definition of done (DoD)
All the acceptance criteria have been met
The code has been merged with the main one
The documentation and Bill of Materials have been updated
The code is complete and uploaded in GitHub
Functionality has been reviewed
The tests to ensure the feature is working have been created
DoD general criteria
Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release