incquerylabs / repo-crawler Goto Github PK

When the repo crawler is executed without parameters, look for a file called config.json in the working directory and parse it instead of getting command line parameters.

The configuration file should contain at least the required fields for the current version (e.g. username, password, server, port, etc.).

We should extend it to log the fully qualified name of the parent element of a missing elements

TWC provides a method to download elements at once. Issue POST request to /elements, i.e., /osmc/resources//revisions//elements, /osmc/resources//branches//elements. The request body is a list of UUID separated by commas, e.g., 4eeb2e6d-9cbc-4f59-9d74-69263e52ba54,520bc74d-b5b4-40d5-87cb-9d0b8aba5d2e,7a057ae7-6281-462d-9dcd-de9931633f5c and the Content-Type in the request header should be “text/plain”.

• introduce interface for mocking API calls or start a mock HTTP server on localhost
• add a basic test that runs the crawler and checks for completion and optionally successful session cleanup

We should improve repo-crawler in a way that helps users select a revision to test. This can be done in several ways:

• Print the label/name of visited structure elements to standard output when receiving this information
• Store the structure information in-memory and save it to a file at the end of the execution
• Stream structure information to a file while running