fetch calls fetch recursively.
This is a place where we can speed up crawling by introducing multithreading.

To implement multithreading:

• use a concurrent.futures.ThreadPoolExecutor
• make the ResourceClient class threadsave when accessing the shared state _ids.
• make sure the api supports multithreading

Hints:

• Maybe rename fetch to _fetch and make fetch create the ThreadPoolExecutor
  • pass the number of threads
  • wait for the completion of the call

This should be a list as described by the JSONAPI specification:

{
   "data": [
      { "attributes": { ... resource ... }}, 
      { "attributes": { ... resource ... }}, 
      ...
  ]
}

• add tests
• add test for malformed data
• add description to the readme
• add an example , see #4

Add this documentation to the README.rst file:

This crawler

• is part of the resources api v1
  • describe
  • add a link
• tests against the test server
  • describe
  • add link
• is part of the architecture
  • describe
  • link to blog post

• add a docker file
• execute the crawler on start with arguments, see __main__.py

If you use --help, you can see the parameters.
They are described in cli.py.

• add all parameters to the readme file
• describe what they do

We should allow file:// urls for locally generated content.

• test that file:// urls can be crawled
• make sure that a url list from http(s) cannot access file:// urls
  This way, resources from the internet can not access local files.
• file:// urls from a file:// list can be accessed (recursively)

The crawler can be used to crawl examples.

• create a branch for examples
• include an install.sh file to install the crawler
• start the test server in an extra file, silently
• add examples
  • one url from examples
  • at least two urls from examples
  • show what happens with invalid resources (file an issue if crawler fails)
  • a file with a list of urls
  • using the file:// schema
    • point out that this eases crawling to the local file system
• add a link to the example usages to the README
• add a very simple example to the readme

To work on the issue, choose one point and create a pull-request for it.

Test what happens with

• invalid resources
• invalid json
• invalid urls
• empty urls
• urls with white space the the end/beginning

This option

• allows deleting all crawled resources.

• disables crawling

• remove the comments from the tests 344b158

• make the tests run

Hints:

• You will need to edit the cli.py file