The current design is a proof of concept that spawns a new worker per every new URL. It works, the concept is proved.

The next step is to create a worker pool which will have two types of processes:

Coordinator process

Spawns other processes, register results and assigns work to workers.

Coordinator flow:

1. Allocate a queue for URLs and N channels. The queue will have one value at creation, the starting URL.
2. Spawn N workers, assign one channel to each.
3. Send starting URL to one of the workers.
4. Every time a message is received through a channel:
   4.1. Empty message = URL request. Send the next URL from the queue through this channel.
   4.2. Message with URL. Put URL in the queue if it hasn't already been processed before. Reply to this same channel with next URL from the queue.

Worker process

Crawls pages.

Worker flow:

1. Request URL from coordinator.
2. Wait for URL.
3. Crawl URL.
4. Send results back.
5. GOTO 2.