I received some feedback on the README:

• Promote the Spider/Application section to the top. The fact that Onigumo is a framework and not a ready-to-use solution is essential information. (Agreed.)
• Calling applications “Spiders” is unintuitive. Either explain it right at the top and use punctuation to emphasize it is a custom term, or ditch it entirely and revert to “application” instead.

The URL input file is read as a Stream, line by line. This is how the URLs should be handled all the way down to save resources and have a cleaner code. The goal is to set up processing the whole input line as a Stream of actions. Like that it would be possible to stack all the steps, without keeping the whole lists in memory. That could get problematic e.g. when having the actual HTTP request and save to the disk, as suggested in #19, as separate steps.

The flow would become like this:

1. Read a line from the input file. (Done naturally by File.stream!/3)
2. Trim the trailing newline to get a valid URL.
3. Perform an HTTP request.
4. Validate the response.
5. Extract the body from the response.
6. Generate a filename. (Not in place yet.)
7. Save the body to a file.

Steps 1. to 2. are addressed by #66. Steps 3. to 5. would be really resource-heavy if a whole list would be rebuilt every time.

• URLs file reader
• HTTP client, response extractor
• Integrating downloader

reproduce error:

cd onigumo
mix compile

error message: ** (Mix) You're trying to run :onigumo on Elixir v1.10.3 but it has declared in its mix.exs file it supports only Elixir ~> 1.12

Unfortunately last stable version for bullseye is 1.10.3 - https://packages.debian.org/search?suite=stable&section=all&arch=any&searchon=names&keywords=elixir

elixir ver. 1.12 is testing unstable - https://manpages.debian.org/bullseye/elixir/elixir.1.en.html

We'd like to divide Onigumo.download to smaller parts with philosophy one func make one task.
This approach is more suitable for deep testing also. We'd like to use tmp_dir in test for testing files, for this intention we need to have the way how to give filepath as argument. When we reach it, we will be able to give tmp_dir as argument to make integration tests.
Onigumo.download make three different tasks at this moment.

Three tasks are:

1. make request
2. check request and match body of request to variable
3. write body content to file

• Add @nappex, #101
• bump the year. #100

A follow-up to #32 and #40. More information in a review.

tl;dr

def download(url, path, http)

feels better than

def download(url, http, path)

because then the behaviors (modules) are on one side of the call and literals (strings) on the other side together.

Follow up to #12 and #24.

• build
• run
• test

$ mix escript.build
$ mix test

If we use function pattern matching feature available in elixir then we can use same name for some functions in Downloader module.

Consider to rename main as download and download_urls_from_file also as download then these will diff just by defined arguments. Download without any args will behave as our main and download with defined arg as only one a needs to be file will behave as our download_urls_from_file. This approach will solve our naming issue.

function pattern matching - https://inquisitivedeveloper.com/lwm-elixir-23/

Use escript

• https://medium.com/blackode/writing-the-command-line-application-in-elixir-78a8d1b1850

• https://elixirschool.com/en/lessons/intermediate/escripts

• https://hexdocs.pm/mix/main/Mix.Tasks.Escript.Build.html

• https://blog.kstaykov.eu/dev/building-elixir-cli-app/

• https://whatdidilearn.info/2017/12/10/writing-command-line-app-in-elixir.html

• https://zorbash.com/post/building-command-line-applications-with-elixir/

For consistency with HttpPoison. I believe acronyms should behave as words in identifiers. In the snake case, it would be cli, not c_l_i.

This may not be possible now because of lacking parametrization.

Add some basic test to find out that commands $ ./onigumo or $ ./onigumo Downloader are running without any errors.

But how? What is the correct behavior? Currently the program crashes. Maybe retry? But what if it does not help? I’d say skip and log. But we don’t log anything yet.

The flowchart whitespace has become inconsistent. It no longer serves for proper centering or alignment. Either reformat it or collapse the whitespace into single spaces.

I prefer the latter because although whitespace alignment looks nice, it brings a considerable overhead to maintenance. Also, once the chart becomes more complex, it may even become impossible to format it nicely.

• Whitespace #109
• Arrows #110

This article advise to use at the start of each elixir project tools for control the quality of code.

basic tool for check:

• mix format
• mix credo --strict
• mix dialyzer

We can use all tools above in one with name hygiene with command mix hygiene.

Currently, dependencies update by $ mix deps.update change mix.lock file. Particulary, hashes of dependencies that are created by its version.
I think mix.lock should be generated locally by run commands as $ mix deps.update, it should not be created or rewritten by git pull.

The current Onigumo module does the downloading part of the workflow. Other components like the Parser and the Operator become distinct modules. The rename would improve semantics and prevent confusion when more modules appear in the codebase.

For the app to become barely usable, its main components must run in an endless loop. The Operator, Downloader, and Parser must be able to pick new items without a need to re-run them manually.

The Downloader can already make use of such a loop.

The loop may be better outside, in the umbrella, rather than the concrete workers. How does Elixir handle workers and scaling? OTP and thus Elixir utilizes the concept of Applications. GenServers and Supervisors look related too.

Check https://hexdocs.pm/elixir/1.12/typespecs.html

• About #91 #104
• Architecture #91 #107
• Operator #112
• Downloader #121
• Parser
• Applications (Spiders)

So file names are unique and based on the URL. MD5?

Can be just a hello world.

Escripts are for development tasks, not for running an application. Citing the Mix documentation:

Escripts should be used as a mechanism to share scripts between developers and not as a deployment mechanism. For running live systems, consider using mix run or building releases.

This suggests using Erlang libraries like native https or Hackney. But mentions HTTPoison.

A plain text file with a list of URLs.

Some of tests seem to be similar. For example download a single URL and download multiple URLs.

some useful links:

• https://gist.github.com/ramalho/1fa98648737e6bb2d4d4b77a938388f7
• https://dev.to/niku/parameterized-testing-with-exunit-1j3d
• https://gist.github.com/ishikawa/2ad8c415c2a5b6a619e99c18870d6392

The command line usage brings in the concept of a current working directory. The CWD does not make sense for a hypothetical GUI or web application. In the case of library usage, the parent application has an interface and determines the working directory for Onigumo.

Moving the working path determination to the CLI module brings another benefit in making the main function testable (#56). It prevents the usage of :tmp_dir in the tests.

Blocks #56

The transition of this project to Elixir is complete, and its Ruby origin is long gone. For the sake of easiness, let’s rename the main branch back to master.

HTTP 3xx codes.

And link it from the README.

Log when a download starts and when it finishes.

#45 moved the URLs input filename/path to a configuration file. It needs to be clear that this must be a relative path.

For the runtime, it may make sense to have an absolute path in there, but not much. The application is planned to operate in a single directory, now that being a current working directory, further to be configurable by an argument #54.

The tests using temporary directories however need for any files read and written to be stored in a relative path.

This needs good mocks, probably also creating a behavior. Maybe even splitting the Onigumo module.

We'd like to run by github actions:

1. tests by - mix test
2. check formatting - mix format --check-formatted resp. mix hygiene see #23

To keep the changeset as small as possible we reverted some improvements originally introduced in #29. Add those back more atomically. The list is in #29 (comment).

• #34
• #35
• #36
• String composition in the URLs input file from interpolation to concatenation.
• #37

I reverted my own change too:

• #48

defp assert_downloaded(url, tmp_dir) do

defp verify_downloaded_files(url, tmp_dir) do

because -> https://slovniky.lingea.cz/anglicko-cesky/ov%C4%9B%C5%99it

Currently, we parse CLI args and options just with pattern matching #145. It will be nice if we parse args from CLI by built-in module OptionParse check official docs or tutorial - elixir school. This approach ensure to handle errors more elegant with help message and so on. Currently we get some ugly error. It is not very user friendly interface.

Use Application config to store the input URL file name. Like that it can be shared between the executive code and tests as well as overridden for the test environment.

• The Operator checks the existence of .urls files. If it’s there, the Operator has already processed the corresponding structured data in the .json file.
• The Downloader checks the existence of .raw files. If it’s there, the Downloader already downloaded the corresponding URL.
• The Parser checks the existence of .json files. If it’s there, the Parser already parsed the corresponding downloaded content in the .raw file.

Consolidate the numbered lists so they describe the check consistently.

This will require introducing argument parsing. Elixir’s OptionParser will come handy.

$ ./onigumo --path .

$ ./onigumo --path pycz

Onigumo.download already does stuff. Test it!

Test that it downloads a file and writes it to body.html. Simple as that.

https://github.com/christopheradams/elixir_style_guide

The flowchart lacks details about how the data flows from Onigumo to a Spider and back. I don’t want the flowchart to become overly complicated, but the distinction between the framework and its apps is crucial, in my opinion.

My idea is something like

Start → Onigumo Operator → Spider Operator → Downloader → Onigumo Parser → Spider Parser → back to Onigumo Operator → Spider Materialization

The main point here is that Onigumo only determines what to process, and the Spiders do the actual work. Spider Operator receives a list of parsed entities and tells what to download. Spider Parser gets a list of unparsed entities and provides structured data.

There is no parsed data at the beginning, which marks a starting state. When there is nothing more to download, it is time for materialization.

The flowchart may be a bit more colorful to distinguish between the core and the userspace.

Our code compiles without any warnings; newly appearing ones point to genuine errors. I suggest configuring the pipeline to fail if the compilation yields any.

Očekáváme předem známa vstupní data stažená downloaderem. Donwloader stahne textovy obsah stranky.
data tvořící text budou ve formě co řadek to jedna url (jako kdysi napč. gopher).
Vystupem parseru bude json soubor ktery bude obsahovat seznam vsech url nalezenutych v textu dané stranky.