Create module parser about onigumo HOT 6 OPEN

dalsi problemy

Jake formaty chceme nacitat.
Jake vystupni formaty chceme.

from onigumo.

#118 slightly blocks this because, without the rename, we’d end up with a module called Onigumo that is essentially a Downloader and then a new Parser module. Or even worse, to fit into the naming and architecture, we’d stuff everything into the existing Onigumo module.

from onigumo.

#118 got merged; I am removing the “blocked” label.

from onigumo.

To answer your questions, @nappex:

The input of the parser will be anything downloaded from the Internet. Typically, this is an HTML page, but it can be a plain text file or a JSON API response. It is Spider’s job to know and discern the format; Onigumo can only provide a set of tools (like #87) to make everyday tasks easier.

The output format will be a structured representation of the input data. JSON was my first pick for its universality, and I preferred it over YAML because the data will be processed only by a machine. The output can be, however, whatever kind of structured data, be it a serialized object of a programming language of the Spider, an XML document, or some binary format.

Theoretically, Onigumo doesn’t even need to force a concrete structure format. Having one would make it easy to define specific standardized nodes in the schema, allowing to control the core without writing any code. On the other hand, a piece of Spider code that knows its file format can provide the same data just as fine.

Feel free to share any thoughts on the topic!

from onigumo.

The work has begun: #146

from onigumo.

I extracted the steps described in your description (#89 (comment) and #89 (comment)), @nappex, to a new follow-up Issue #169. Feel free to take a look!

from onigumo.