wopehq / muninn Goto Github PK

View Code? Open in Web Editor NEW

136.0 7.0 3.0 969 KB

Muninn is a fast and flexible HTML parsing tool that simplifies the process of extracting data from HTMLs.

Home Page: https://wopehq.gitbook.io/muninn/

License: MIT License

TypeScript 100.00%

html parser json parse cheerio css-selectors htmlparser

muninn's Introduction

Muninn

Muninn is a fast and flexible HTML parsing tool that simplifies the process of extracting data from HTMLs. It allows you to create a configuration file, making it easy to keep parser settings up to date even when selectors change. With its easy-to-learn syntax and the power of the cheerio library for parsing, Muninn is a versatile solution for various parsing needs.

Features

Fast and efficient HTML parsing
Easy-to-learn syntax for creating parser configurations
Flexibility to handle changing selectors
Powered by the popular cheerio library

Documentation - Changelog

Sample

import { parse } from 'muninn';

const config = {
  schema: {
    title: '#productTitle',
    price: '#priceblock_ourprice',
    rating: {
      selector: '#acrPopover span | float',
      regex: /\d+\.?\d?/
    },
    features: {
      selector: '#productOverview_feature_div tr.a-spacing-small | array',
      schema: {
        name: 'td:nth-child(1)',
        value: 'td:nth-child(2)'
      }
    }
  }
};

// The `data` is an HTML Content of type string.
// https://www.amazon.com/AMD-Ryzen-3700X-16-Thread-Processor/dp/B07SXMZLPK/
const data = '<html>...</html>';

const result = parse(data, config);

Output

{
  "title": "AMD Ryzen 7 3700X 8-Core, 16-Thread Unlocked Desktop Processor with Wraith Prism LED Cooler",
  "price": "$308.99",
  "rating": 4.9,
  "features": [
    {
      "name": "Brand",
      "value": "AMD"
    },
    {
      "name": "CPU Model",
      "value": "AMD Ryzen 7"
    },
    {
      "name": "CPU Speed",
      "value": "4.4 GHz"
    },
    {
      "name": "CPU Socket",
      "value": "Socket AM4"
    },
    {
      "name": "Processor Count",
      "value": "8"
    }
  ]
}

License

Distributed under the MIT License. See LICENSE for more information.

muninn's People

Contributors

Stargazers

Watchers

Forkers

fatihky halilatilla serdarkaracay

muninn's Issues

If there is an element that does not have the desired attribute to be accessed, that field is returned completely empty.

Input

<div class="test">Content</test>

Selector

{
    detail: ".test",
    link: ".test @ href"
}

Output

{
    detail: "Content"
}

Expected

{
    detail: "Content",
    link: null
}

Values for non-existent selectors must return null, not empty strings.

Regex definitions should be able to be made with a regex object.

There should be three ways to use regex.

Using predefined regexes. ✅

"regex": "email",

Define as an object. ✅

"regex": { "pattern": "\\d+", "flags": "g" },

TODO: Define it as a regex object.

"regex": /\d+/g,

Support for default value assignment will be added for values that are not found.

{
  title: { initial: "initial value" }
}

Migration to Typescript

It'd be good to use Typescript instead of Javascript in this project.

If a field's selector value is not specified, it should work as 'self' by default.

{
  self: true, // <- should not be necessary
  schema: {
    title: '.title'
  }
}

How to handle arrays of objects

This is my html

<!DOCTYPE >
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head> </head>
  <body>
    <div>
      <table>
        <tr>
          <td>
            <p><a href="#" onclick="return do_hathdl('780')">780x</a></p>
          </td>
          <td>
            <p><a href="#" onclick="return do_hathdl('980')">980x</a></p>
          </td>
        </tr>
      </table>
    </div>
  </body>
</html>

This is my config

{
  schema: {
    items: {
      // What should I do ?
      type: "array",
      regex: new RegExp(
        '<a href="[^"]*" onclick="return do_hathdl\\(\'([0-9]+|org)\'\\)">([^<]+)</a>',
      ),
    },
  },
}

Expect

{
  "items": [
    {
      "res": "780",
      "name": "780x"
    },
    {
      "res": "980",
      "name": "980x"
    }
  ]
}

Pre-defined regex templates should be added

Fix parse multiple getValue method

Selector scope is lost when trying to get the value in array type.

Add multiple selector support with different attributes

Current Support:

{
  "selector": "selector, otherSelector @ attr"
}

Expected Support:

{
  "selector": "selector @ attr, otherSelector @ otherAttr"
}

Boolean fields

We may add support for boolean fields based on the existence of the selector/attribute.

Make library more generic

I think some changes would make this library suitable for more use cases. The changes are:

Exporting array instead of object in configuration JSON to accept one than more selectors.
Renaming blockSelector to selector.
Renaming collection property to union.
Accepting schema property alongside the selector to allow parse single type instead of union types.

`cli` package must be made for muninn

Some commands:

-d --dir Input directory Path
-f --file Input File Path
-c --config Config File Path
-o --output Output Directory Path

muninn --file index.html --config config.json --output output/

muninn --dir html/ --config config.json --output output/

Add Regex support

Regular expressions from config should be able to be applied on the data collected.

Accepting multiple selectors for a single Collection Item

In some cases, a field might have multiple selectors. Accepting selectors property in the Collection Item would allow us to specify multiple selectors for a single field.

regex not working

Regex not working

This is my html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01">
<html>
	<head>
	    <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
		<title>Please stand by...</title>
	</head>
	<body>
		<div id="redirectwrap">
			<h4>Thanks</h4>
			<p>You are now logged in as: lau8<br /><br />Please wait while we transfer you...</p>
		</div>
	</body>
</html>

This is my config

{
  schema: {
    username: {
      regex: new RegExp("<p>You are now logged in as: (.+?)<"),
    }
  },
}

Unfortunately, it doesn't work

Expect

working

attribute
type transform
regex
html

{
 url: {
  selector: 'a',
  attr: 'href'
 }
}

shorcut alternative

{
  url: 'a @ href'
}

Adding "trim" option by default

Unnecessary spaces in the collected data should be removed. It should be able to be disabled with this configuration.

key: {
 selector: ".selector",
 trim: false
}

wopehq / muninn Goto Github PK

muninn's Introduction

Muninn

Features

Documentation - Changelog

Sample

Output

License

muninn's People

Contributors

Stargazers

Watchers

Forkers

muninn's Issues

How to handle arrays of objects

This is my html

This is my config

Expect

Regex not working

This is my html

This is my config

Expect

Recommend Projects

Recommend Topics

Recommend Org