Hey there,

how can I prevent this error thrown by jsonld-parser.js?
The input html code has no jsonld, this why w-a-e is throwing the error.

Maybe, w-a-e should try if json-ld is present, because the error message is wrong at the moment if now json-ld is in the html - right?

Error in jsonld parse - SyntaxError: Unexpected end of JSON input                                                                 
POST /detailpage 200 4729.179 ms - 678
POST /detailpage 200 5661.727 ms - 2388
Error in jsonld parse - SyntaxError: Unexpected end of JSON input
POST /detailpage 200 4616.781 ms - 664
Error in jsonld parse - SyntaxError: Unexpected end of JSON input                                                                 POST /detailpage 200 4659.136 ms - 1339  

I'm using the web auto extractor to retrieve recipes embedded as microdata. Unfortunately, the extractor is not extracting the ingridient list as an array. Instead the ingridients replace each other in the object so that only the last ingridient remains.

Example:

<ul class="ingredientList">`
  <li itemprop="ingredients">&frac12; cup butter</li>`
  <li itemprop="ingredients">&frac12; cup powdered sugar</li>`
  <li itemprop="ingredients">&frac12; cup chocolate chips</li>`
</ul>

The web auto extractor only extracts:
{ ..., ingedients : "½ cup chocolate chips", ...}
Instead of:
{ ..., ingedients : ["½ cup butter","½ cup powdered sugar","½ cup chocolate chips"], ...}

An example page with recipe microdata can be found here: getmecooking.com

Is this behavior by design?

The official schema.org Recipe example allows this kind of multi declaration of ingridients.

You are using 1% of lodash... is it worth including?

_.find
_.keys
_.isArray

can all be switched to

arr.find
Object.keys(obj)
Array.isArray

Hello,

if I take the example in the README then run:
browserify parser.js -o bundle.js
It fails and writes only:

^
ParseError: Unexpected character ' '

I tried with another plugin to parse schema.org, it works with browserify, but their usage is horrible, as their doc :/

Any idea how to fix it?

Thanks

When I try to parse this page (https://www.discountnaturalhealth.com/bilberry-eye-support-advanced-30-tablets-blackmores/), I get the following error:

contentScript.bundle.js:190 TypeError: currentScope[attribs[PROP]].push is not a function
    at Object.onopentag (contentScript.bundle.js:146442:37)
    at Parser.onopentagend (contentScript.bundle.js:48652:23)
    at Tokenizer._stateBeforeAttributeName (contentScript.bundle.js:49170:19)
    at Tokenizer._parse (contentScript.bundle.js:49664:18)
    at Tokenizer.write (contentScript.bundle.js:49639:10)
    at Tokenizer.end (contentScript.bundle.js:49817:21)
    at Parser.end (contentScript.bundle.js:48833:21)
    at exports.default (contentScript.bundle.js:146528:44)
    at Object.parse (contentScript.bundle.js:146253:48)
    at isCategoryPage (contentScript.bundle.js:143319:87)

You can see in the W3C validator that this page has some errors, including a stray </i> tag: https://validator.w3.org/nu/?doc=https%3A%2F%2Fwww.discountnaturalhealth.com%2Fbilberry-eye-support-advanced-30-tablets-blackmores%2F

Here's the code you can use to reproduce the issue:

try {
  const parsed = WAE().parse(document?.body?.innerHTML);
} catch {
}

I would expect the parse to fail due to bad html, but I would have expected it to catch, rather than causing an error.

Thanks for creating such a great project!

I ran into a bug parsing microdata content where itemprop contained multiple properties, like in these examples and thought I'd share what I ran into:

<meta data-rh="true" property="article:published" itemprop="datePublished dateCreated" content="2019-07-21T09:00:06.000Z"/>

<span itemProp="publisher copyrightHolder provider sourceOrganization" itemscope="" itemType="http://schema.org/NewsMediaOrganization" itemID="https://www.nytimes.com">

<figure itemprop="associatedMedia image" itemscope itemtype="http://schema.org/ImageObject" data-component="image" class="element element-image img--landscape  fig--narrow-caption fig--has-shares " data-media-id="f82028d62b1edd7417d7d3773c4abf0d4fa86174" id="img-3">
  <meta itemprop="url" content="https://i.guim.co.uk/img/media/f82028d62b1edd7417d7d3773c4abf0d4fa86174/0_272_6435_3861/master/6435.jpg?width=700&amp;quality=85&amp;auto=format&amp;fit=max&amp;s=016df6a3f33eabe3cbca39eb389a60fb">
</figure>

Markup like this is parsed correctly in Google's Structured Data Testing Tool, but web-auto-extractor does not currently split input based on spaces.

I resolved this in a project which uses web-auto-extractor by doing this:

const __transformStructuredData = (structuredData) => {
   let result = structuredData
   Object.keys(result.microdata).forEach(schema => {
     result.microdata[schema].forEach(object => {
       Object.keys(object).forEach(key => {
         if (key.includes(' ')) {
           key.split(' ').forEach(newKey => {
             object[newKey] = object[key]
           })
           delete object[key]
         }
       })
     })
   })
   return result
 }

I'm aware there are some other PRs related to handling whitespace trimming open.

If an enhancement like this appeals I'd be happy to raise a PR.

Given the following HTML:

<div itemscope itemtype="http://schema.org/Product"><h1 itemprop="name"><span>Foo</span> Bar</h1></div>

I would expect the library to extract a Product with the name of Foo Bar, but it extracts FooBar omitting the space.

Do you think this would be an easy fix?

Hi!

// in src/index.js
export default function

becomes

// in dist/index.js
exports.default = function () {

so the ES import currently needs to look like this:

import WAE from 'web-auto-extractor'
WAE.default().parse(html)

instead of

import WAE from 'web-auto-extractor'
WAE.parse(html)

which is what the readme says.

That could maybe be solved by making ES imports directly import the src files, something like this:

// in package.json
{
    "exports": {
        "import": "./src/index.js",
        "require": "./dist/index.js"
    }
}

see documentation

Hi,

There is two ld+json scripts at https://www.rakuten.de/produkt/ncm-milano-plus-48vtrekking-e-bike-16ah-768wh-panasonic-zellen-akku-plus-weiss-26-V2998816670.

First one is organization and second is product but parser only displays org.

  | {
  | "@context": "http://schema.org",
  | "@type": "Product",
  | "name": "NCM Milano Plus 48V,Trekking E-Bike, 16Ah 768Wh Panasonic Zellen Akku",
  | "category": "Elektro Räder",
  | "url": "https://www.rakuten.de/produkt/ncm-milano-plus-48vtrekking-e-bike-16ah-768wh-panasonic-zellen-akku-plus-weiss-26-V2998816670",
  | "image": "https://files.rakuten-static.de/dbfcb25a05c4143dc35c09b98afed7f7/images/0b67f328ad3a324dd098bca6a16dab1f.jpg",
  | "offers": {
  | "@type": "Offer",
  | "price": "1599",
  | "priceCurrency": "EUR"
  | }
  | ,"brand": [
  | {
  | "@type": "Brand",
  | "name": "NCM",
  | "image": "https://files.rakuten-static.de/brands/ncm-b122721.png"
  | }
  | ]
  | }
  | }
  | </script>

The current JSON-LD parser assumes a perfect world scenario.

• Some websites (e.g. www.empireonline.com/) have JSON that includes new-lines, i.e. invalid JSON.
• Some website (e.g. Variety) have JSON that is surrounded in CDATA comments, e.g. https://gist.github.com/gajus/4a2653b4a5235ccebedc44467a2896f2. Furthermore, this includes ; at the end of the JSON.

Here is how I've implemented a LD+JSON parser in my local project:

(html: string): $ReadOnlyArray<Object> => {
  const dom = new JSDOM(html);

  const nodes = Object.values(dom.window.document.querySelectorAll('script[type="application/ld+json"]'));

  return nodes.map((node) => {
    if (!node || typeof node.innerHTML !== 'string') {
      throw new TypeError('Unexpected content.');
    }

    let body = node.innerHTML;

    debug('body', body);

    // Some websites (e.g. Empire) have JSON that includes new-lines, i.e. invalid JSON.
    body = body.replace(/\n/g, '');

    // Some website (e.g. Variety) have JSON that is surrounded in CDATA comments, e.g.
    // https://gist.github.com/gajus/4a2653b4a5235ccebedc44467a2896f2
    body = body.slice(body.indexOf('{'), body.lastIndexOf('}') + 1);

    return JSON.parse(body);
  });
};

Thus far it works with all the sites I have been testing.

Hi,

Thanks for the great library. Due to limitation of time, I am not able to submit a pull request but I think its worth raising an issue here. The Keys of the Object created from the data are directly pulled from the HTML. So for example if a user is using itemprop="Description" then it will add Description as key to microdata but if a user is using itemprop="description" then it will use the lowercase description key.

Although the burden of using a valid schema with proper cases lie on the webmaster of the site, It will be nice if WAE can transform the keys before adding them into the object.

Currently I add a fix to my code by using then following recursive function:

const _ = require('lodash'); const lowerCaseKeys = function (obj) { return _.transform(obj, (result, value, key) => { let k = key; if (_.isString(key)) { k = key.toLowerCase(); } if (_.isObject(value) || _.isArray(value)) { result[k] = this.lowerCaseKeys(value); } else { result[k] = value; } }, []); },

I use it for small objects so its not a problem for me yet but if anyone else wish to use it, be cautious as it creates a stack for every iteration. If the keys get modified before being added to the object then it will eliminate extra processing like above.

I am happy to help further if the Library Owner wants/needs.

My team is comparing this package, web-auto-extractor, with node-metainspector and meta-extractor.

TITLE - web-auto-extractor seems to return all the meta tags including og tags, along with the LRMI tags. Great. But, it doesn't seem to return some basic HTML elements like 'Title', which is actually important. Was this intentional? Perhaps I'm overlooking something. Perhaps the intent of this package is different than what I'm trying to do.

REDIRECTS - node-metainspector will also follow redirects up to 5 redirects I believe. This is nice and many sites do actually redirect as their main page changes. Does web-auto-extractor have the ability to follow redirects?

USER-AGENT - Both other packages allow the user-agent to be set. Does web-auto-extractor have this ability too?

Thanks.

Do you mind adding 'http-equiv' to the list of recognized meta-tag keys?

return ['name', 'property', 'itemprop', 'http-equiv'].indexOf(attr) !== -1;