Request an http(s) url and scrape its metadata. Many of the metadata fields returned are Open Graph Protocol (og:) so far.
Under the hood, this package does some post-request processing on top of the request module.
If you want a new feature, please open an issue or pull request in GitHub.
To use in an npm/ Node.js project, install from your CLI:
$ npm install url-metadata --save`
Then in your project file (from example/basic.js):
const urlMetadata = require('url-metadata')
urlMetadata('http://bit.ly/2ePIrDy').then(
function (metadata) { // success handler
console.log(metadata)
},
function (error) { // failure handler
console.log(error)
})
If you'd like to override the default options (see below), pass in a second argument:
const urlMetadata = require('urlMetadata')
urlMetadata('http://bit.ly/2ePIrDy', {fromEmail: '[email protected]'}).then(...)
This package's default options are the values below that you may want to override:
{
// custom name for the user agent and email that will make url request:
userAgent: 'MetadataScraper',
fromEmail: '[email protected]',
// module will follow a maximum of 10 redirects
maxRedirects: 10,
// timeout in milliseconds, default below is 10 seconds:
timeout: 10000,
// number of characters to truncate description to:
descriptionLength: 750,
// force image urls in selected tags to use https,
// valid for 'image', 'og:image' and 'og:image:secure_url' tags:
ensureSecureImageRequest: true,
// object containing key/value pairs used for `source` attribution;
// defaults to empty object, see usage details below:
sourceMap: {},
// custom function to decode special-case encodings;
// defaults to undefined:
decode: undefined,
// custom function to encode the metadata fields before they are returned;
// defaults to undefined:
encode: undefined
}
This module introduces and supports a metadata field called source
. More details about source
in the Returns
section below. Example usage can be found in example/source-map.js
sourceMap
is used to override the default source
attribution behavior, which derives source
from the web host that the url resolves to.
sourceMap
is a simple object containing key/value pairs where the key is a YouTube username and the value is the source you'd like to attribute the content to (such as a domain, as in example below).
const options = {
sourceMap: { 'the guardian': 'theguardian.com' }
}
If you'd like to extend this functionality beyond YouTube attribution, create an issue or pull request in GitHub.
You can supply a custom function to decode the metadata scraped from the url. Example decoding of EUC-JP (Japanese) metadata can be found in example/decode.js
.
If you pass in an options.decode() function, this module will force the request module to return the scraped metadata as a buffer to decode(). This module is not opinionated about what you do in the decode() function, only that it accepts a buffer as its argument and returns a string.
You can supply a custom function to encode the metadata fields before they are returned from this module, see example/encode.js
:
const options = {
encode: function (value) {
return encodeURIComponent(value).replace(/['*]/g, escape)
}
}
Returns a promise that gets resolved with the following url metadata if the url request response returns successfully. Note that the url
field returned below will be the last hop in the request chain. So if you passed in a url that was generated by a link shortener, for example, you'll get back the final destination of the link as the url
.
{
'url' : '',
'canonical' : '',
'title' : '',
'image' : '',
'author' : '',
'description' : '',
'keywords' : '',
'source' : '', // * see note below
'og:url' : '',
'og:locale' : '',
'og:locale:alternate' : '',
'og:title' : '',
'og:type' : '',
'og:description' : '',
'description' : '',
'og:determiner' : '',
'og:site_name' : '',
'og:image' : '',
'og:image:secure_url' : '',
'og:image:type' : '',
'og:image:width' : '',
'og:image:height' : '',
'price': : '',
'priceCurrency': : '',
'availability' : ''
}
Additional fields are also returned if the url has an og:type
set to article
. These fields are:
{
'article:published_time' : '',
'article:modified_time' : '',
'article:expiration_time' : '',
'article:author' : '',
'article:section' : '',
'article:tag' : '',
'og:article:published_time' : '',
'og:article:modified_time' : '',
'og:article:expiration_time' : '',
'og:article:author' : '',
'og:article:section' : '',
'og:article:tag' : ''
}
This module introduces and supports a metadata field called source
which may be useful for attributing content hosted elsewhere to an original source.
The default behavior of this module is to derive the source
field from the url's host (in cases where redirects take place before the last hop, the host is the last hop in the request chain):
metadata.set({ source: url.split('://')[1].split('/')[0] })
You may be able to override this default behavior with the sourceMap
option.