harthur / glossary Goto Github PK

[UNMAINTAINED] Extract terms and keywords from a piece of text

License: MIT License

JavaScript 100.00%

glossary's Introduction

glossary

glossary is a JavaScript module that extracts keywords from text (aka "term extraction" or "auto tagging"). It takes a string of text and returns an array of terms that are relevant to the content:

var glossary = require("glossary");

var keywords = glossary.extract("Her cake shop is the best in the business");

console.log(keywords)  // ["cake", "shop", "cake shop", "business"]

glossary is standalone and uses part-of-speech analysis to extract the relevant terms.

install

For node with npm:

npm install glossary

API

blacklisting

Use blacklist to remove unwanted terms from any extraction:

var glossary = require("glossary")({
   blacklist: ["library", "script", "api", "function"]
});

var keywords = glossary.extract("JavaScript color conversion library");

console.log(keywords); // ["color", "conversion"]

minimum frequency

Use minFreq to limit the terms to only those that occur with a certain frequency:

var glossary = require("glossary")({ minFreq: 2 });

var keywords = glossary.extract("Kasey's pears are the best pears in Canada");

console.log(keywords); // ["pears"]

sub-terms

Use collapse to remove terms that are sub-terms of other terms:

var glossary = require("glossary")({ collapse: true });

var keywords = glossary.extract("The Middle East crisis is getting worse");

console.log(keywords); // ["Middle East crisis"]

verbose output

Use verbose to also get the count of each term:

var glossary = require("glossary")({ verbose: true });

var keywords = glossary.extract("The pears from the farm are good");

console.log(keywords); // [ { word: 'pears', count: 1 }, { word: 'farm', count: 1 } ]

propers

glossary Uses jspos for POS tagging. It's inspired by the python module topia.termextract.

glossary's People

Contributors

Stargazers

Watchers

Forkers

ekorn netconstructor levjj jcruz2us adamramadhan tabzone victorcastelan marcoippolito darksea shukii timecolour yawetse ngocphuongnb wesavetheworld gdg elfakamal ishan-marikar cruelmoney 5l1v3r1

glossary's Issues

Probably shouldn't include "I"

This, with minFreq: 2 and collapse: true:

http://kotaku.com/an-album-a-minecraft-style-game-both-1625202335

Yields this:

[ 'video game', 'album', 'Rustie', 'I', 'song', 'musicians' ]

But blacklist: ['I'] weirdly drops it to ['album', 'song']

Cannot call method 'indexOf' of undefined

I don't have the time right now to look deeper into this issue but I'll try to update this report with more information before the end of the week.

> glossary.extract("android - Constructor Vibrator() is not visible - Stack Overflow http://bit.ly/AEurrP")

TypeError: Cannot call method 'indexOf' of undefined
    at startsWith (/tmp/node_modules/glossary/node_modules/pos/POSTagger.js:21:18)
    at POSTagger.tag (/tmp//node_modules/glossary/node_modules/pos/POSTagger.js:75:13)
    at Glossary.extract (/tmp/node_modules/glossary/glossary.js:20:32)
    at repl:1:11

Nice lib by the way !

Why does the example not include "best"?

As is your introduction:
"It takes a string of text and returns an array of terms that are relevant to the content".

Your example:

var keywords = glossary.extract("Her cake shop is the best in the business");

console.log(keywords)  // ["cake", "shop", "cake shop", "business"]

I'm wondering, why is "best" not considered "relevant to the content"? Isn't this a very important keyword, describing the sentiment of the message?

Update natural dependency

It has made lots of versions, I wonder if upgrade can be done easy.

Hi Heather,
it would be nice if glossary could not just filter with a blacklist, but also/additional with a list of regular expressions. At the moment I sent the keyword output in this function to achieve this:

function regExFilter(terms, reExs){
return _.reject(terms, function(term){
return _.any(reExs, function(regEx){
return term.match(regEx);
});
});
}

UPDATE:
In the fork is a updated version

Cheers Nico