google / categorybuilder Goto Github PK

License: Apache License 2.0

Python 100.00%

categorybuilder's Introduction

Category Builder

This repository contains data and code for the Category Builder system.

Category Builder can do set expansion while dealing robustly with polysemy. See category_builder_paper.pdf in this directory.

Installation

Download code and data using git. You will need to have installed git lfs.

git clone https://github.com/google/categorybuilder
cd categorybuilder
git lfs pull

Initialization

Note: This will take a few minutes to initalize. An sqlite database of about 5.7 GB is produced.

Note: The initialization uses alive_progress to show progress. You can get it by 'pip3 install alive_progress'.

python initialize.py

How to use Category Builder

python category_builder.py ford nixon
python category_builder.py --rho=2 --n=20 ford chevy

The seeds to expand are provided on the command line as positional arguments and should be lowercase. Compound names (e.g., "New York") should be quoted.

python category_builder.py chicago "new york"

Example Output

Seeds	Expansion
ford, nixon	nixon, obama, bush, johnson, clinton, ford, reagan, ...
ford, chevy	ford, chevy, toyota, chevrolet, honda, bmw, nissan, ...
ford, stallone	ford, stallone, khan, kapoor, sylvester stallone, depp, tom cruise, ...
cancer, diabetes	cancer, diabetes, disease, asthama, infection, breast cancer, syndrome, ...
cancer, taurus	virgo, pisces, libra, taurus, scorpio, saggitarius, cancer, aries, capricorn, aquarius, gemini, leo, ...
safari, trip	trip, tour, trips, safari, vacation, adventure, holiday, excursion, cruisetours, journey, ...
safari, ie	firefox, internet explorer, chrome, explorer, safari, ie, google chrome, browsers, web browser, ...
beautiful, serene	beautiful, serene, peaceful, tranquil, picturesque, quite, lovely, stunning, scenic, secluded, ...
beautiful, poignant	beautiful, poignant, romantic, evocative, gorgeous, poetic, haunting, funny, sad, vivid, ...
beautiful, chic	elegant, chic, stylish, beautiful, gorgeous, stunning, trendy, lovely, vintage, classy, sleek, ...

How to do analogies

The same system can solve analogies such as "What is the mount everest of africa?"

python analogy.py "mount everest" africa

Note that these are harder than "proportional" analogies such as "hand:glove::foot:?". People don't need to be provided the first term ("hand") and can answer "What is the glove for a foot?"

Example Output

The items are labeled B and C because analogies are often shown as A:B::C:D.

B	C	What is the B of C?
mount everest	africa	kilimanjaro
mount everest	alaska	denali
glove	foot	shoe
darwin	physics	einstein
corolla	honda	honda civic
sacramento	indiana	indianapolis
dollar	india	rupee
football	india	cricket
voldemort	tolkien	sauron
voldemort	star wars	vader
tolkien	voldemort	rowling

How to run the evaluation suite

python3 eval_set_expansion.py eval_data/cat_eval_data/nfl-teams

python3 eval_analogy.py eval_data/analogy_eval_data/questions-words.txt

categorybuilder's People

Contributors

Stargazers

Watchers

categorybuilder's Issues

rebuilds shelf every time you run a query?

I've followed the installation instructions and saw the expected console output during initialization. However, now I ran the command python categorybuilder.py hunger emergency and I'm seeing similar output to the init process -- which took hours on the host machine, and a repeat of which would make this tool unusable.

(py27) [mssammon@shelley categorybuilder] >python category_builder.py  hunger emergency
	Creating shelf. Processed 10000 lines out of 200000
	Creating shelf. Processed 20000 lines out of 200000
	Creating shelf. Processed 30000 lines out of 200000
	Creating shelf. Processed 40000 lines out of 200000
	Creating shelf. Processed 50000 lines out of 200000
	Creating shelf. Processed 60000 lines out of 200000
	Creating shelf. Processed 70000 lines out of 200000

Error using set-expansion evaluation

@amahabal
Trying the evaluation script in Python 2.7 and getting the following error:

(env27) vagelos-ve504-0909:categorybuilder daniel$ python2.7 eval_set_expansion.py eval_data/cat_eval_data/break_verbs 
defaultdict(<type 'int'>, {'deform': 19, 'melts': 5, 'scratch': 6, 'degrades': 69, 'cuts': 43, 'rot': 4, 'tore': 15, 'shattered': 3, 'cut': 43, 'ruin': 34, 'torn': 15, 'fall off': 42, 'suffocate': 35, 'fade': 10, 'crack': 17, 'stain': 25, 'topple': 37, 'melting': 5, 'crumbling': 20, 'breaks': 1, 'shatter': 3, 'crushed': 7, 'break': 1, 'cutting': 43, 'crushes': 7, 'smash': 8, 'tear': 15, 'obliterate': 36, 'explode': 29, 'scorch': 38, 'broke up': 67, 'crushing': 7, 'breaks up': 11, 'tip over': 13, 'wear out': 50, 'smashes': 8, 'peel': 51, 'melt': 5, 'crush': 7, 'abrade': 59, 'broken up': 67, 'deforms': 19, 'blow': 48, 'ripped': 9, 'splinter': 14, 'disintegrated': 18, 'dry out': 44, 'burn': 21, 'smudge': 27, 'broken': 1, 'evaporate': 54, 'disintegrating': 18, 'implode': 55, 'chipped': 2, 'crumbles': 20, 'chop': 24, 'shred': 66, 'crumbled': 20, 'chipping': 2, 'sink': 57, 'chopped': 24, 'penatrate': 41, 'come off': 16, 'smashing': 8, 'break down': 68, 'pulverized': 40, 'ripping': 9, 'pulverize': 40, 'vaporize': 64, 'disintegrate': 18, 'splintered': 14, 'hitting': 47, 'broken down': 68, 'break up': 67, 'erode': 61, 'collapse': 70, 'hits': 47, 'warp': 52, 'shattering': 3, 'degrade': 69, 'broke': 1, 'tearing': 15, 'unhinge': 60, 'cracks': 17, 'annihilate': 71, 'deformed': 19, 'corrode': 32, 'harden': 58, 'melted': 5, 'smashed': 8, 'cracked': 17, 'dissolve': 22, 'chip': 2, 'sag': 65, 'rip': 9, 'crumble': 20, 'rips': 9, 'bend': 23, 'deforming': 19, 'delaminate': 63, 'hit': 47, 'tarnish': 28, 'breaking': 1, 'damage': 49, 'scrape': 56, 'trample': 33, 'split': 45, 'destroy': 26, 'shrink': 53, 'deteriorate': 46, 'deflate': 31, 'came off': 16, 'cracking': 17, 'shatters': 3, 'derail': 30, 'dissolves': 22, 'calcify': 62, 'wears out': 50, 'crumple': 39})
SEEDS TO SELECT FROM:  ['break', 'chip', 'shatter', 'rot', 'melt', 'scratch', 'crush', 'smash', 'rip', 'fade', 'break up', 'break down', 'tip over', 'splinter', 'tear', 'come off', 'crack', 'disintegrate', 'deform', 'crumble', 'burn', 'dissolve', 'bend', 'chop', 'stain', 'destroy', 'smudge', 'tarnish', 'explode', 'derail', 'deflate', 'corrode', 'trample', 'ruin', 'suffocate', 'obliterate', 'topple', 'scorch', 'crumple', 'pulverize', 'penatrate', 'fall off', 'cut', 'dry out', 'split', 'deteriorate', 'hit', 'blow', 'damage', 'wear out', 'peel', 'warp', 'shrink', 'evaporate', 'implode', 'scrape', 'sink', 'harden', 'abrade', 'unhinge', 'erode', 'calcify', 'delaminate', 'vaporize', 'sag', 'shred', 'break up', 'break down', 'degrade', 'collapse', 'annihilate']
Traceback (most recent call last):
  File "eval_set_expansion.py", line 105, in <module>
    map_n=flags.map_n, rho=flags.rho, n=flags.n)
  File "eval_set_expansion.py", line 73, in Eval
    expansion = GetExpansion(seeds, rho=rho, n=n)
  File "/Users/daniel/ideaProjects/categorybuilder/eval_util.py", line 41, in GetExpansion
    return GetExpansionCBGivenQuery(modified_seeds, rho, n)
  File "/Users/daniel/ideaProjects/categorybuilder/eval_util.py", line 35, in GetExpansionCBGivenQuery
    universal_newlines=True).strip().split(', ')
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['python', 'category_builder.py', '--cutpaste', '--n', '100', '--rho', '3.0', '--expansion_size', '500', 'ruin', 'sink', 'crumple']' returned non-zero exit status 1

The syntax looks good to me, but not sure why this is happening. If you have any ideas as to what could be the cause of this, please let me know.

does not support py3

Even after the recent bugfixes the code doesn't support python3

Performance evaluation

I would want to evaluate the accuracy as such of the model, how can I possibly do so?

Undefined variable in `analogy` evaluations

In the analogy evaluations, there seems to be an undefined variable: use_plain_ppmi :
https://github.com/google/categorybuilder/blob/master/eval_analogy.py#L95

FYI @amahabal

HASH: Out of overflow pages. Increase page size

hi,
I'm trying to run python3 initialize.py after installing.... (python 3.7)

I receive an error:
Initializing two matrices.
Checking if we need to produce './i-to-f-shelf' from './candidate_release-i-to-f.csv.bz2'
Processing './candidate_release-i-to-f.csv.bz2'. This may take a few minutes.
HASH: Out of overflow pages. Increase page size
Traceback (most recent call last):
File "initialize.py", line 21, in
util.CreateShelves(True)
File "/categorybuilder/category_builder_util.py", line 104, in CreateShelves
CreateShelf(GetPath(I_TO_F_INPUT), GetPath(I_TO_F_SHELF), linecount=200000, verbose=verbose)
File "/categorybuilder/category_builder_util.py", line 94, in CreateShelf
s[key] = rest.strip()
File "/Users/stefano/anaconda3/lib/python3.7/shelve.py", line 125, in setitem
self.dict[key.encode(self.keyencoding)] = f.getvalue()
_dbm.error: cannot add item to database

Thank yuo for any help.

IOError: invalid data stream

Followed the instructions, but I'm getting errors:

IOError: invalid data stream # python 2.7

I tried 2to3 -w . and that seemed to work for converting to python 3 compatible code. Similar error:

OSError: Invalid data stream # python 3.7

Using arch linux

Does not matter whether I run category_builder.py or analogy.py. See trace:

pascal@archbook:~/gits/categorybuilder$ python2 category_builder.py ford nixon
Traceback (most recent call last):
  File "category_builder.py", line 29, in <module>
    CB = util.CategoryBuilder()
  File "/home/pascal/gits/categorybuilder/category_builder_util.py", line 116, in __init__
    CreateShelves()
  File "/home/pascal/gits/categorybuilder/category_builder_util.py", line 62, in CreateShelves
    CreateShelf(GetPath(F_TO_I_INPUT), GetPath(F_TO_I_SHELF))
  File "/home/pascal/gits/categorybuilder/category_builder_util.py", line 48, in CreateShelf
    for line in csvreader:
IOError: invalid data stream

initialize.py throwing compilation error

After running python initialize.py cmd this error came --

Traceback (most recent call last):
File "initialize.py", line 18, in
import category_builder_util as util
File "C:\Users\samy\categorybuilder\category_builder_util.py", line 63
print "Checking if we need to produce '%s' from '%s'" % (outfile, infile)
^
SyntaxError: invalid syntax

google / categorybuilder Goto Github PK

categorybuilder's Introduction

Category Builder

Installation

Initialization

How to use Category Builder

Example Output

How to do analogies

Example Output

How to run the evaluation suite

categorybuilder's People

Contributors

Stargazers

Watchers

Forkers

categorybuilder's Issues

Recommend Projects

Recommend Topics

Recommend Org