Git Product home page Git Product logo

movietaster-open's Introduction

MovieTaster-Open

A movie recommend project based on Item2vec.

Reference:

  • Barkan, Oren, and Noam Koenigstein. "Item2vec: neural item embedding for collaborative filtering." Machine Learning for Signal Processing (MLSP), 2016 IEEE 26th International Workshop on. IEEE, 2016.
  • JayveeHe, https://github.com/JayveeHe/MusicTaster. Github.

Demo>

More details for this project, plese refer to blog> or zhihu>

Project Struct

  • datas: to store corpus
  • models: to store item vectors for movies
  • utils: a collection of tool functions

Usage

  1. Compile fasttext under root of the project

  2. Process corpus, the data file would be generated under ./datas/.

$ cd datas && tar -xzvf corpus.tar.gz
$ cd ..
$ python utils/process.py
  1. Train model by running fasttext. Please refer to ./run.sh for other configuration.
$ ./fasttext skipgram -input ./datas/doulist_0804_09.movie_id -output ./models/fasttext_model_0804_09_skipgram -minCount 5 -epoch 50 -neg 100
  1. Here's a result from the default configuration.
$ python utils/eval.py
movie_name_id_dict is 130206 from [./datas/doulist_0804_09.json]
movie_name_id_dict is 130206 from [./datas/doulist_0804_09.json]
movie_id_name_dict is 130206 from [./datas/doulist_0804_09.json]
[2017-08-18 07:52:53,579][pid:22831] eval.topk_like: INFO: [1210]小时代 top 5 likes:
[2017-08-18 07:52:54,149][pid:22831] eval.topk_like: INFO: [2396]小时代2：青木时代 0.820323
[2017-08-18 07:52:54,149][pid:22831] eval.topk_like: INFO: [3387]分手合约 0.606063
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [4087]不二神探 0.604207
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [3839]天台爱情 0.601879
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [821]致我们终将逝去的青春 0.600836
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [144]倩女幽魂 top 5 likes:
[2017-08-18 07:52:54,690][pid:22831] eval.topk_like: INFO: [2719]倩女幽魂3：道道道 0.685621
[2017-08-18 07:52:54,690][pid:22831] eval.topk_like: INFO: [1812]倩女幽魂2：人间道 0.681594
[2017-08-18 07:52:54,691][pid:22831] eval.topk_like: INFO: [466]胭脂扣 0.678032
[2017-08-18 07:52:54,691][pid:22831] eval.topk_like: INFO: [261]青蛇 0.671541
[2017-08-18 07:52:54,692][pid:22831] eval.topk_like: INFO: [156]东邪西毒 0.664057
[2017-08-18 07:52:54,692][pid:22831] eval.topk_like: INFO: [2602]悟空传 top 5 likes:
[2017-08-18 07:52:55,253][pid:22831] eval.topk_like: INFO: [5648]闪光少女 0.671337
[2017-08-18 07:52:55,253][pid:22831] eval.topk_like: INFO: [2189]绣春刀II：修罗战场 0.646861
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [8297]逆时营救 0.634753
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [12571]京城81号Ⅱ 0.625549
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [10545]父子雄兵 0.623032
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [56]美国往事 top 5 likes:
[2017-08-18 07:52:55,789][pid:22831] eval.topk_like: INFO: [18]天堂电影院 0.756449
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [14]辛德勒的名单 0.737502
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [17]教父 0.735216
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [69]闻香识女人 0.734119
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [42]西西里的美丽传说 0.732334
[2017-08-18 07:52:55,791][pid:22831] eval.topk_like: INFO: [2644]战狼2 top 5 likes:
[2017-08-18 07:52:56,346][pid:22831] eval.topk_like: INFO: [1456]大护法 0.612700
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [2107]战狼 0.580637
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [2602]悟空传 0.580365
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [9941]建军大业 0.575422
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [19040]阿唐奇遇 0.573613

MovieTaster-Open

使用Item2Vec做电影推荐

参考:

  • Barkan, Oren, and Noam Koenigstein. "Item2vec: neural item embedding for collaborative filtering." Machine Learning for Signal Processing (MLSP), 2016 IEEE 26th International Workshop on. IEEE, 2016.
  • JayveeHe, https://github.com/JayveeHe/MusicTaster. Github.

Demo>

原理详情请参考>

目录结构

  • datas: 存放语料
  • models: 存放电影向量
  • utils: 工具类，包括生成推荐电影列表等

使用说明

  1. 在根目录下载并编译fasttext

  2. 处理语料。结束后将在./datas/下生成fasttext所需要的格式

$ cd datas && tar -xzvf corpus.tar.gz
$ cd ..
$ python utils/process.py
  1. 参考./run.sh中生成词向量的参数配置，任选一个。
$ ./fasttext skipgram -input ./datas/doulist_0804_09.movie_id -output ./models/fasttext_model_0804_09_skipgram -minCount 5 -epoch 50 -neg 100
  1. 查看模型效果
$ python utils/eval.py
movie_name_id_dict is 130206 from [./datas/doulist_0804_09.json]
movie_name_id_dict is 130206 from [./datas/doulist_0804_09.json]
movie_id_name_dict is 130206 from [./datas/doulist_0804_09.json]
[2017-08-18 07:52:53,579][pid:22831] eval.topk_like: INFO: [1210]小时代 top 5 likes:
[2017-08-18 07:52:54,149][pid:22831] eval.topk_like: INFO: [2396]小时代2：青木时代 0.820323
[2017-08-18 07:52:54,149][pid:22831] eval.topk_like: INFO: [3387]分手合约 0.606063
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [4087]不二神探 0.604207
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [3839]天台爱情 0.601879
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [821]致我们终将逝去的青春 0.600836
[2017-08-18 07:52:54,150][pid:22831] eval.topk_like: INFO: [144]倩女幽魂 top 5 likes:
[2017-08-18 07:52:54,690][pid:22831] eval.topk_like: INFO: [2719]倩女幽魂3：道道道 0.685621
[2017-08-18 07:52:54,690][pid:22831] eval.topk_like: INFO: [1812]倩女幽魂2：人间道 0.681594
[2017-08-18 07:52:54,691][pid:22831] eval.topk_like: INFO: [466]胭脂扣 0.678032
[2017-08-18 07:52:54,691][pid:22831] eval.topk_like: INFO: [261]青蛇 0.671541
[2017-08-18 07:52:54,692][pid:22831] eval.topk_like: INFO: [156]东邪西毒 0.664057
[2017-08-18 07:52:54,692][pid:22831] eval.topk_like: INFO: [2602]悟空传 top 5 likes:
[2017-08-18 07:52:55,253][pid:22831] eval.topk_like: INFO: [5648]闪光少女 0.671337
[2017-08-18 07:52:55,253][pid:22831] eval.topk_like: INFO: [2189]绣春刀II：修罗战场 0.646861
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [8297]逆时营救 0.634753
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [12571]京城81号Ⅱ 0.625549
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [10545]父子雄兵 0.623032
[2017-08-18 07:52:55,254][pid:22831] eval.topk_like: INFO: [56]美国往事 top 5 likes:
[2017-08-18 07:52:55,789][pid:22831] eval.topk_like: INFO: [18]天堂电影院 0.756449
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [14]辛德勒的名单 0.737502
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [17]教父 0.735216
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [69]闻香识女人 0.734119
[2017-08-18 07:52:55,790][pid:22831] eval.topk_like: INFO: [42]西西里的美丽传说 0.732334
[2017-08-18 07:52:55,791][pid:22831] eval.topk_like: INFO: [2644]战狼2 top 5 likes:
[2017-08-18 07:52:56,346][pid:22831] eval.topk_like: INFO: [1456]大护法 0.612700
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [2107]战狼 0.580637
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [2602]悟空传 0.580365
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [9941]建军大业 0.575422
[2017-08-18 07:52:56,347][pid:22831] eval.topk_like: INFO: [19040]阿唐奇遇 0.573613

movietaster-open's People

Contributors

lujiaying avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

karlyang2013 fydlzr sudoconf hjnhjn123 zhangjunqiang tjucxq colinsongf berryhn jiongxiao qq1074123922 wyangbs jiangguangxing music1353 gucasbrg zhihaosun ethaniz duxiangkun yifenzhong1920 lunargithub afcarl diffyzh leempan zackery65 xiaoyesoso captainxudui xueyeelong l-seven renlang97 tiffen lunamachinelearning wusamx redfriday suesueyin linlin994395 yueyedeai lifloveyou hijuly pfjob09 gegetang wanke15 zhouyonglong notfoundgg zdfccdanfeng wqw123 linessiex danisyang zhouhuanirving cuixuage xiaojunzeng jiajiaw yoohu war518 zdpg linuxfl sungirl129 zenghaihong shineloveyc 459548764 jiaojiaolou xxqcheers meteorstone liuxieric123 fjteam ethanhwang2015 jehovahxu yu3401 onlyjibran yukunren ningshiqi michaelldd joetag jepsonwong aramilhan magicleo lxngoddess5321 jiaxinhuang note-hub joelpires yunwgui joingreat baohanddd alieinapril corgilee sczhai ykukey milizaitianbian bdopple lmxhappy iq-scm sharangabo

movietaster-open's Issues

the keras model can not converge to a good result

i have tried your keras code, looks like it can not get good model.
1.would you mind to post some keras model results?
2.did you cut the doulist if it contains few movies. this OP may be useful?
thx in advance.

电影的category有全的吗

movie_0849_09.json文件里一部电影只有一种类型，但实际上一部电影有好几种类型，有这种数据集吗，想要跑跑精度，但是类型不全

process2corpus() 方法keyerror

process2corpus() 方法keyerror

movie_name_id_dict is 130206 from [doulist_0804_09.json]
total movie is 130206 from [doulist_0804_09.json], [movie_0804_09.json]

KeyError Traceback (most recent call last)
in ()
1 if name == 'main':
----> 2 process2corpus()

1 frames
in (.0)
7 doulist_dict = json.loads(line.strip())
8 doulist_movies = [.encode('utf8') for _ in doulist_dict['movie_names']]
----> 9 doulist_movie_ids = [str(movie_name_id_dict[]) for _ in doulist_movies]
10 fwrite.write('%s\n' % ('\t'.join(doulist_movies)))
11 fwrite_1.write('%s\n' % (' '.join(doulist_movie_ids)))

KeyError: b'\xe7\x9b\x96\xe6\x96\x87\xc2\xb7\xe6\x96\xaf\xe9\x80\x9a\xe5\xa4\x8d\xe6\xb4\xbb'

run.sh怎么运行？

譬如 ./fasttext skipgram -input ./datas/doulist_0804_09.movie_id -output ./models/fasttext_model_0804_09_skipgram -minCount 5 -epoch 50 -neg 100
报错：
'.' 不是内部或外部命令，也不是可运行的程序
或批处理文件。

模型问题

从模型训练来看，我没有找到什么地方用了movie_0804_09.json这个数据，只是用了电影的名字，也就是说最后出来的向量其实是根据豆列里面的相似电影名称，只是用了名称，训练出来的向量？

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.