howie6879 / owllook Goto Github PK

python 小白，不是很懂，原本准备自己造一个小说采集器，故找到了这个项目

环境是 ubuntu，安装了 python3，通过如下的命令启动
启动命令：python3 novels_search/server.py

2017-04-05 03:57:05,659 WARNING aiocache.log(13) | cPickle module not found, using pickle
Traceback (most recent call last):
File "novels_search/server.py", line 12, in
from novels_search.views.novels_blueprint import novels_bp
ImportError: No module named 'novels_search'

安装模块的时候，是不是哪里不对，没安装成功
root@localhost:~/novels-search# python3 setup.py install
running install
Checking .pth file support in /usr/local/lib/python3.5/dist-packages/
/usr/bin/python3 -E -c pass
TEST PASSED: /usr/local/lib/python3.5/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing novels_search.egg-info/PKG-INFO
writing dependency_links to novels_search.egg-info/dependency_links.txt
writing top-level names to novels_search.egg-info/top_level.txt
reading manifest file 'novels_search.egg-info/SOURCES.txt'
writing manifest file 'novels_search.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying novels_search.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying novels_search.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying novels_search.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying novels_search.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/novels_search-0.0.0-py3.5.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing novels_search-0.0.0-py3.5.egg
Removing /usr/local/lib/python3.5/dist-packages/novels_search-0.0.0-py3.5.egg
Copying novels_search-0.0.0-py3.5.egg to /usr/local/lib/python3.5/dist-packages
novels-search 0.0.0 is already the active version in easy-install.pth

Installed /usr/local/lib/python3.5/dist-packages/novels_search-0.0.0-py3.5.egg
Processing dependencies for novels-search==0.0.0
Finished processing dependencies for novels-search==0.0.0

你好,我很高兴可以找到类似的项目.
我编译时出现了问题:
/Users/yuhang/Android/python/owllook-master/venv/bin/python /Users/yuhang/Android/python/owllook-master/owllook/server.py
msgpack not installed, MsgPackSerializer unavailable
[2018-11-05 17:14:14,818] 37407-INFO init::load_config():l24: owllook 启动模式：DEV
Traceback (most recent call last):
File "/Users/yuhang/Android/python/owllook-master/owllook/server.py", line 12, in
from owllook.views import admin_bp, api_bp, except_bp, md_bp, novels_bp, operate_bp
File "/Users/yuhang/Android/python/owllook-master/owllook/views/init.py", line 1, in
from .admin_blueprint import admin_bp
File "/Users/yuhang/Android/python/owllook-master/owllook/views/admin_blueprint.py", line 9, in
from owllook.fetcher.cache import get_the_latest_chapter
File "/Users/yuhang/Android/python/owllook-master/owllook/fetcher/cache.py", line 16, in
from owllook.fetcher.decorators import cached
File "/Users/yuhang/Android/python/owllook-master/owllook/fetcher/decorators.py", line 10, in
from aiocache.utils import get_args_dict, get_cache
ModuleNotFoundError: No module named 'aiocache.utils'

进入到 decorators.py文件发现有问题文件过时了

from aiocache.log import logger
from aiocache.utils import get_args_dict, get_cache

导入文件的时候 logger get_args_dict, get_cache 这两个方法没有

目前最新的方法是封装成了类.
我不知道该怎么解决.可以帮助我吗

最新的:
https://github.com/argaen/aiocache/blob/634348f40ce8caa01c7c35010acf32d8c3e17ba6/aiocache/decorators.py

把mongodb和redis都加入到docker中，然后可以直接运行，不用装mongodb和redis

docker-compose.yml中的‘’都需要删掉 ，等号后面空着就好 ，不能用‘’搞什么空字符串 ，不然会报错

无法访问 ，阿里云的防火墙里面设置了端口，不知道怎么绑定IP
直接用IP：端口 访问，无法访问....

在移动端下体验非常差，想要优化下这部分结构。

Only the sanic standard log is output

pip3 install -r requirements.txt
执行报错是哪里错了吗....这个错误我百度谷歌了半天都没找到答案..头疼

你好，我的部署方式为docker-compose，具体截图如下：

目前的问题是，排行榜打开后数据为空，具体截图如下：

麻烦排查一下是不是部署方式存在问题导致的！

rt

• owllook数据排行
• 起点
• 纵横百度月票榜：已经失效 故去除
• 纵横

Create a service:

sudo vi /etc/systemd/system/rc-local.service
Add your code there:
[Unit]
Description=/etc/rc.local Compatibility
ConditionPathExists=/etc/rc.local
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99
[Install]
WantedBy=multi-user.target

Create and make sure /etc/rc.local is executable and add this code inside it:

sudo chmod +x /etc/rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

exit 0

Enable the service:
sudo systemctl enable rc-local
Start service and check status:

sudo systemctl start rc-local.service
sudo systemctl status rc-local.service

/etc/rc.local中这么写的。

发现不行，也没有提示错误信息。
求帮助

时不时会报这个错误，是啥原因造成的呢？
而且我也没有找到requirements.txt。

ValueError
Set of coroutines/Futures is empty.
Traceback (most recent call last):

File /Users/cals/.local/share/virtualenvs/Owllook-4Lm_G_fn/lib/python3.6/site-packages/sanic/app.py, line 556, in handle_request

response = await response

File /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py, line 110, in __next__

return self.gen.send(None)

File /Users/cals/PycharmProjects/Owllook/owllook/views/novels_blueprint.py, line 303, in owllook_search

parse_result = await get_novels_info(class_name='baidu', novels_name=novels_name)

File /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py, line 110, in __next__

return self.gen.send(None)

File /Users/cals/PycharmProjects/Owllook/owllook/fetcher/novels_tools.py, line 13, in get_novels_info

novels_info = await novels_module.start(novels_name)

File /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py, line 110, in __next__

return self.gen.send(None)

File /Users/cals/PycharmProjects/Owllook/owllook/fetcher/decorators.py, line 128, in wrapper

result = await func(*args, **kwargs)

File /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py, line 110, in __next__

return self.gen.send(None)

File /Users/cals/PycharmProjects/Owllook/owllook/fetcher/novels_factory/baidu_novels.py, line 101, in start

return await BaiduNovels.start(novels_name)

File /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py, line 110, in __next__

return self.gen.send(None)

File /Users/cals/PycharmProjects/Owllook/owllook/fetcher/novels_factory/base_novels.py, line 48, in start

return await cls().novels_search(novels_name)

File /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py, line 110, in __next__

return self.gen.send(None)

File /Users/cals/PycharmProjects/Owllook/owllook/fetcher/novels_factory/baidu_novels.py, line 88, in novels_search

done_list, pending_list = await asyncio.wait(tasks)

File /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py, line 304, in wait

raise ValueError('Set of coroutines/Futures is empty.')

ValueError: Set of coroutines/Futures is empty. while handling path /search

uvloop是不是不支持windows系统..

我在VPS上成功安装和部署了项目， VPS是可以访问的，但是在别的地址访问就是会跳转到https://www.owllook.net/ 请问一下这个是在什么地方配置

直接下载镜像
docker pull howie6879/owllook
创建dev_owllook.env文件
vim dev_owllook.env
写入一些环境变量，改成了我的设置，如下
MODE=DEV
REDIS_ENDPOINT= 127.0.0.1
REDIS_PORT= 6379
REDIS_PASSWORD=''
MONGO_HOST= 127.0.0.1
MONGO_PORT= 27017
MONGO_USERNAME=''
MONGO_PASSWORD=''
运行下面的
docker run --env-file ./dev_owllook.env -d -p 8001:8001 howie6879/owllook:lates
访问
ip:8001
为什么会跳转到https://www.owllook.net，你的网站了。
哪里有出错了？
谢谢

我有一个想法，仅供参考。
目前的分类机制是对于一个书名，有不同的提供网站。
但实际上每本书都有一个或几个官方发布网站，网站可以抓取到书的目录和每一章的前两段，那么
1.通过网友的共同劳动，像Wiki一样建立一个书名与官方发布页面地址的映射目录。
2.抓取官方发布的目录建立书籍的目录，然后通过每章前两段内容与其他网站内容的比对抽取完整章节。
而不用显示多个源。

可能存在以下问题
1.抓取官方目录和前两段可能涉嫌侵权。
2.文字对比可能正确率低。

我又用debian9试着装，
debian9上安装了宝塔面板，搭建了lnmp
在安装你的教程，安装，都蛮顺利的，到这里
(python36) root@ip-172-26-13-169:/owllook# pipenv run pip install pip==18.0
Creating a virtualenv for this project…
Pipfile: /root/owllook/Pipfile
Using /root/anaconda3/envs/python36/bin/python3.6m (3.6.8) to create virtualenv…
⠦ Creating virtual environment...Using base prefix '/root/anaconda3/envs/python36'
New python executable in /root/.local/share/virtualenvs/owllook-kjRL-ddR/bin/python3.6m
Also creating executable in /root/.local/share/virtualenvs/owllook-kjRL-ddR/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /root/anaconda3/envs/python36/bin/python3.6m
✔ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/owllook-kjRL-ddR
Collecting pip==18.0
Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
100% |████████████████████████████████| 1.3MB 19.0MB/s
Installing collected packages: pip
Found existing installation: pip 19.0
Uninstalling pip-19.0:
Successfully uninstalled pip-19.0
Successfully installed pip-18.0
(python36) root@ip-172-26-13-169:/owllook# pipenv install
Installing dependencies from Pipfile.lock (39f5cc)…
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 15/52

进行到15/52这里时，不动了，不知道是什么情况？

报错信息如下：
novel | [2018-12-14 13:31:49,682] 12-INFO log::info():l15: GET: http://r.qidian.com/?chn=1 novel | [2018-12-14 13:31:49,974] 12-ERROR log::exception():l30: there are no users authenticated novel | Traceback (most recent call last): novel | File "/opt/owllook/spiders/qidian_ranking.py", line 93, in save novel | upsert=True) novel | File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run novel | result = self.fn(*self.args, **self.kwargs) novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/collection.py", line 995, in update_one novel | session=session), novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/collection.py", line 851, in _update_retryable novel | _update, session) novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1243, in _retryable_write novel | return self._retry_with_session(retryable, func, s, None) novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1196, in _retry_with_session novel | return func(session, sock_info, retryable) novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/collection.py", line 847, in _update novel | retryable_write=retryable_write) novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/collection.py", line 817, in _update novel | retryable_write=retryable_write).copy() novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/pool.py", line 579, in command novel | unacknowledged=unacknowledged) novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/network.py", line 150, in command novel | parse_write_concern_error=parse_write_concern_error) novel | File "/root/.local/share/virtualenvs/opt-zvmYt2-H/lib/python3.6/site-packages/pymongo/helpers.py", line 155, in _check_command_response novel | raise OperationFailure(msg % errmsg, code, response) novel | pymongo.errors.OperationFailure: there are no users authenticated

建议每次发docker包使用新的tag ，这样可以使用指定的tag运行 ，现在看tag全都是latest

有些网站看书可以解析出来目录 ，点击目录之后出现500异常，建议增加检测机制检测是否可以看

加入书架按钮从源码上来看是

<p>状    态：连载中,
  <a href="javascript:;" onclick="addBookCase(43995);" rel="nofollow">加入书架</a>,
  <a href="#footer">直达底部</a>
</p>

而 这个 href="javascript:;" 在浏览器（chrome, firefox）中均被解析为http://www.owllook.net/owllook_content?url=http://www.23txt.com/javascript:;&name=...打开了一个新页面，而没有触发 addBookCase 函数

改成 href="javascript:void(0)" 应该就ok了:smile:

1. 上下章按钮样式无效，点击没有反应，https://www.owllook.net/owllook_content?url=http://www.jueshitangmen.info/doupocangqiong/6105.html&name=%E7%AC%AC14%E7%AB%A0%20%E5%90%B8%E6%8E%8C&chapter_url=http://www.jueshitangmen.info/doupocangqiong/&novels_name=%E6%96%97%E7%A0%B4%E8%8B%8D%E7%A9%B9
2. 上下章按钮只能翻一章，之后点击没有反应，https://www.owllook.net/owllook_content?url=https://www.ybdu.com/xiaoshuo/22/22829/9890703.html&name=34%E3%80%81%E7%AC%AC%E4%BA%94%E9%A2%97%E6%98%9F%E8%BE%B0&chapter_url=https://www.ybdu.com/xiaoshuo/22/22829/&novels_name=%E5%A4%A7%E7%8E%8B%E9%A5%B6%E5%91%BD 搜狗浏览器PC版
3. 点击翻页按钮三次以上，标题更换到下一页，但是内容没变https://www.owllook.net/owllook_content?url=https://www.ybdu.com/xiaoshuo/8/8473/1578138.html&chapter_url=https://www.ybdu.com/xiaoshuo/8/8473/&novels_name=%E6%88%91%E7%9A%84%E8%BA%AB%E4%B8%8A%E6%9C%89%E7%BE%8E%E5%A5%B3，而且页面没有回到顶部
   4.猎文网章节显示不正常，https://www.owllook.net/chapter?url=https://www.liewen.cc/b/0/952/&novels_name=%E6%88%91%E7%9A%84%E7%BE%8E%E5%A5%B3%E6%83%85%E5%8A%AB

目前开发完成的排行榜包含起点排行和纵横排行 ，可能很多的人比较热衷于看全本 ，建议开发一个全本排行

如题：python3 server.py出现如下错误：

owllook/fetcher/cache.py", line 52, in wrapper
    if await cache_instance.exists(cache_key):
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 105, in __next__
    return self.gen.send(None)
......
exists = await redis.exists(key)
AttributeError: 'RedisConnection' object has no attribute 'exists'

版本：Python 3.5.2

按正常方式安装的屏蔽了redirect，host已经改为0.0.0.0
python server.py 之后就报错
KeyError
'session'
Traceback (most recent call last):
File /root/.local/share/virtualenvs/owllook-kjRL-ddR/lib/python3.6/site-packages/sanic/app.py, line 556, in handle_request
response = await response
File /root/.miniconda3/lib/python3.6/asyncio/coroutines.py, line 110, in next
return self.gen.send(None)
File /root/owllook/owllook/views/novels_blueprint.py, line 82, in index
user = request['session'].get('user', None)
KeyError: 'session' while handling path /

欢迎提建议，为了使项目更好，谢谢。
这里提供多种解决方案：https://github.com/owllook-for-you

推荐源biqugexsw没有解析成功
https://www.owllook.net/chapter?url=http://www.biqugexsw.com/1_1020/&novels_name=%E4%B8%87%E7%95%8C%E5%A4%A9%E5%B0%8A

在运行爬虫python3 ../../owllook/spiders/spider_console.py 时会报如题的错误，这是什么原因

如题，代码里有如下：
if mode == 'PRO':
from .pro_config import ProConfig
return ProConfig

https://www.x23us.com/html/66/66656/

rt

rt

#39
对于哪家榜单比较有说服力，哪位有研究可提供一下

可以提供一些服务端api，供移动端app使用

用docker创建执行之后，http://127.0.0.1:8001/ 报错Internal Server Error，这是什么原因，刚接触python。

看书过程中经常会出现莫名弹窗，疑是爬取的页面内容中包含script信息（未确认），建议对script标签进行预处理。
访问地址：http://novel.mikepeng.cn/owllook_content?url=https://www.ybdu.com/xiaoshuo/11/11386/2482438.html&chapter_url=https://www.ybdu.com/xiaoshuo/11/11386/&novels_name=%E5%A8%B1%E4%B9%90%E5%9C%88%E5%A4%96%E6%8C%82%E5%85%89%E7%8E%AF

建议收录一些文学站点（我搜到过《活着》，但是打开提示Internal Server Error），增加用户面

我如果设置在物理接口上，直接就是一个302重定向，是不是只能工作在本地的loopback接口呢？

谢谢

且用且珍惜吧
https://www.owllook.net/

出现bug

Internal Server Error

The server encountered an internal error and cannot complete your request.

将功能转移到实验室页面https://www.owllook.net/md

注册用户才有的功能

旧功能转移：

• 书架
• 书签
• 书友推荐
• 章节以及内容更加智能的获取方式
• 搜索展示

新功能添加：

• 个人设置
• 修改密码
• 主题切换
• 作者新书更新提醒
• 书单
• 推书
  ......
  后续更新

https://mp.weixin.qq.com/s/0CqLiKsyDQ-pVmeo3R-UlA

用你公众号上的教程安装
请问，安装好后，访问地址是什么？
运行owllook：
cd owllook
python run.py
怎么才能加入系统服务，随机器启动？

直接替换成书名+全文阅读就可以，这样对小说和图书都有效果，不然太过侧重小说