Git Product home page Git Product logo

gerapy's Introduction

Gerapy

Build Read the Docs PyPI - Python Version GitHub stars PyPI - Downloads Docker Pulls PyPI - License

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js.

Documentation

Documentation is available online at https://docs.gerapy.com/ and https://github.com/Gerapy/Docs.

Support

Gerapy is developed based on Python 3.x. Python 2.x may be supported later.

Usage

Install Gerapy by pip:

pip3 install gerapy

After the installation, you need to do these things below to run Gerapy server:

If you have installed Gerapy successfully, you can use command gerapy. If not, check the installation.

First use this command to initialize the workspace:

gerapy init

Now you will get a folder named gerapy. Also you can specify the name of your workspace by this command:

gerapy init <workspace>

Then cd to this folder, and run this command to initialize the Database:

cd gerapy
gerapy migrate

Next you need to create a superuser by this command:

gerapy createsuperuser

Then you can runserver by this command:

gerapy runserver

Then you can visit http://localhost:8000 to enjoy it. Also you can vist http://localhost:8000/admin to get the admin management backend.

If you want to run Gerapy in public, just run like this:

gerapy runserver 0.0.0.0:8000

Then it will run with public host and port 8000.

In Gerapy, You can create a configurable project and then configure and generate code of Scrapy automatically. But this module is unstable, we're trying to refine it.

Also you can drag your Scrapy Project to projects folder. Then refresh web, it will appear in the Project Index Page and comes to un-configurable, but you can edit this project through the web page.

As for deployment, you can move to Deploy Page. Firstly you need to build your project and add client in the Client Index Page, then you can deploy the project just by clicking button.

After the deployment, you can manage the job in Monitor Page.

Docker

Just run this command:

docker-compose up

Then it will run at port 8000. You can use the temp admin account (username: admin, password: admin) to login. And please change the password later for safety.

Command Usage:

docker run -d -v <workspace>:/home/gerapy -p <public_port>:<container_port> germey/gerapy

Please specify your workspace to mount Gerapy workspace by -v <workspace>:/app/gerapy and specify server port by -p <public_port>:<container_port>.

If you run Gerapy by Docker, you can visit Gerapy website such as http://localhost:8000 and enjoy it, no need to do other initialzation things.

TodoList

  • Add Visual Configuration of Spider with Previewing Website
  • Add Scrapyd Auth Management
  • Add Gerapy Auth Management
  • Add Timed Task Scheduler
  • Add Visual Configuration of Scrapy
  • Add Intelligent Analysis of Web Page

Communication

If you have any questions or ideas, you can send Issues or Pull Requests, your suggestions are really import for us, thanks for your contirbution.

gerapy's People

Contributors

6run0 avatar bardusco avatar dependabot[bot] avatar ekkog avatar germey avatar hakandilek avatar jiangslee avatar luopeixiong avatar meepoaii avatar thsheep avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gerapy's Issues

学习中

python+vue+django 非常不错,学习

一点愚见,崔大

首先,作为崔大的学员。希望这个项目能更好。下面是我的一点愚见

看了TodoList。 感觉最紧要需要的功能,不应该是可视化创建爬虫什么的。这是一个分布式部署项目,不是八爪鱼之类的可视化爬虫生成项目。 我个人觉得目前最应该更新的功能如下:

  1. scrapyd 认证,或者是 gerapy 认证。 加scrapyd认证,那么gerapy就放在本地,加gerapy认证,则gerapy放服务器上,都能实现基本的安全认证。不然这个东西根本没办法在生成环境中使用。 就只是一个玩玩的东西

  2. 定时任务,生产中最基本的功能,没有这个就不会有人使用gerapy。基本事实就是这样。

希望崔大,先完成这两个基本功能。 别的功能都只是锦上添花,基本的是根基啊。
最后,希望崔大好好干,不断壮大gerapy,写一个广受使用的分布式爬虫部署管理平台。

django runserver does not find module

Hi,
I was trying to run django server but I am receiving the below error.
Could you please have look into the following message.
Should I use gerapy runserver?
Could you tell me how to execute the above command?

Thanks ahead for your help in this regard.

→ ./manage.py runserver
Traceback (most recent call last):
  File "./manage.py", line 27, in <module>
    manage()
  File "./manage.py", line 23, in manage
    execute_from_command_line(sys.argv)
  File "/home/mtaziz/.virtualenvs/onescience_dist_env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/mtaziz/.virtualenvs/onescience_dist_env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 308, in execute
    settings.INSTALLED_APPS
  File "/home/mtaziz/.virtualenvs/onescience_dist_env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/home/mtaziz/.virtualenvs/onescience_dist_env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/mtaziz/.virtualenvs/onescience_dist_env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 110, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named gerapy.server.server.settings

删除client错误

view文件中 Task.objects.filter(client_id=client_id).delete()报错
Task中没有client_id属性

English Language Support Feature

Hi @Germey ,

Hope you are doing great. I am deeply happy to see you continuously working so hard to improve the performance & adding new feature of Gerapy.

I know that this is probably not an ideal question to ask you hereon github issue section but I was wondering if you won't mind to let me know when you are expecting to have English support for such an excellent Framework Gerapy.

"In our earlier conversation", you said that "I'm Chinese from Beijing, China. 😁 If you feel any inconvenience I'm glad to convert it in the next version.".

I am patiently & enthusiastically looking forward to see support for English.

Thank you so much for your dedication, time, effort in building such amazing Framework.

Thank you.

bash: gerapy: command not found

我的机器装有Python2和Python3 我用pip3安装gerapy成功之后 直接输入gerapy提示找不到命令 但是我用pip安装之后直接输入gerapy可以正常运行 这应该是环境变量的问题 然后我export PATH=$PATH:(python3包的安装路径)并且卸载了pip安装的gerapy 再次运行gerapy还是报这个错

翻译成英文

如果我要将所有用户看到的中文翻译成英文,要怎么做?

我现在的想法是使用正则 [\u4e00-\u9fa5] 搜索所有 js 文件,然后一个一个翻译。

Gerapy 第一次generate代码时失败

这里的gerate指的是下面(第一次建立爬虫时):

b5e9c885cc45518cdc3beea212bdfa79

异常如下:

33f7e24aff9cb68032e5bc8b6343a81b

当保存后,也就是第二次进入改项目管理下,generate代码是成功的!!

AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_EVP_PKEY_get_set_tls_encodedpoint'

$ gerapy init
$ cd gerapy/
$ ls
projects
$ gerapy migrate
Traceback (most recent call last):
File "/home/datacrawl/.local/bin/gerapy", line 11, in
sys.exit(cmd())
File "/home/datacrawl/.local/lib/python3.5/site-packages/gerapy/cmd/init.py", line 27, in cmd
server()
File "/home/datacrawl/.local/lib/python3.5/site-packages/gerapy/cmd/server.py", line 6, in server
manage()
File "/home/datacrawl/.local/lib/python3.5/site-packages/gerapy/server/manage.py", line 23, in manage
execute_from_command_line(sys.argv)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/management/base.py", line 332, in execute
self.check()
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/management/base.py", line 364, in check
include_deployment_checks=include_deployment_checks,
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 58, in _run_checks
issues.extend(super()._run_checks(**kwargs))
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/management/base.py", line 351, in _run_checks
return checks.run_checks(**kwargs)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/checks/registry.py", line 73, in run_checks
new_errors = check(app_configs=app_configs)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/core/checks/urls.py", line 23, in check_resolver
return check_method()
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/urls/resolvers.py", line 397, in check
for pattern in self.url_patterns:
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/utils/functional.py", line 36, in get
res = instance.dict[self.name] = self.func(instance)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/urls/resolvers.py", line 536, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/utils/functional.py", line 36, in get
res = instance.dict[self.name] = self.func(instance)
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/urls/resolvers.py", line 529, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/home/datacrawl/.local/lib/python3.5/site-packages/gerapy/server/server/urls.py", line 21, in
url(r'^', include('gerapy.server.core.urls')),
File "/home/datacrawl/.local/lib/python3.5/site-packages/django/urls/conf.py", line 34, in include
urlconf_module = import_module(urlconf_module)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/home/datacrawl/.local/lib/python3.5/site-packages/gerapy/server/core/urls.py", line 2, in
from . import views
File "/home/datacrawl/.local/lib/python3.5/site-packages/gerapy/server/core/views.py", line 1, in
import json, os, requests, time, pytz, pymongo, string
File "/home/datacrawl/.local/lib/python3.5/site-packages/requests/init.py", line 84, in
from urllib3.contrib import pyopenssl
File "/home/datacrawl/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 46, in
import OpenSSL.SSL
File "/home/datacrawl/.local/lib/python3.5/site-packages/OpenSSL/init.py", line 8, in
from OpenSSL import crypto, SSL
File "/home/datacrawl/.local/lib/python3.5/site-packages/OpenSSL/crypto.py", line 16, in
from OpenSSL._util import (
File "/home/datacrawl/.local/lib/python3.5/site-packages/OpenSSL/_util.py", line 6, in
from cryptography.hazmat.bindings.openssl.binding import Binding
File "/home/datacrawl/.local/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 156, in
Binding.init_static_locks()
File "/home/datacrawl/.local/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 137, in init_static_locks
cls._ensure_ffi_initialized()
File "/home/datacrawl/.local/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 124, in _ensure_ffi_initialized
cls.lib = build_conditional_library(lib, CONDITIONAL_NAMES)
File "/home/datacrawl/.local/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 84, in build_conditional_library
if not getattr(lib, condition):
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_EVP_PKEY_get_set_tls_encodedpoint'

Gerapy 已更新到 0.8.3

Gerapy 已更新到 0.8.3
更新如下:

  • 增加Docker支持
  • 通过中间件解决json loads bytes error 的问题
  • 项目删除的bug解决
  • 汉英部分文字不能切换的bug修复
  • 修正了前端的加载bug
  • Log日志文件中文乱码修复
  • Find Egg不存在时异常处理

欢迎大家更新试用,如有bug请发issue,也非常欢迎pr贡献~
Relase:https://github.com/Gerapy/Gerapy/releases
PyPi:https://pypi.python.org/pypi/gerapy
Docker: https://github.com/Gerapy/Gerapy#docker

Django2.0报错

django2.0 在gerapy migrate这一步会报错,降级到django1.11没问题了。

$ gerapy migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, core, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying core.0001_initial... OK
Applying core.0002_auto_20180119_1210... OK
Applying core.0003_auto_20180123_2304... OK
Applying core.0004_auto_20180124_0032...Traceback (most recent call last):
File "/usr/local/bin/gerapy", line 11, in
load_entry_point('gerapy==0.8.5', 'console_scripts', 'gerapy')()
File "/usr/local/lib/python3.5/dist-packages/gerapy/cmd/init.py", line 26, in cmd
server()
File "/usr/local/lib/python3.5/dist-packages/gerapy/cmd/server.py", line 6, in server
manage()
File "/usr/local/lib/python3.5/dist-packages/gerapy/server/manage.py", line 24, in manage
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/dist-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/migrate.py", line 200, in handle
fake_initial=fake_initial,
File "/usr/local/lib/python3.5/dist-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.5/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.5/dist-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python3.5/dist-packages/django/db/migrations/migration.py", line 122, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python3.5/dist-packages/django/db/migrations/operations/fields.py", line 304, in database_forwards
to_model._meta.get_field(self.new_name),
File "/usr/local/lib/python3.5/dist-packages/django/db/backends/sqlite3/schema.py", line 81, in alter_field
any(r.field_name == old_field.name for r in model._meta.related_objects)):
File "/usr/local/lib/python3.5/dist-packages/django/db/backends/sqlite3/schema.py", line 81, in
any(r.field_name == old_field.name for r in model._meta.related_objects)):
AttributeError: 'ManyToManyRel' object has no attribute 'field_name'

gerapy no command

我的centos7.4 and python3.6
pip3 install gerapy
之后
gerapy no command
求解

提个小小想法

加个按钮在管理页面里面,实现从git上拉取到projects文件夹中然后部署。

gerapy init failed complaining cannot import name 'version'

I've recently installed gerapy on a Python 3.6.3 virtual environment. After successful installation I executed:
gerapy init

and got this:
Traceback (most recent call last):
File "/home/muhammad/development/virtualenvs/py3-pocs/bin/gerapy", line 7, in
from gerapy.cmd import cmd
File "/home/muhammad/development/virtualenvs/py3-pocs/lib/python3.6/site-packages/gerapy/cmd/init.py", line 14, in
from gerapy import version
ImportError: cannot import name 'version'

python2部署爬虫错误

python2.7, windows7环境下,使用 gerapy 在打包项目 egg 包成功后的 部署 操作时会报错
命令行显示 "POST /api/client/1/project/ICBR/deploy HTTP/1.1" 500 70
检查发现是 gerapy\server\core\views 文件下
第 336 行 scrapyd.add_version(project_name, int(time.time()), egg_file.read()) 引起的
暂时可以选择注释掉这行,便会部署成功,继续调度,爬虫也能正常运行。

项目文件部分py读取不了

如题:
具体代码PO主可以测试下~
项目在我的github: 地址
(出错原因见下图~)

注: 代码能够部署.就是本地无法通过页面进行编辑.

请求支持设置暂停重启。

现有对于爬虫状态的管理只有停止和run。希望可以添加自定义爬虫开始命令。添加管理状态暂停。

scrapy crawl zhihu -s JOBDIR=job_pause/003

能否添加 爬虫 定时调度 的功能?

这个项目解决了爬虫部署的时候一些手工的操作麻烦的问题(虽然也能用脚本完成,但是不如这个直观),还能统一查看数据;
如果能够添加爬虫的定时调度功能或者有接口就好了。

bugbugbugbug

File "d:\programdata\gerapy\lib\site-packages\gerapy\server\core\views.py", line 454, in project_file_read
return HttpResponse(f.read())
UnicodeDecodeError: 'gbk' codec can't decode byte 0xad in position 88: illegal multibyte sequence

工程文件无法在页面中加载、编辑

添加 client 问题

1.添加 client 的时候,成功了,但是没有跳回到列表页面;
2.能够创建多个同一名称、IP、端口的 client, 虽然id是不一样,但是感觉有点奇怪;

Spider Build Failed

I am having issues in building the spider . Let me explain, for example, how I am running the spider steps given below:

  • First off, command
    gerapy runserver

  • add client

  • copy the spider project in gerapy/projects

  • refresh the front end

  • mifarma_es_spider being appeared on the front end

  • click on deploy

  • write description under Build Project

  • click build

  • getting Failed to Build and log below

[01/Nov/2017 11:26:14] "POST /api/project/mifarma_es_spider/build HTTP/1.1" 500 103186
{'settings': 'mifarma_es_spider.settings', 'project': 'mifarma_es_spider'}
Built mifarma_es_spider into /home/scraper/py36projects/gerapy/projects/mifarma_es_spider/mifarma_es_spider-1.0-py3.6.egg
Internal Server Error: /api/project/mifarma_es_spider/build
Traceback (most recent call last):
  File "/home/scraper/projects/py36env/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/scraper/projects/py36env/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/scraper/projects/py36env/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/scraper/projects/py36env/lib/python3.6/site-packages/gerapy/server/core/views.py", line 394, in project_build
    return JsonResponse(data)
  File "/home/scraper/projects/py36env/lib/python3.6/site-packages/gerapy/server/core/response.py", line 32, in __init__
    data = json.dumps(data, cls=encoder, **json_dumps_params)
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/scraper/projects/py36env/lib/python3.6/site-packages/gerapy/server/core/encoder.py", line 48, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'Client' is not JSON serializable

Screenshot for client  status can be found

I am not sure what causing this.

AttributeError: 'ManyToManyRel' object has no attribute 'field_name'

gerapy migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, core, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying core.0001_initial... OK
Applying core.0002_auto_20180119_1210... OK
Applying core.0003_auto_20180123_2304... OK
Applying core.0004_auto_20180124_0032...Traceback (most recent call last):
File "/root/anaconda3/bin/gerapy", line 11, in
load_entry_point('gerapy==0.8.5', 'console_scripts', 'gerapy')()
File "/root/anaconda3/lib/python3.6/site-packages/gerapy-0.8.5-py3.6.egg/gerapy/cmd/init.py", line 26, in cmd
server()
File "/root/anaconda3/lib/python3.6/site-packages/gerapy-0.8.5-py3.6.egg/gerapy/cmd/server.py", line 6, in server
manage()
File "/root/anaconda3/lib/python3.6/site-packages/gerapy-0.8.5-py3.6.egg/gerapy/server/manage.py", line 24, in manage
execute_from_command_line(sys.argv)
File "/root/anaconda3/lib/python3.6/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/root/anaconda3/lib/python3.6/site-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/root/anaconda3/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/root/anaconda3/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/root/anaconda3/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 200, in handle
fake_initial=fake_initial,
File "/root/anaconda3/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/root/anaconda3/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/root/anaconda3/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/root/anaconda3/lib/python3.6/site-packages/django/db/migrations/migration.py", line 122, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/root/anaconda3/lib/python3.6/site-packages/django/db/migrations/operations/fields.py", line 304, in database_forwards
to_model._meta.get_field(self.new_name),
File "/root/anaconda3/lib/python3.6/site-packages/django/db/backends/sqlite3/schema.py", line 81, in alter_field
any(r.field_name == old_field.name for r in model._meta.related_objects)):
File "/root/anaconda3/lib/python3.6/site-packages/django/db/backends/sqlite3/schema.py", line 81, in
any(r.field_name == old_field.name for r in model._meta.related_objects)):
AttributeError: 'ManyToManyRel' object has no attribute 'field_name'

gerapy python2.7的错误怎么解决?

File "D:\python27\lib\site-packages\django\db\backends\base\base.py", line 213, in ensure_connection
self.connect()
File "D:\python27\lib\site-packages\django\db\backends\base\base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "D:\python27\lib\site-packages\django\db\backends\sqlite3\base.py", line 198, in get_new_connection
conn = Database.connect(**conn_params)
django.db.utils.OperationalError: unable to open database file

调度爬虫的时候可否增加 添加参数的功能?

平常调度爬虫的时候会 curl ..... -a arg1=value1 -a arg2=value2 -a arg3=value3 传参数,在使用 Gerapy的时候,调度爬虫,点run后就直接提交运行了。

这个虽然可以直接修改文件里面的参数来做这样的操作,但是还是要修改文件代码,总觉得有点麻烦。

要是能直接在调度的时候添加的话就挺好的。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.