Git Product home page Git Product logo

accountpool's Introduction

AccountPool

build Docker Pulls

简易高效的账号池,提供如下功能:

  • 定时模拟登录账号,将 Cookies 或 JWT 等信息存储到 Redis 数据库。
  • 定时测试,剔除不可用 Cookies 或 JWT。
  • 提供 API,随机取用测试通过的可用 Cookies 或 JWT。

使用要求

可以通过两种方式来运行账号池,一种方式是使用 Docker(推荐),另一种方式是常规方式运行。

Docker

如果使用 Docker,则需要安装如下环境:

  • Docker
  • Docker-Compose

常规方式

常规方式要求有 Python 环境、Redis 环境,具体要求如下:

  • Python>=3.6
  • Redis

Docker 运行

如果安装好了 Docker 和 Docker-Compose,只需要一条命令即可运行。

docker-compose up

运行结果类似如下:

redis4accountpool is up-to-date
Recreating accountpool ... done
Attaching to redis4accountpool, accountpool
redis4accountpool    | 1:C 31 Aug 2023 03:53:10.335 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis4accountpool    | 1:C 31 Aug 2023 03:53:10.335 * Redis version=7.2.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis4accountpool    | 1:C 31 Aug 2023 03:53:10.335 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis4accountpool    | 1:M 31 Aug 2023 03:53:10.335 * monotonic clock: POSIX clock_gettime
redis4accountpool    | 1:M 31 Aug 2023 03:53:10.336 * Running mode=standalone, port=6379.
redis4accountpool    | 1:M 31 Aug 2023 03:53:10.336 * Server initialized
redis4accountpool    | 1:M 31 Aug 2023 03:53:10.336 * Ready to accept connections tcp
redis4accountpool    | 1:C 31 Aug 2023 04:03:11.226 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis4accountpool    | 1:C 31 Aug 2023 04:03:11.226 * Redis version=7.2.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis4accountpool    | 1:C 31 Aug 2023 04:03:11.226 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis4accountpool    | 1:M 31 Aug 2023 04:03:11.226 * monotonic clock: POSIX clock_gettime
redis4accountpool    | 1:M 31 Aug 2023 04:03:11.227 * Running mode=standalone, port=6379.
redis4accountpool    | 1:M 31 Aug 2023 04:03:11.227 * Server initialized
redis4accountpool    | 1:M 31 Aug 2023 04:03:11.227 * Ready to accept connections tcp
accountpool          | 2023-08-31 04:06:20,737 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
accountpool          | 2023-08-31 04:06:20,739 INFO supervisord started with pid 1
accountpool          | 2023-08-31 04:06:21,742 INFO spawned: 'generator' with pid 10
accountpool          | 2023-08-31 04:06:21,744 INFO spawned: 'server' with pid 11
accountpool          | 2023-08-31 04:06:21,746 INFO spawned: 'tester' with pid 12
accountpool          | 2023-08-31 04:06:21.990 | DEBUG    | accountpool.scheduler:run_tester:31 - tester loop 0 start...
accountpool          | 2023-08-31 04:06:21.990 | DEBUG    | accountpool.scheduler:run_generator:46 - getter loop 0 start...
accountpool          |  * Running on all addresses.
accountpool          |    WARNING: This is a development server. Do not use it in a production deployment.
accountpool          |  * Running on http://172.24.0.3:6777/ (Press CTRL+C to quit)
accountpool          | 2023-08-31 04:06:22.004 | DEBUG    | accountpool.processors.generator:run:39 - start to run generator
accountpool          | 2023-08-31 04:06:22.005 | DEBUG    | accountpool.processors.generator:run:43 - start to generate credential of admin1
accountpool          | 2023-08-31 04:06:23,007 INFO success: generator entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
accountpool          | 2023-08-31 04:06:23,007 INFO success: server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
accountpool          | 2023-08-31 04:06:23,007 INFO success: tester entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

可以看到 Redis、Generator、Server、Tester 都已经启动成功。

另外还需要导入一些账号信息到 Redis 数据库里面,由于已经用 Docker 启动了 Redis 数据库,运行在 6333 端口上。

这时候可以执行脚本:

export REDIS_PORT=6333
python3 importer.py antispider7

运行完成之后如果没有报错就说明账号导入成功了,可以自行连上 Redis 看下。

过一会访问 http://localhost:6777/antispider7/random 即可获取一个 antispider7 的随机可用 Cookies。

常规方式运行

如果不使用 Docker 运行,配置好 Python、Redis 环境之后也可运行,步骤如下。

安装和配置 Redis

本地安装 Redis、Docker 启动 Redis、远程 Redis 都是可以的,只要能正常连接使用即可。

首先可以需要一下环境变量,代理池会通过环境变量读取这些值。

设置 Redis 的环境变量有两种方式,一种是分别设置 host、port、password,另一种是设置连接字符串,设置方法分别如下:

设置 host、port、password,如果 password 为空可以设置为空字符串,示例如下:

export REDIS_HOST='localhost'
export REDIS_PORT=6379
export REDIS_PASSWORD=''
export REDIS_DB=0

或者只设置连接字符串:

export REDIS_CONNECTION_STRING='redis://[password]@host:port/db'

如果没有密码也要设置为:

export REDIS_CONNECTION_STRING='redis://@host:port/db'

这里连接字符串的格式需要符合 redis://[password]@host:port/db 的格式,注意不要遗漏 @

以上两种设置任选其一即可。

安装依赖包

这里强烈推荐使用 Condavirtualenv 创建虚拟环境,Python 版本不低于 3.6。

然后 pip 安装依赖即可:

pip3 install -r requirements.txt

运行代理池

两种方式运行账号池,一种是 Tester、Generator、Server 全部运行,另一种是按需分别运行。

一般来说可以选择全部运行,命令如下:

python3 run.py <website>

运行之后会启动 Tester、Generator、Server,这时访问 http://localhost:6777//random 即可获取一个随机可用代理。

或者如果你弄清楚了账号池的架构,可以按需分别运行,命令如下:

python3 run.py <website> --processor getter
python3 run.py <website> --processor tester
python3 run.py <website> --processor server

这里 processor 可以指定运行 Tester、Generator 还是 Server。

可配置项

账号池可以通过设置环境变量来配置一些参数。

开关

  • ENABLE_TESTER:允许 Tester 启动,默认 true
  • ENABLE_GENERATOR:允许 Generator 启动,默认 true
  • ENABLE_SERVER:运行 Server 启动,默认 true

环境

  • APP_ENV:运行环境,可以设置 dev、test、prod,即开发、测试、生产环境,默认 dev
  • APP_DEBUG:调试模式,可以设置 true 或 false,默认 true

Redis 连接

  • REDIS_HOST:Redis 的 Host
  • REDIS_PORT:Redis 的端口
  • REDIS_PASSWORD:Redis 的密码
  • REDIS_DB:Redis 的数据库索引,如 0、1
  • REDIS_CONNECTION_STRING:Redis 连接字符串
  • REDIS_KEY:Redis 储存代理使用字典的名称

处理器

  • CYCLE_TESTER:Tester 运行周期,即间隔多久运行一次测试,默认 20 秒
  • CYCLE_GETTER:Getter 运行周期,即间隔多久运行一次代理获取,默认 100 秒
  • API_HOST:代理 Server 运行 Host,默认 0.0.0.0
  • API_PORT:代理 Server 运行端口,默认 6777
  • API_THREADED:代理 Server 是否使用多线程,默认 true

日志

  • LOG_DIR:日志相对路径
  • LOG_RUNTIME_FILE:运行日志文件名称
  • LOG_ERROR_FILE:错误日志文件名称

部署

本项目提供了 Kubernetes 部署脚本,如需部署到 Kubernetes,执行如下命令即可:

cat deployment.yml | sed 's/\${TAG}/latest/g' | kubectl apply -f -

待开发

  • 前端页面管理
  • 使用情况统计分析

如有一起开发的兴趣可以在 Issue 留言,非常感谢!

LICENSE

MIT

accountpool's People

Contributors

germey avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

accountpool's Issues

requirements.txt 中包版本的问题

在requirements.txt 中使用的Flask版本是0.12.1,代码运行的时候会报ImportError: cannot import name 'Markup' from 'jinja2'错误,

Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from accountpool.scheduler import Scheduler
  File "/data/web-spider2/chapter10/10.4/AccountPool/accountpool/scheduler.py", line 3, in <module>
    from accountpool.processors.server import app
  File "/data/web-spider2/chapter10/10.4/AccountPool/accountpool/processors/server.py", line 2, in <module>
    from flask import Flask, g
  File "/root/.virtualenvs/accountpool/lib/python3.8/site-packages/flask/__init__.py", line 19, in <module>
    from jinja2 import Markup, escape
ImportError: cannot import name 'Markup' from 'jinja2' (/root/.virtualenvs/accountpool/lib/python3.8/site-packages/jinja2/__init__.py)

和缺少 MarkupSafe ,报以下错误:

Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from accountpool.scheduler import Scheduler
  File "/data/web-spider2/chapter10/10.4/AccountPool/accountpool/scheduler.py", line 3, in <module>
    from accountpool.processors.server import app
  File "/data/web-spider2/chapter10/10.4/AccountPool/accountpool/processors/server.py", line 2, in <module>
    from flask import Flask, g
  File "/root/.virtualenvs/accountpool/lib/python3.8/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
  File "/root/.virtualenvs/accountpool/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/root/.virtualenvs/accountpool/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/root/.virtualenvs/accountpool/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/root/.virtualenvs/accountpool/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/root/.virtualenvs/accountpool/lib/python3.8/site-packages/markupsafe/__init__.py)

需要把改成 Flask==0.12.1改成Flask==1.1.4,和添加MarkupSafe==2.0.1

KeyError: 'antispider7'

Describe the bug
docker-compose up报错如下
Traceback (most recent call last):
accountpool | File "run.py", line 13, in
accountpool | getattr(Scheduler(), f'run_{args.processor}')(website)
accountpool | File "/app/accountpool/scheduler.py", line 43, in run_generator
accountpool | generator = getattr(generators, GENERATOR_MAP[website])(website)
accountpool | KeyError: 'antispider7'

KeyError: 'accountpool'

Describe the bug
Traceback (most recent call last):
accountpool | File "run.py", line 13, in
accountpool | getattr(Scheduler(), f'run_{args.processor}')(website)
accountpool | File "/app/accountpool/scheduler.py", line 43, in run_generator
accountpool | generator = getattr(generators, GENERATOR_MAP[website])(website)
accountpool | KeyError: 'accountpool'

Environments (please complete the following information):

  • Python:3.6
  • Docker:version 20.10.12, build e91ed57

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.