Git Product home page Git Product logo

aiocr-api's Introduction

基于LUMEN开发的高性能图片识别平台API接口源码

此项目是整个AI图片 识别项目 的API接口端

原本是作为商业项目开发的,但项目未上线就宣告结束了,出于共同学习的目的,开源此项目

因为项目并未真正上线运营,所以有一些不足或考虑不周全之处请自行忽略,若发现错误,还请指正。

用户管理系统&业务官网源码请移步至 https://github.com/hookover/aiocr-web.git

根据预期业务,将接口设计为:承载每天2000万张图片上传并相应处理的API

API接口 很少

/servers | 获取服务器列表

/user/login | 登录 /user/point?token=:token | 获取用户剩余积分

/decode/upload-base64?token=:token | 以base64字符串方式上传图片 /decode/upload?token=:token | 以form-data数据流方式上传图片

/result?token=:token | 获取识别结果 /report?token=:token | 若图片识别错误,可报错

接口的详细使用,请搭建https://github.com/hookover/aiocr-web.git 后点击API文档查看

技术目标

高并发、高性能、安全稳定、最佳实践方案

用到的技术栈

RESTful规范

JWT,加密算法修改为了最快的SHA256

REDIS长连接,测试了memcached

高性能自增ID生成器:php_snowflke (测试每秒生成10万+不重复ID)

性能分析工具使用的是xhprof

APIDOC 根据代码注释自动生成API文档

MYSQL事件+存储过程 将上传图片日志表自动按天分区,并删除7天之前的分区

MYSQL事件+存储过程 自动统计每个用户/开发工程师每天产生的数据

LUMEN 层面

自定义Auth,将token验证放到了Auth服务模块中
定义了日志服务
统一的API响应错误代码机制
自增ID采用2个数据字段存储,并使用联合索引提升性能

   SQL语句优化    还有一些细节已经忘记了

安装

1、 为php安装此扩展 https://github.com/hookover/php_snowflake.git
2、 克隆此代码并composer安装代码依赖
3、 配置nginx/apache 参考lumen的配置
4、 cp .env.example .env
5、 项目数据表的创建和基础数据在这里: https://github.com/hookover/aiocr-web.git,建议先搭建业务系统再搭建API

代码层面优化

借助xhprof查看性能瓶颈,根据性能瓶颈提出解决方案,并实施,最终实现单个API请求处理在10ms左右,数据并发起来,业务逻辑也不会出现任何问题。

高并发是一个系统工程

个人认为,要想让整个系统实现高并发,需要从软件到硬件再到服务器架构配置做好全面协调,任何一个地方的短板都可能成为整个系统的短板。

代码层面无非就是减少对数据库的读写压力、将一些频繁数据放入缓存,PHP到缓存到数据库都通过长连接提高连接速度,数据库表合理的使用好索引,注意SQL质量和调整好一些耗费资源的算法。

服务器软件配置方面又会涉及到LINUX本身的优化,服务软件之间的协调,数据库之间的协调,负载均衡等。

硬件层面又需要考虑CPU瓶颈,硬盘读写性能,内存大小,服务器间网络通信状况等..

简单压测

开发机: Intel® Core™ i7-4770 CPU @ 3.40GHz × 8 + 15.6 GiB 内存

环境:ubuntu + nginx + php7 + php-fpm tcp

压测工具: ab

开发机上同时运行了phpstorm,网页等

nginx和php-fpm是使用的apt安装后的默认配置,未做深入调优

若需要下载代码自己压测,或者真正到线上运行, 还需要调整nginx、php、php-fpm、mysql、redis的配置,以便让这些服务器软件协同工作。

测试数据:

100并发1万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      100
Time taken for tests:   10.254 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      2450000 bytes
HTML transferred:       820000 bytes
Requests per second:    975.25 [#/sec] (mean)
Time per request:       102.537 [ms] (mean)
Time per request:       1.025 [ms] (mean, across all concurrent requests)
Transfer rate:          233.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:     6  102 267.1     31    4582
Waiting:        6  102 267.1     31    4582
Total:          6  102 267.1     31    4583

200并发10万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      200
Time taken for tests:   104.265 seconds
Complete requests:      100000
Failed requests:        17
   (Connect: 0, Receive: 0, Length: 17, Exceptions: 0)
Non-2xx responses:      17
Total transferred:      24501853 bytes
HTML transferred:       8201717 bytes
Requests per second:    959.09 [#/sec] (mean)
Time per request:       208.531 [ms] (mean)
Time per request:       1.043 [ms] (mean, across all concurrent requests)
Transfer rate:          229.49 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     6  208 1630.6     43   60001
Waiting:        6  208 1630.6     43   60001
Total:          6  208 1630.6     44   60001

500并发,10万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      500
Time taken for tests:   106.200 seconds
Complete requests:      100000
Failed requests:        114
   (Connect: 0, Receive: 0, Length: 114, Exceptions: 0)
Non-2xx responses:      114
Total transferred:      24511451 bytes
HTML transferred:       8210864 bytes
Requests per second:    941.62 [#/sec] (mean)
Time per request:       531.001 [ms] (mean)
Time per request:       1.062 [ms] (mean, across all concurrent requests)
Transfer rate:          225.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0      14
Processing:     5  530 3091.5    151   63039
Waiting:        5  530 3091.5    151   63039
Total:          5  530 3091.9    151   63049

1000并发1万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      1000
Time taken for tests:   27.926 seconds
Complete requests:      10000
Failed requests:        64
   (Connect: 0, Receive: 0, Length: 64, Exceptions: 0)
Non-2xx responses:      64
Total transferred:      2456016 bytes
HTML transferred:       825824 bytes
Requests per second:    358.09 [#/sec] (mean)
Time per request:       2792.559 [ms] (mean)
Time per request:       2.793 [ms] (mean, across all concurrent requests)
Transfer rate:          85.89 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   6.7      0      27
Processing:    62 1170 3005.8    256   27883
Waiting:       62 1170 3005.8    256   27883
Total:         62 1172 3008.8    256   27906

aiocr-api's People

Contributors

hookover avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.