anhnongdan / cbimax Goto Github PK
View Code? Open in Web Editor NEWOfficial source code for cBIMAX - the analytics system for CDN
Official source code for cBIMAX - the analytics system for CDN
When change to Display simple table, widget goes blank.
After that choosing 'vertical bar graph' causes 500 error.
Reloading page or dragging the widget away will fix the error.
++ Context: PW5 on VNPT 90 is mistakenly excluded from archive, PW5 on 89 is archived as normal. 90 runs on 5 php child while 89 has standard dual 25 pools.
https://bitbucket.org/anhnongdan/bimax_seed/issues/41/php-compare-performance-of-php-fpm
PW5 90 after running a long archive ~30m, show numbers with quite good accuracy
At 06-29 11:12AM, latest Done Archiving! is at:
(19:40 ICT) - No error log showed
(/var/log/flow.log)
INFO [2017-06-28 12:40:06] Archived website id = 5232, 1 API requests, Time elapsed: 2.211s [1/1 done]
INFO [2017-06-28 12:40:06] Done archiving!
INFO [2017-06-28 12:40:06] ---------------------------
INFO [2017-06-28 12:40:06] SUMMARY
INFO [2017-06-28 12:40:06] Total visits for today across archived websites: 1077
INFO [2017-06-28 12:40:06] Archived today's reports for 1 websites
INFO [2017-06-28 12:40:06] Archived week/month/year for 1 websites
All subsequence archive can't finish:
INFO [2017-06-29 04:10:03] - Archiving was last executed without error 15 hours 29 min ago
INFO [2017-06-29 04:10:03] - Will only process the following periods: day (--force-periods)
INFO [2017-06-29 04:10:03] - Will process 1 websites (--force-idsites)
Done Archive
Thu Jun 29 11:10:03 ICT 2017: 2 seconds
DataTable returned by Referrers' API has subtables, filter 'ColumnDelete' doesn't work well with getReferrerType.
Real time thu-put and DL-speed of cBimax is retrieved from the counter which receives both raw log from syslog-ng of both VNPT and FPT
=> QoS get the total Number for both
Piwik calculate both
=> All numbers are consistent.
Should we separate this or not?
Archive seems not to be running.
No log in /var/log/flow.log
Rotation and archiving halt:
| piwik_log_link_visit_action_2017_06_23_11_0 |
| piwik_log_link_visit_action_2017_06_23_11_10 |
| piwik_log_link_visit_action_2017_06_23_11_20 |
| piwik_log_link_visit_action_2017_06_23_11_30 |
| piwik_log_link_visit_action_2017_06_23_11_40 |
| piwik_log_link_visit_action_tracker
=> archive tables missed:
| piwik_archive_numeric_temp_2017_06_18 |
| piwik_archive_numeric_temp_2017_06_19 |
| piwik_archive_numeric_temp_2017_06_20 |
| piwik_archive_numeric_temp_2017_06_21 |
| piwik_archive_numeric_temp_2017_06_22 |
| piwik_archive_numeric_temp_2017_06_26 |
| piwik_archive_numeric_temp_2017_06_30 |
Count: piwik_log_link_visit_action_tracker: 409535, piwik_log_link_visit_action: 323031
Docker compose logs: pw instance log dominates, no log from db (and redis):
pw1_1 | Jun 30 06:39:01 VT-HNI-BIMAX-01 CRON[32541]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1 | Jun 30 06:46:47 VT-HNI-BIMAX-01 CRON[13391]: (CRON) info (No MTA installed, discarding output)
pw1_1 | Jun 30 07:09:01 VT-HNI-BIMAX-01 CRON[8970]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1 | Jun 30 07:17:01 VT-HNI-BIMAX-01 CRON[19972]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
pw1_1 | Jun 30 07:39:01 VT-HNI-BIMAX-01 CRON[17713]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1 | Jun 30 08:09:01 VT-HNI-BIMAX-01 CRON[26452]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1 | Jun 30 08:17:01 VT-HNI-BIMAX-01 CRON[5073]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
pw1_1 | Jun 30 08:39:01 VT-HNI-BIMAX-01 CRON[2839]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1 | Jun 30 09:09:01 VT-HNI-BIMAX-01 CRON[11562]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1 | Jun 30 09:17:01 VT-HNI-BIMAX-01 CRON[22564]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
config:
[RedisCache] port = 6004
[QueuedTracking] queueEnabled = 1 redisPort = 6004 processDuringTrackingRequest = 1 numQueueWorkers = 32 numRequestsToProcess = 1
[RedisException]
Redis server went away
WARNING [2017-06-28 04:15:42] /usr/share/nginx/www/plugins/QueuedTracking/Queue/Backend/Redis.php(236): Warning - Redis::connect(): php_network_getaddresses: getaddrinfo failed: Name or service not known - Piwik 3.0.4 - Please report this message in the Piwik forums: http://forum.piwik.org (please do a search first as it might have been reported already)
PW5 queuetracking plugin works as normal but no log is imported (log_visit is empty).
Redis queue shows the normal log queue => Queued tracking is working normally.
[root@VNPT-HNI-ANALYTIC-03 bimax]# ./flow.sh monitor 5
Queue is enabled
Request sets in the queue will be processed automatically after a tracking request
Up to 32 workers will be used
Processor will start once there are at least 1 request sets in the queue
22 (1+2+0+0+0+1+0+0+0+1+1+2+0+0+1+1+0+0+2+3+0+1+1+1+0+1+0+1+0+0+2+0) request sets left in queue. 1.01M used memory (1.21M peak). 5 wo7 (0+1+0+0+0+0+0+0+0+0+0+0+0+0+0+1+0+0+0+1+0+1+0+0+0+1+0+1+0+0+1+0) request sets left in queue. 917.48K used memory (1.21M peak). 5 w10 (0+0+1+0+0+0+0+0+0+0+1+0+0+0+1+2+0+0+0+0+1+1+0+1+0+0+0+1+0+0+0+1) request sets left in queue. 995.84K used memory (1.21M peak). 5 11 (1+1+0+0+2+1+0+1+0+0+2+0+0+1+0+0+0+0+0+0+1+0+0+0+0+1+0+0+0+0+0+0) request sets left in queue. 963.91K used memory (1.21M peak). 5 workers active.
!! Piwik warns about the config file:
The Piwik configuration file (config/config.ini.php) is not writable, some of your changes might not be saved. Please change permissions of the config file to make it writable.
Sometime, server returns 504, 404.
Overall UI is unresponsive, error show frequently:
Request with too much data...
Oops… there was a problem during the request. Maybe the server had a temporary issue, or maybe you requested a report with too much data. Please try again. If this error occurs repeatedly please contact your Piwik administrator for assistance.
Need more help? FAQ – Community Help – Professional Help.
Problem occurs after merging
d35d0ec
=> Pageviews is an action when Action type is an URL
-> CDN shows no downloads, no outlink, etc..
Maxmind DB was set up.
Still no Location show.
Note:
Verify within an hour 03-02:
10:07 = 1725 log lines (skip over, stop the import log) -> 129 visits 1778 hits
10:38 = 3000 lines (run until stop) ->
=> subtotal: 244 visits and 4778 actions (Yes!!)
10:57 = 1250 lines (run archive wrong and then invalidate and run again)
=> sub: 352 visits 6028 act (!!!YES YES)
--- It seems that everything is fine on LOCAL, so why it's weird on Server.
13:57: 5000 lines => 156 visits and 5000 action
=> total: 274+156=430?508 visit ||| 11028 act => It seems that
So, actions count is very precise while Visit count is WRONG
(Only visit per server time is wrong)
14:57: 1000 lines => 75 visit and 1000 act
total: 274+156+75=505 ?583 (=508+75) visits and 12028 act =>
15:04 1250 lines
15:11 750 lines
=> Run archive @ 15:17
Server time: 15h 86vts 2780 atc, [14h: 15vts, 220 action]
total: 594(~505 + 86) vts 13278 (12028 + 2780=14808) atc =>
(274+156+15+86 = 531)
[other plugins show total of 594]
=> Run archive @ 15:26
total: 669 vts, 14028 act => should be: 12028 + 1250 + 750 = 14028 action and count(*) log_visit 771.
Server time shows the same (274+156+15+86 = 531)
[other plugins show total of 669]
SO IT'S LIKELY THAT TIME PLUGIN HAS PROBLEM.
No error in nginx error log (on 4003).
why there's this error in log?
VNPT has no pw2 running
2017/07/14 23:21:02 [error] 9620#0: *1264 connect() failed (111: Connection refused) while connecting to upstream, client: 113.164.27.2, server: cdn.vnpt.bimax.tv, request: "GET //index.php?module=Installation&action=getEmptyPageForSystemCheck HTTP/1.1", upstream: "http://172.40.4.90:4002//index.php?module=Installation&action=getEmptyPageForSystemCheck", host: "cdn.vnpt.bimax.tv"
2017/07/14 23:25:43 [error] 9649#0: *1310 connect() failed (111: Connection refused) while connecting to upstream, client: 113.164.27.2, server: cdn.vnpt.bimax.tv, request: "GET //index.php?module=Installation&action=getEmptyPageForSystemCheck HTTP/1.1", upstream: "http://172.40.4.90:4002//index.php?module=Installation&action=getEmptyPageForSystemCheck", host: "cdn.vnpt.bimax.tv"
MultiSite hits and actions don't match.
Hit, action count follow up: #25
THVL: Visit Map shows 16.6k visit while total of User Country is ~ 36.1k.
Local cBIMAX is OK, standard Piwik is OK
On PW5 (old Core code) UserCountry and others plugin match with new Code's number, Visitor Map give number that matches other (but it's wrong).
Core change: anhnongdan/BimaxCore#6
Some reports on UI show error when request data with wide date-range setup.
+ Need to try other cases.
docker ps shows no instance.
Website return 502.
Quick response:
start by docker-compose up
-> instances come back.
https://bitbucket.org/anhnongdan/bimax_seed/issues/31/pw5-vnpt-doesnt-process-queue
Worker number seems to be forked based on incoming traffic.
Stopped THVL traffic around 22:30 and Mem usage keep the same then go up when restart the extractor.
Without View Access:
Problem on PW5 and PW3, PW1 is OK.
For THVL, segment works fine for both single CDN and RollUp, while on VNPT, no data show up.
Both realtime visitor of Piwik and QoS don't work.
And archive actually doesn't work as well.
Archive only calculate the first window (10 mins) and then the log is rotated without calculating.
-> which plugins causes this?
Current New User's default Settings and Dashboard need vast effort to make it right.
After pull changes from Git. Access on browser returns 500.
No error log of PHP, no exception from Docker log.
Changes as #27 might make VI lang completely incompatible.
Both Pw5 (moded) and PW3 (standard PW 3.0.4) have this problem so it should be problem in extractor for splitter
MK show VT has large traffic:
http://mon.sbddigital.com/m/check_mk/index.py?start_url=%2Fm%2Fpnp4nagios%2Findex.php%2Fgraph%3Fhost%3DVT-HNI-BIMAX-03%26srv%3Dhh_live.thvli.vn%26theme%3Dmultisite%26baseurl%3D..%2Fcheck_mk%2F
Live plugin now calc. on tracker table.
!! Updated document.
=> Fixed:
37396af
Mem and CPU consumption:
bimax_db3_1 OK - cpu=10.71% mem=10.67% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db4_1 OK - cpu=108.34% mem=10.73% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db5_1 OK - cpu=2.24% mem=10.24% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db7_1 OK - cpu=18.31% mem=10.56% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db_3 OK - tracker=110099 action=66437023 Reschedule an immediate check of the 'Check_MK' service 2017-03-29 16:52:57 41 sec
OK bimax_db_4 OK - tracker=228608 action=8593197 Reschedule an immediate check of the 'Check_MK' service 2017-03-21 15:56:17 41 sec
OK bimax_db_5 OK - tracker=7077 action=397155 Reschedule an immediate check of the 'Check_MK' service 2017-03-21 15:56:17 41 sec
OK bimax_db_7 OK - tracker=68265 action=1388137 Reschedule an immediate check of the 'Check_MK' service 2017-03-29 14:47:57 41 sec
OK bimax_pw3_1 OK - cpu=57.42% mem=1.19% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_pw4_1 OK - cpu=184.67% mem=0.82% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_pw5_1 OK - cpu=125.41% mem=1.56% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_pw7_1 OK - cpu=35.40% mem=0.68% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis3_1 OK - cpu=4.05% mem=0.03% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis4_1 OK - cpu=23.21% mem=0.07% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis5_1 OK - cpu=0.08% mem=0.04% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis7_1 OK - cpu=0.59% mem=0.03% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
For CDN: no more pageview, donwload, search, outlink, etc.
Bounce rate is not relevant to
=> Get rid of all these irrelevant metrics.
All CDNs dashboard (Total: 2,933 sessions, 1,530,371 hits, 603.24 G tranffered overall)
All CDNs dashboard (Total: 3,826 sessions, 1,493,011 hits, 570.71 G tranffered overall)
ott-vnpt.kplus.vn | 1,443 | 554,483 | 213.28G
ott-vnpt.kplus.vn | 1,764 | 531,663 | 203.98 G
kplusupstream.local | 1,211 | 771,707 | 355.62 G
kplusupstream.local | 1,635 | 741,141 | 335.97 G
ott-fpt.kplus.vn | 165 | 40,055 | 8.86 G
ott-fpt.kplus.vn | 239 | 42,487 | 8.5 G
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.