anhnongdan / cbimax Goto Github PK

When change to Display simple table, widget goes blank.
After that choosing 'vertical bar graph' causes 500 error.

Reloading page or dragging the widget away will fix the error.

++ Context: PW5 on VNPT 90 is mistakenly excluded from archive, PW5 on 89 is archived as normal. 90 runs on 5 php child while 89 has standard dual 25 pools.

https://bitbucket.org/anhnongdan/bimax_seed/issues/41/php-compare-performance-of-php-fpm

PW5 90 after running a long archive ~30m, show numbers with quite good accuracy

There's error with permission on plugins, changed to 777 but error still exists:

At 06-29 11:12AM, latest Done Archiving! is at:
(19:40 ICT) - No error log showed

(/var/log/flow.log)

INFO [2017-06-28 12:40:06] Archived website id = 5232, 1 API requests, Time elapsed: 2.211s [1/1 done]
INFO [2017-06-28 12:40:06] Done archiving!
INFO [2017-06-28 12:40:06] ---------------------------
INFO [2017-06-28 12:40:06] SUMMARY
INFO [2017-06-28 12:40:06] Total visits for today across archived websites: 1077
INFO [2017-06-28 12:40:06] Archived today's reports for 1 websites
INFO [2017-06-28 12:40:06] Archived week/month/year for 1 websites

All subsequence archive can't finish:

INFO [2017-06-29 04:10:03] - Archiving was last executed without error 15 hours 29 min ago
INFO [2017-06-29 04:10:03] - Will only process the following periods: day (--force-periods)
INFO [2017-06-29 04:10:03] - Will process 1 websites (--force-idsites)
Done Archive
Thu Jun 29 11:10:03 ICT 2017: 2 seconds

DataTable returned by Referrers' API has subtables, filter 'ColumnDelete' doesn't work well with getReferrerType.

• Real time thu-put and DL-speed of cBimax is retrieved from the counter which receives both raw log from syslog-ng of both VNPT and FPT
  => QoS get the total Number for both

• Piwik calculate both

=> All numbers are consistent.

Should we separate this or not?

Archive seems not to be running.

• No log in /var/log/flow.log

• Rotation and archiving halt:

| piwik_log_link_visit_action_2017_06_23_11_0  |
| piwik_log_link_visit_action_2017_06_23_11_10 |
| piwik_log_link_visit_action_2017_06_23_11_20 |
| piwik_log_link_visit_action_2017_06_23_11_30 |
| piwik_log_link_visit_action_2017_06_23_11_40 |
| piwik_log_link_visit_action_tracker

=> archive tables missed:

| piwik_archive_numeric_temp_2017_06_18 |
| piwik_archive_numeric_temp_2017_06_19 |
| piwik_archive_numeric_temp_2017_06_20 |
| piwik_archive_numeric_temp_2017_06_21 |
| piwik_archive_numeric_temp_2017_06_22 |
| piwik_archive_numeric_temp_2017_06_26 |
| piwik_archive_numeric_temp_2017_06_30 |

• Count: piwik_log_link_visit_action_tracker: 409535, piwik_log_link_visit_action: 323031

• Docker compose logs: pw instance log dominates, no log from db (and redis):

pw1_1     | Jun 30 06:39:01 VT-HNI-BIMAX-01 CRON[32541]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1     | Jun 30 06:46:47 VT-HNI-BIMAX-01 CRON[13391]: (CRON) info (No MTA installed, discarding output)
pw1_1     | Jun 30 07:09:01 VT-HNI-BIMAX-01 CRON[8970]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1     | Jun 30 07:17:01 VT-HNI-BIMAX-01 CRON[19972]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
pw1_1     | Jun 30 07:39:01 VT-HNI-BIMAX-01 CRON[17713]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1     | Jun 30 08:09:01 VT-HNI-BIMAX-01 CRON[26452]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1     | Jun 30 08:17:01 VT-HNI-BIMAX-01 CRON[5073]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
pw1_1     | Jun 30 08:39:01 VT-HNI-BIMAX-01 CRON[2839]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1     | Jun 30 09:09:01 VT-HNI-BIMAX-01 CRON[11562]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
pw1_1     | Jun 30 09:17:01 VT-HNI-BIMAX-01 CRON[22564]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

config:
[RedisCache] port = 6004
[QueuedTracking] queueEnabled = 1 redisPort = 6004 processDuringTrackingRequest = 1 numQueueWorkers = 32 numRequestsToProcess = 1

[RedisException]
Redis server went away

WARNING [2017-06-28 04:15:42] /usr/share/nginx/www/plugins/QueuedTracking/Queue/Backend/Redis.php(236): Warning - Redis::connect(): php_network_getaddresses: getaddrinfo failed: Name or service not known - Piwik 3.0.4 - Please report this message in the Piwik forums: http://forum.piwik.org (please do a search first as it might have been reported already)

• PW5 queuetracking plugin works as normal but no log is imported (log_visit is empty).

• Redis queue shows the normal log queue => Queued tracking is working normally.
  [root@VNPT-HNI-ANALYTIC-03 bimax]# ./flow.sh monitor 5
  Queue is enabled
  Request sets in the queue will be processed automatically after a tracking request
  Up to 32 workers will be used
  Processor will start once there are at least 1 request sets in the queue
  22 (1+2+0+0+0+1+0+0+0+1+1+2+0+0+1+1+0+0+2+3+0+1+1+1+0+1+0+1+0+0+2+0) request sets left in queue. 1.01M used memory (1.21M peak). 5 wo7 (0+1+0+0+0+0+0+0+0+0+0+0+0+0+0+1+0+0+0+1+0+1+0+0+0+1+0+1+0+0+1+0) request sets left in queue. 917.48K used memory (1.21M peak). 5 w10 (0+0+1+0+0+0+0+0+0+0+1+0+0+0+1+2+0+0+0+0+1+1+0+1+0+0+0+1+0+0+0+1) request sets left in queue. 995.84K used memory (1.21M peak). 5 11 (1+1+0+0+2+1+0+1+0+0+2+0+0+1+0+0+0+0+0+0+1+0+0+0+0+1+0+0+0+0+0+0) request sets left in queue. 963.91K used memory (1.21M peak). 5 workers active.

!! Piwik warns about the config file:
The Piwik configuration file (config/config.ini.php) is not writable, some of your changes might not be saved. Please change permissions of the config file to make it writable.

Sometime, server returns 504, 404.

Overall UI is unresponsive, error show frequently:
Request with too much data...

Oops… there was a problem during the request. Maybe the server had a temporary issue, or maybe you requested a report with too much data. Please try again. If this error occurs repeatedly please contact your Piwik administrator for assistance. 

Need more help? FAQ – Community Help – Professional Help.

Problem occurs after merging
d35d0ec

https://bitbucket.org/anhnongdan/bimax_seed/issues/36/pw5-high-traffic-mysql-dead-lock-with

=> Pageviews is an action when Action type is an URL

-> CDN shows no downloads, no outlink, etc..

Maxmind DB was set up.
Still no Location show.

https://viettelidc.bimax.tv/pw3

Note:

• 11h: 100 visits and 2250 hits - 2250 actions
• 16H : 65 visit and 1250 hits
  => subtotal 165 and 3500 [!!! exactly]
• 17H: 75 visits 1500 hits
  => 240 and 5000 [!!Yes]

Verify within an hour 03-02:

• 10:07 = 1725 log lines (skip over, stop the import log) -> 129 visits 1778 hits

• 10:38 = 3000 lines (run until stop) ->
  => subtotal: 244 visits and 4778 actions (Yes!!)

• 10:57 = 1250 lines (run archive wrong and then invalidate and run again)
  => sub: 352 visits 6028 act (!!!YES YES)
  --- It seems that everything is fine on LOCAL, so why it's weird on Server.

• 13:57: 5000 lines => 156 visits and 5000 action
  => total: 274+156=430?508 visit ||| 11028 act => It seems that
  So, actions count is very precise while Visit count is WRONG
  (Only visit per server time is wrong)

• 14:57: 1000 lines => 75 visit and 1000 act
  total: 274+156+75=505 ?583 (=508+75) visits and 12028 act =>

• 15:04 1250 lines

• 15:11 750 lines

=> Run archive @ 15:17
Server time: 15h 86vts 2780 atc, [14h: 15vts, 220 action]
total: 594(~505 + 86) vts 13278 (12028 + 2780=14808) atc =>
(274+156+15+86 = 531)
[other plugins show total of 594]

=> Run archive @ 15:26
total: 669 vts, 14028 act => should be: 12028 + 1250 + 750 = 14028 action and count(*) log_visit 771.
Server time shows the same (274+156+15+86 = 531)
[other plugins show total of 669]

SO IT'S LIKELY THAT TIME PLUGIN HAS PROBLEM.

No error in nginx error log (on 4003).

why there's this error in log?
VNPT has no pw2 running

2017/07/14 23:21:02 [error] 9620#0: *1264 connect() failed (111: Connection refused) while connecting to upstream, client: 113.164.27.2, server: cdn.vnpt.bimax.tv, request: "GET //index.php?module=Installation&action=getEmptyPageForSystemCheck HTTP/1.1", upstream: "http://172.40.4.90:4002//index.php?module=Installation&action=getEmptyPageForSystemCheck", host: "cdn.vnpt.bimax.tv"
2017/07/14 23:25:43 [error] 9649#0: *1310 connect() failed (111: Connection refused) while connecting to upstream, client: 113.164.27.2, server: cdn.vnpt.bimax.tv, request: "GET //index.php?module=Installation&action=getEmptyPageForSystemCheck HTTP/1.1", upstream: "http://172.40.4.90:4002//index.php?module=Installation&action=getEmptyPageForSystemCheck", host: "cdn.vnpt.bimax.tv"

90f9ddf

• CDN log surely doesn't have outlink tracking
• What about download? What about OBJ-download CDN vs Live or VoD CDNs

MultiSite hits and actions don't match.
Hit, action count follow up: #25

THVL: Visit Map shows 16.6k visit while total of User Country is ~ 36.1k.
Local cBIMAX is OK, standard Piwik is OK

On PW5 (old Core code) UserCountry and others plugin match with new Code's number, Visitor Map give number that matches other (but it's wrong).

Core change: anhnongdan/BimaxCore#6

For THVL PW5 on VNPT

For K+ PW2 on VT

Some reports on UI show error when request data with wide date-range setup.

• When request data for RollUp with date range before some of site was created.
• Request data for more than a week range.

+ Need to try other cases.

• QoS API:
  https://viettelidc.bimax.tv/pw5/index.php?module=API&method=QoS.overviewGetUserSpeed&token_auth=bce12b0011714f1836a4b4cccdbb190f&idsite=5233&lastMinutes=5&metrics=random&refreshAfterXSecs=5

https://viettelidc.bimax.tv/pw5/index.php?date=2017-07-17&module=API&method=QoS.getAvgDl&format=json&lastMinutes=5&metric=avg_speed&refreshAfterXSecs=5&idSite=5233&period=day

• RollUp API:
  https://viettelidc.bimax.tv/pw7/?module=API&method=RollUpReporting.getRollUps&token_auth=bce12b0011714f1836a4b4cccdbb190f

docker ps shows no instance.
Website return 502.

Quick response:
start by docker-compose up
-> instances come back.

https://bitbucket.org/anhnongdan/bimax_seed/issues/31/pw5-vnpt-doesnt-process-queue

Worker number seems to be forked based on incoming traffic.

• Need to flushall redis, and restart pw docker instance to have system up and running again.
  (from 504 state)

Stopped THVL traffic around 22:30 and Mem usage keep the same then go up when restart the extractor.

Without View Access:

• User can't view QoS info of RollUp
• User can't see the new CDN

Problem on PW5 and PW3, PW1 is OK.

For THVL, segment works fine for both single CDN and RollUp, while on VNPT, no data show up.

Both realtime visitor of Piwik and QoS don't work.
And archive actually doesn't work as well.

Archive only calculate the first window (10 mins) and then the log is rotated without calculating.

-> which plugins causes this?

Current New User's default Settings and Dashboard need vast effort to make it right.

After pull changes from Git. Access on browser returns 500.
No error log of PHP, no exception from Docker log.

Changes as #27 might make VI lang completely incompatible.

Both Pw5 (moded) and PW3 (standard PW 3.0.4) have this problem so it should be problem in extractor for splitter

MK show VT has large traffic:
http://mon.sbddigital.com/m/check_mk/index.py?start_url=%2Fm%2Fpnp4nagios%2Findex.php%2Fgraph%3Fhost%3DVT-HNI-BIMAX-03%26srv%3Dhh_live.thvli.vn%26theme%3Dmultisite%26baseurl%3D..%2Fcheck_mk%2F

Device, session, hits still work.

• Currently, unique visitor is more than visits
• Need to confirm entirely how Roll Up calculate archive, it's not simply sum all up.

This call Visitor Log plugin to show data.

Live plugin now calc. on tracker table.

!! Updated document.

• Turns out piwik omit all the subdomain and redirect to host name after login

=> Fixed:
37396af

• pw3 on viettel cdn some time returns 404.
  After recover, Web UI is very unresponsive.

Mem and CPU consumption:

bimax_db3_1 OK - cpu=10.71% mem=10.67% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db4_1 OK - cpu=108.34% mem=10.73% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db5_1 OK - cpu=2.24% mem=10.24% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db7_1 OK - cpu=18.31% mem=10.56% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_db_3 OK - tracker=110099 action=66437023 Reschedule an immediate check of the 'Check_MK' service 2017-03-29 16:52:57 41 sec
OK bimax_db_4 OK - tracker=228608 action=8593197 Reschedule an immediate check of the 'Check_MK' service 2017-03-21 15:56:17 41 sec
OK bimax_db_5 OK - tracker=7077 action=397155 Reschedule an immediate check of the 'Check_MK' service 2017-03-21 15:56:17 41 sec
OK bimax_db_7 OK - tracker=68265 action=1388137 Reschedule an immediate check of the 'Check_MK' service 2017-03-29 14:47:57 41 sec
OK bimax_pw3_1 OK - cpu=57.42% mem=1.19% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_pw4_1 OK - cpu=184.67% mem=0.82% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_pw5_1 OK - cpu=125.41% mem=1.56% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_pw7_1 OK - cpu=35.40% mem=0.68% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis3_1 OK - cpu=4.05% mem=0.03% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis4_1 OK - cpu=23.21% mem=0.07% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis5_1 OK - cpu=0.08% mem=0.04% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec
OK bimax_redis7_1 OK - cpu=0.59% mem=0.03% Reschedule an immediate check of the 'Check_MK' service 2017-03-30 15:23:57 41 sec

For CDN: no more pageview, donwload, search, outlink, etc.
Bounce rate is not relevant to

=> Get rid of all these irrelevant metrics.

After changed VNPT's counter API to local network IP, QoS can't get data and UI sometime freeze.

• Hits counts are match closely
• BW has diff ~ 5%
• Visit counts diff ~ 25%

All CDNs dashboard (Total: 2,933 sessions, 1,530,371 hits, 603.24 G tranffered overall)
All CDNs dashboard (Total: 3,826 sessions, 1,493,011 hits, 570.71 G tranffered overall)

ott-vnpt.kplus.vn | 1,443 | 554,483 | 213.28G
ott-vnpt.kplus.vn | 1,764 | 531,663 | 203.98 G

kplusupstream.local | 1,211 | 771,707 | 355.62 G
kplusupstream.local | 1,635 | 741,141 | 335.97 G

ott-fpt.kplus.vn | 165 | 40,055 | 8.86 G
ott-fpt.kplus.vn | 239 | 42,487 | 8.5 G