pirsch-analytics / pirsch Goto Github PK

Right now it contains more than just the IP. That's not a huge issue, but it could be nicer.

Right now, a session cannot live longer than the maximum session lifetime. To change this the following two changes need to be made:

1. use "time" instead of "session" to look up existing sessions so they don't run out
2. update session time in cache when found

SELECT "session" FROM "hit" WHERE fingerprint = $1 AND "session" > $2 LIMIT 1

Probably need to add more test to make sure sessions are continuous, but can run out at some point.

• filtering by path
• filtering by hour of day and path <- cumulated hour of day exists in the Analyzer already

Missed that in 1.4.0...

In case an entry exists already, update it. This is useful to be able to run the processor multiple times a day, whenever and without getting messed up data.

Not sure this is required really.

For a time span add:

• page -> views
• referer -> views

Example:
2020-01-20 - 2020-01-27

• / -> 103 visitors
• https://my-referer.com/ -> 52 visitors

Must be counted for distinct fingerprints and not grouped by path for the total number of visitors.

Add IP ranges to filter for. I'm not sure this is actually that useful, since we filter out bot user agents anyways? It IS useful but keeping track of IPs is cumbersome and not reliable. I might use heuristics for this issue (like how many sessions does a client have, is it reasonable and so on).

https://github.com/GoAdminGroup/go-admin

• Browser
• Operating System
• Device Type (mobile, tablet, desktop)

Test data:

• https://github.com/jenssegers/agent/blob/master/tests/AgentTest.php
• https://github.com/serbanghita/Mobile-Detect/blob/master/tests/ualist.json
• https://github.com/mssola/user_agent/blob/master/all_test.go
• https://github.com/faisalman/ua-parser-js/blob/master/test/os-test.json

referrer := r.Header.Get("Ref")

This should be Referer.

Funnels can probably be implemented by combining multiple filtering steps. Example:

• filter by route /checkout
• filter by event "completed checkout" + previous filter
• filter by conversion goal /checkout/confirmation/** + previous filters
• calculate the loss in between (percentage)
• show visitors from step 1 -> step n

200      --(-10%)--> 180       --(-50%)--> 90
Filter 1             Filter 2              Filter 3

Add a simple Js script to call an endpoint. Easy.

• include screen size (window.screen)
• make sure it's not blocked by the cache
• run after DOMContentLoaded (event listener, and support older browsers)
• endpoint must be configurable
• opt-in
• process screen size and add to analyzer

SELECT count(DISTINCT("fingerprint")) "visitors",
count(DISTINCT("fingerprint", "session")) "sessions"
FROM hit
;

Should show some difference, but it doesn't.

[ERROR] Error processing tracking data err=sql: Scan error on column index 2, name "language": converting NULL to string is unsupported

• browser
• os
• languages
• referrer

To track multiple applications/subdomains or to separate data in other ways.

They are either

• bots or
• people who do not want to be tracked

So I think it's fine to ignore them.

For "today" and "yesterday". The number for yesterday is included twice: yesterday*2+today

• Pages
• Browser
• OS
• Platform

This will be a huge refactoring to simplify storing/processing the statistics and to allow them to be filtered by tenant + page + period, where each part is optional. The other views can be cumulated from these or extracted from the hits directly (live data).

This is still broken. The statistics are created for the last existing day, instead of a new one. This is probably due to how the save functions find the latest row.

https://dev.maxmind.com/geoip/geoip2/geolite2/

Go-http-client
Twingly
evc-batch
mailto
newspaper
FeedReader
Magic Browser
Ruby
Weavr
HackerNews
CFNetwork
SocialBeeAgent
Embed PHP library (?)

Creating new sql.NullInt64s everywhere is annoying. Add a function to do that by passing an integer (0 = null, everything else = valid ID).

main_test.go needs some abstraction/simplification in order to add new storage backends more easily.

Count every visitor who leaves after visiting the first page as bounced.

Need to add hits.

The event handler might not fire, and some other improvements can be made.

So that the fingerprint cannot be unhashed again.

This is not an issue as long as you (the user) don't decrypt the hashes and no one gets access to your database. Consumer IPs usually change after a while, as ISPs use IP pools for their customers.

Analyzer:
if addToday && len(stats) > 0 {

I would like to ask if you planning or maybe accept a PR to accept Mongodb integration?

Track events from JS and the API.

Track HTTP status codes to get statistics on failed requests. Should be optional and set when calling the Hit function.

To prevent anyone from reassembling fingerprints.