snowplow / snowplow Goto Github PK

So SnowPlow users can retrieve the minified sp.js from CDNJS as well as our own CloudFront distribution

Sometimes the ETL process doesn't complete successfully (e.g. because the Processing Bucket is not empty, or there's a problem in the logs being processed).

It would be nice to be able to rerun EmrEtlRunner from a specific point via a command-line argument, e.g:

--from start (the default)
--from emr
--from archive

Still needs doing

Because rdm isn't used for cachebusting - it's used for de-duping when Cf requests the same file from multiple sources

The sync constructor looks like this:

getTracker: function (accountId) {
  return new SnowPlow.Tracker(accountId);
},

There is no equivalent option for creating a sync tracker based on a URL, rather than account ID.

Currently SnowPlow tracks the page name and page URL - but these are not very precise from an analytics perspective:

1. What if a new version of the page is launched on the same URL?
2. What if split testing means two different users see different versions of the page on the same URL?

The idea then is to allow the JavaScript to set a custom page-id:

setPageId('<<CMS/JavaScript generated page ID>>');

Then obviously this page ID would need to be passed through on the querystring and then extracted by the Serde...

The alternative to this approach would be to have a big, constantly changing lookup table on the analytics side when different versions of pages went live - and even this wouldn't meet the A/B testing requirement.

Currently we have hive-format-etl.q let's have non-hive-format-etl.q too

On https://github.com/snowplow/snowplow/wiki/SnowPlow-overview
the link "For engineers" seems wrong

We're planning on moving the pixel to /i (see #29) from /ice.png to gain some extra chars.

Might as well add this support into the Hive serde sooner rather than later.

So that you do something like this:

window.onerror = function(errorMessage, url, line) {
      _snaq.push['trackError', errorMessage, line];
  };

Current limitation where the HiveQL and Serde need to be uploaded into a US bucket in S3.

Elasticity already supports this, so just need to add this into the EmrEtlRunner configuration file somehow...

Removing need for Hive in the ETL step.

See Google Analytics addSocial here: https://developers.google.com/analytics/devguides/collection/gajs/gaTrackingSocial

Again, make the fields first class citizens

Waiting on clarification from @shermozle in shermozle/SnowCannon#23

Change this line in your tag:

sp.src = ('https:' == document.location.protocol ? 'https' : 'http') + '://snplow.com/sp.js';

To be:

sp.src = ('https:' == document.location.protocol ? 'https://d2nqfiix2qwfci.cloudfront.net' : 'http://snplow.com') + '/sp.js';

Where platform is 'web', 'mobile', 'desktop', 'console' etc

Functionality so you can use the same SnowPlow install for multiple sites

Screenshots, wizard steps in the guide doesn't correspond to whats in AWS

setAccount() is actually CloudFront collector specific - should reflect this in the name:

setCollectorCf('xxx')

To complement the existing setCollectorUrl('yyy')

Introduce this in two phases: first deprecate the existing setAccount with a warning like so:

if (typeof console !== 'undefined') {
  console.log("setAccount is deprecated and will be removed in an upcoming version. Please use setCollectorCf() instead.");
}

Some weirdness between Specs2 and Hadoop - it looks like Specs is running tests in parallel and they're getting into conflict because they're sharing a cached Hive record.

For mobile, sometimes you want to batch the updates and send them in after the fact. This means that you cannot rely on the CloudFront timestamp. Update the serde to support a client-timestamp field which overrides the CloudFront timestamp if set.

(I'm making this issue vague since I'll probably update it a few times)

GitHub doesn't support pull requests on wiki's, for shame.

So I've pushed your wiki wtih a few patches at github.comricho/snowplow.wiki

It'd be great if you could pull from the doc_fixes branch

On https://github.com/snowplow/snowplow/wiki/Integrating-SnowPlow-into-your-website
there is an extra ] here:
_trackEvent('Checkout', 'Add', 'ASO01043', 'blue:xxl', '2.0']);

Lower priority - please vote if you want this one.

When you observe two events in your event stream with the same event_id, one of three things could be happening:

Thinking about this further, a simple de-duplication algorithm would be:

1. If the payload matches exactly, then delete all but one copy
2. If the payload differs in any way, then give one event a new ID and preserve its relationship to "parent" event

With this approach, distinguishing between ID collisions and synthetic copies can still be done (if needed) at analysis time.

Could this be done using bloom filters? (http://webdocs.cs.ualberta.ca/~drafiei/papers/DupDet06Sigmod.pdf) Not directly:

unless event_id **definitely not in set** of bloom[event_ids]:
    if hash(event) **definitely not in set** of bloom[event hashes]:
        assign new event_id and store old event_id in original_event_id field
    else:
        delete event # Deletes some false positives!

Rather than delete some false positives (causing data loss), it would be safer to err on the side of caution:

unless event_id **definitely not in set** of bloom[event_ids]:
    assign new event_id and store old event_id in original_event_id field

But this approach will still cause inflated counts with at least once processing systems. So a hybrid model, using a KV cache of N days of events and another KV cache of N days of event hashes:

if event_id in n_days_cache[event_ids]:
    if hash(event) in n_days_cache[event hashes]:
        delete event
    else:
         assign new event_id and store old event_id in original_event_id field
else:
    unless event_id **definitely not in set** of bloom[event_ids]:
        assign new event_id and store old event_id in original_event_id field

This prevents natural copies within an N day window and safely renames ID collisions and synthetic copies all the way back through time.

Title was: Occasionally CloudFront registers an event twice

Content was:

Presumably because it hits two separate nodes at the same time.

Update the ETL to dedupe this - algorithm would be to hash the querystring and check for duplicates within a X minute timeframe. (Hashing the full raw querystring would implicitly include txn_id, the random JavaScript-side identifier, in the uniqueness check.)

Leaving tabs in the serde output can break tab delimited files

Hi,

I notice that you depend on Amazon S3, are there any plans on supporting or has anyone tried to see if this works with the private cloud equivalents such as Eucalyptus Walrus (http://open.eucalyptus.com/wiki/EucalyptusStorage_v1.4)?

Have had to be commented out. Tests don't work, but Deserializer works fine (returns null for appropriate rows)

Because CloudFront does not store cs(Referer) if HTTP status is not 200 (e.g. 304 Not Modified or 403 Forbidden).

Update docs too.

httpclient 4.2 has a querystring parse method which avoids creating a new URI object. This should be a slight speedup over the current verison.

Here's the javadoc:

http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/utils/URLEncodedUtils.html#parse(java.lang.String, java.nio.charset.Charset)

Field passed on the querystring is called aid, short for Application ID.

An idea from Simon Andersson.

Basically: there is a standard ETL job, but you can add extra capabilities to it via plugins, or even swap out the standard querystring logic with an alternative implementation. The ETL process becomes closer to a framework, rather than an opinionated implementation.

Need to make sure that this doesn't impact the subsequent sub-systems (storage and analytics) too badly (in terms of those sub-systems having to deal with lots of different potential data structures).

On https://github.com/snowplow/snowplow/wiki/Integrating-SnowPlow-into-your-website

We have these:
_snaq.push(['trackEvent', 'Videos', ...
_snaq.push(['_trackEvent', 'Mixes', ...

trackEvent or _trackEvent?

1. ETL records its own version, e.g. "hive-0.4.3"
2. ETL records versioning from both tracker (e.g. "js-0.6.1") and collector (e.g. "snowcannon-0.1.0")

Currently the SnowPlow tracker uses a GET on an image tag to send an event. This has a limitation around URL length in IE - see http://support.microsoft.com/kb/208427

Potentially for mobile/server clients we can use POST in the future (with a non-CloudFront collector), but browsers have a couple of limitations which rule out POST:

1. You can't do POSTs asynchronously, which is a key advantage of the GET image tag approach
2. You can't do POSTs cross-domain from JavaScript

A solution, then, for big events, is to split them over multiple GETs (probably only in IE). And then the ETL phase would be responsible for stitching them back together.

To avoid the occasional 403 from CloudFront.

With the current Hive Deserializer (0.4.8/0.4.9), if a row which has a new querystring field in it, the whole row is nullified. This is an overreaction - the field should be silently ignored instead.

This scenario could easily happen if an older collector is run on newer tracker data.

In other words: not sure the behaviour in the BadQsFieldTest specification is fair.

If the page title contains an (escaped) line break, then this line break is unescaped and written out to the event output file, breaking the data row onto two lines.

Need to update the serde so that line breaks (and possibly the row delimiter) are not unescaped.

That gives us an extra 6 characters for data! Plus a 1x1 GIF is a bit smaller than a PNG, and there is no code for generating a 1x1 transparent PNG (versus GIF) easily available (because everybody creates GIFs).

Need to update:

• The install guide - there is a ticket to fix this anyway: #25
• Change the static ice.png file in /1-collectors/cloudfront to i, and make it GIF format
• The snowplow.js JavaScript

Note this will be a breaking change for anybody who has uploaded ice.png and is using our hosted JavaScript.

Chrome giving warnings:

Resource interpreted as Image but transferred with MIME type application/octet-stream:

JavaScript v0.6 has new ecommerce tracking fields. These need to be handled in our ETL and make their way into the final flatfile structure.

See discussion here for rationale:

72ab49c#commitcomment-2024343