Currently RDB Loader accepts --skip shred which forces it to skip all shredded types.

Migrated from snowplow/snowplow#2217

Migrated from snowplow/snowplow#393

Here's the error:

RDB Loader successfully completed following steps: [Discover, Load, Analyze]
INFO: Logs successfully dumped to S3 [s3://foo]
[INFO] [08/22/2017 17:58:18.018] [snowplow-scala-tracker-akka.actor.default-dispatcher-4] [akka://snowplow-scala-tracker/user/IO-HTTP/group-0] Message [spray.can.Http$Connect] from Actor[akka://snowplow-scala-tracker/user/IO-HTTP/host-connector-0/0#52160612] to Actor[akka://snowplow-scala-tracker/user/IO-HTTP/group-0#1063404807] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Exception in thread "Thread-2" akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://snowplow-scala-tracker/user/IO-HTTP#1209416068]] after [1000 ms]
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
        at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
        at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
        at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
        at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
        at java.lang.Thread.run(Thread.java:748)

Migrated from snowplow/snowplow#1017

Migrated from snowplow/snowplow#2875

Migrated from snowplow/snowplow#487

Migrated from snowplow/snowplow#3204

Migrated from snowplow/snowplow#389

Would be little bit more complicated since we moved it to EMR cluster, but still feasible with Dataflow Runner.

@alexanderdean do you think it should include entries for StorageLoader? Because otherwise it would be quite strange to start CHANGELOG from 0.12.0.

Both RDB Loader and RDB Shredder going to be released together (by lucky coincidence they both now versioned 0.12.0).

If it's helpful - the hanging seems to occur after the load transaction.

• AWS_STAGING_ACCESS_KEY_ID
• AWS_STAGING_SECRET_ACCESS_KEY

RDB Shredder is now part of this repository and cross-batch deduplication tests require access to DynamoDB as per snowplow/snowplow#3114

Redshift/postgresql has massive amounts of knobs to turn for optimizing query performance and one recommended by AWS is increasing wlm_query_slot_count in a session before running a query that needs extra CPU and memory.

Docs:
http://docs.aws.amazon.com/redshift/latest/dg/cm-c-defining-query-queues.html

Would be awesome to allow users to set this in the redshift config. We saw massive (10x faster vacuums) improvements when increasing ours to 45, but we need to do that manually for now.

Migrated from snowplow/snowplow#1229

High priority given https://snowplow.zendesk.com/agent/tickets/5872

Migrated from snowplow/snowplow#3384 (comment)

Cannot confirm yet this is a bug, but worth to look into it.

Migrated from snowplow/snowplow#2281

This should discover data on S3 and print all COPY/INSERT statements.

@alexanderdean, I think I copied this behavior (two distinct transactions) from StorageLoader, but wasn't single transaction a whole point of having manifest in the same storage target?

Migrated from snowplow/snowplow#1838

Cannot confirm yet that bug exists, because I think I successfully used periods in bucket names. Also internals changed significantly since original issue, so likely it isn't problem anymore.

sister ticket to snowplow/snowplow#3427

Currently we use the collector_tstamp for the root_tstamp value.
It would be preferable to use the derived_tstamp once all our client side trackers support generating a dvce_sent_tstamp. (Because that point from an analytics perspective you're only interested in the derived_tstamp.

We need to figure out how we migrate from collector_tstamp -> derived_tstamp e.g. what happens for old users who have events without derived_tstamp values.

Migrated from snowplow/snowplow#2538

Migrated from snowplow/snowplow#3369

Background: https://groups.google.com/forum/#!topic/snowplow-user/0Vi2bhfXDPQ

When we have nested shredding of array types, unless we have an array_index field in shredded tables, then our shredding process will inevitably be lossy: there will be nowhere to record the original order of the elements in the array.

Migrated from snowplow/snowplow#3279

Migrated from snowplow/snowplow#185

Consistency check is not always necessary, especially with --folder.

See snowplow/snowplow#3113

Migrated from snowplow/snowplow#3445 (comment)

Right now we're generating random UUID, which makes all natural duplicates synthetic. We should throw exception and abort shredding instead.

• AWS_ACCESS_KEY
• AWS_SECRET_KEY

Migrated from snowplow/snowplow#3278

Migrated from snowplow/snowplow#3277

It contains newer regions support.

Will have to shade it because 1.10.* installed on EMR cluster.

As in snowplow/snowplow#2280

See: https://discourse.snowplowanalytics.com/t/r90-storage-loading-problems/1357/3

Can be consequence of snowplow/snowplow#1830 or (having that it was tested with SSL enabled) combination of other configuration options.

sister ticket to snowplow/snowplow#3443

Consistency check algorithm makes 5 attempts to compare lists of S3 keys. If 5th comparison still gives inconsistent result - loader just continues to load. We can warn users that it failed.

Won't help to fix anything, but at least users will be aware why do they receive cryptic Redshift errors.

RDB Loader tracks monitoring events for dry run. This must not be the case.

e.g. for atomic.events

This would make it easier to delete data that has expired without having to run expensive VACUUM operations

I decided to split changelog items into RDB Loader and RDB Shredder categories as per #27.

We can unite them later when they will start to share code.

Migrated from snowplow/snowplow#3280

Migrated from snowplow/snowplow#2858

--logkey is EmrEtlRunner-specific and doesn't make lot of sense for Dataflow Runner or CLI run as it'll print everything to stdout anyway.

Cannot promise this will be implemented, @acgray, but what would your implementation look like? Is it another setting in aws.s3.buckets?

Migrated from snowplow/snowplow#386

If this is still a requirement, I think one retry after 5 mins should be enough.