Git Product home page Git Product logo

tap-mssql's Introduction

tap-mssql

CircleCI

Singer tap that extracts data from a Microsoft SQL Server (MSSQL) database and produces JSON-formatted data following the Singer spec.

Requirements

This tap is written in Clojure, and as such, requires the JVM. It has been consistently tested to run using OpenJDK 8, which can be installed on Ubuntu using these commands.

apt-get update && apt-get install -y openjdk-8-jdk

Associated tooling required to use the scripts in this repository follow. (Running the latest versions)

Quick Start

$ bin/tap-mssql --config config.json --discover > catalog.json
$ bin/tap-mssql --config config.json --catalog catalog.json --state state.json | target...

Usage

In the bin folder, there are a few utility scripts to simplify interacting with this tap. Many of these scripts rely on some environment variables being set, see "Testing Infrastructure Design" for more information.

bin/tap-mssql - This script wraps the lein command to run the tap from source code. It is analogous to the command installed by setuptools in Python taps.

As this is a Clojure tap, it supports a non-standard mode of operation by passing the --repl flag. This will start an NREPL server and log the port that it is running on to connect from an IDE for REPL driven development. It is compatible with all other command-line arguments, or can be used on its own. If the tap is invoked in discovery or sync mode along with --repl, the process will be kept alive after the usual Singer process is completed.

Example:
# Discovery
$ bin/tap-mssql --config config.json --discover > catalog.json

# Sync
$ bin/tap-mssql --config config.json --catalog catalog.json --state state.json

# REPL Mode
$ bin/tap-mssql --config config.json --repl

bin/test - This script wraps lein test in order to run the Clojure unit and integration tests against a database running locally.

Example:
$ bin/test

bin/test-db - This script uses docker to run a SQL Server container locally that can be used to run the unit tests against. See the usage text for more information.

Note: It also depends on the mssql-cli tool being installed in order to use the connect option.

To install mssql-cli:

  • create a virtualenv
  • source your new virtualenv
  • pip install mssql-cli

Before running bin/test-db connect, just make sure your virtualenv is sourced.

Example:
$ bin/test-db start
$ bin/test-db connect
$ bin/test-db stop

bin/circleci-local - This script wraps the circleci CLI tool to run the Clojure unit and integration tests in the way CircleCI does, on localhost.

Example:
$ bin/circleci-local

Testing Infrastructure Design

Each actor (developer, CI, etc.) needs their own testing infrastructure so that development can proceed and be verified independently of each other. In order to provide this isolation, we've migrated towards a Docker-based solution.

A script, bin/test-db has been provided that will honor several environment variables and manage the container required by the development and testing.

The environment variables are:

name description
STITCH_TAP_MSSQL_TEST_DATABASE_USER The admin user that should be used to connect to the test database (for docker, this is SA)
STITCH_TAP_MSSQL_TEST_DATABASE_PASSWORD The password for the user (if docker, the SA user will be configured with this password)
STITCH_TAP_MSSQL_TEST_DATABASE_PORT The port for hosting the server. (Default 1433)

To interact with the container, these commands are available:

bin/test-db start - Starts the container under the name sql1

bin/test-db connect - Uses mssql-cli to open a shell to the local MSSQL instance

bin/test-db stop - Tears down and removes the container

Note: There is no volume binding, so all of the data and state in the running container is entirely ephemeral

Observed error messages:

# Bad Host Message

The TCP/IP connection to the host charnock.org, port 51552 has failed.
Error: "connect timed out. Verify the connection properties. Make sure
that an instance of SQL Server is running on the host and accepting
TCP/IP connections at the port. Make sure that TCP connections to the
port are not blocked by a firewall.".

# Unspecified azure server error message

Cannot open server "127.0.0.1" requested by the login. The login
failed. ClientConnectionId:33b6ae38-254a-483b-ba24-04d69828fe0c


# Bad dbname error message

Login failed for user 'foo'.
ClientConnectionId:4c47c255-a330-4bc9-94bd-039c592a8a31

# Database does not exist

Cannot open database "foo" requested by the login. The login
failed. ClientConnectionId:f6e2df79-1d72-4df3-8c38-2a9e7a349003

Copyright © 2019 Stitch

tap-mssql's People

Contributors

bhtowles avatar cosimon avatar dmosorast avatar dsprayberry avatar jacobrobertbaca avatar kallan357 avatar kspeer825 avatar leslievandemark avatar luandy64 avatar manand31 avatar nick-mccoy avatar zachharris1 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

oncoramedical kamd slalom zboldyga fbeauche magusd monkidea stevewithington franklinharry isabella232 zakisaad futuri-pro onemedical nqdev-fork edwinludik-tease

tap-mssql's Issues

Option to Auto-Trust SSL Certificate?

I'm in a situation where I'm running this tap from a linux machine that is not part for the Windows domain, and there's (apparently?) no shared CA or already-installed SSL certificates to facilitate SSL security.

  1. Is there any ability to add an option to auto-trust the SSL certificate?
  2. Is there some other way around this I might not be aware of? I feel like this should be pretty common use - is there a way to get the SSL cert installed automatically? And/or some other solution I haven't through of.

I'm getting up to speed on Clojure as a programming language - happy to contribute code here if there's an option to do so. Thanks!

Replication Method for stream LBLA_dbo_SpecificProduct is invalid: null

I have no idea about what is causing this error

2020-05-13 17:42:03,336Z   main - INFO Running tap-mssql version 1.6.1 and target-stitch version 3.0.2
2020-05-13 17:42:03,582Z   main - INFO Creating ssh tunnel: ['ssh', '-nNT', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', '-oPasswordAuthentication=no', '-oNumberOfPasswordPrompts=0', '-oServerAliveInterval=60', '-oExitOnForwardFailure=yes', '-oConnectTimeout=1', '-p22', '-i/tmp/tmppg_m6hwu', '-L127.0.0.1:59801:<db server>.com:1433', '<user>@<tunneling server>']
2020-05-13 17:42:03,582Z   main - INFO To reproduce: `ssh -nNT -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPasswordAuthentication=no -oNumberOfPasswordPrompts=0 -oServerAliveInterval=60 -oExitOnForwardFailure=yes -oConnectTimeout=1 -p22 -i/tmp/tmppg_m6hwu -L127.0.0.1:59801:<db server>:1433 user@<tunneling server>`
2020-05-13 17:42:04,586Z   main - INFO Successfully connected to tunnel
2020-05-13 17:42:04,587Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-mssql --config /tmp/tap_discover_config.json --discover
2020-05-13 17:42:07,245Z    tap - INFO Started nrepl server at /0.0.0.0:43923
2020-05-13 17:42:07,271Z    tap - INFO Starting discovery mode
2020-05-13 17:42:07,581Z    tap - INFO Successfully connected to the instance
2020-05-13 17:42:07,638Z    tap - INFO Connecting to Microsoft SQL Server version 12.00.6118
2020-05-13 17:42:07,638Z    tap - INFO Discovering  databases...
2020-05-13 17:42:07,697Z    tap - INFO Found 2 non-system databases.
2020-05-13 17:42:07,832Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:08,149Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:08,221Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:08,302Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:08,425Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:08,482Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:08,552Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:08,672Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:08,730Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:08,798Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:08,919Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:08,977Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:09,169Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:09,287Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:09,345Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:09,414Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:09,535Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:09,599Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:09,671Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:09,784Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:09,841Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:09,906Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:10,019Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:10,078Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:10,149Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:10,262Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:10,318Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:10,385Z    tap - INFO Discovering columns and tables for database: LBLA
2020-05-13 17:42:10,506Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:10,573Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:10,644Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:10,836Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:10,901Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:10,975Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:11,099Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:11,157Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:11,352Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:11,465Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:11,523Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:11,589Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:11,701Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:11,759Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:11,832Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:11,948Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:12,013Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:12,080Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:12,194Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:12,251Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:12,318Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:12,430Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:12,488Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:12,557Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:12,669Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:12,727Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:12,794Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:12,908Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:12,970Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:13,046Z    tap - INFO Discovering columns and tables for database: LBLA_Staging
2020-05-13 17:42:13,159Z    tap - INFO Executing query: SELECT tbl.name as table_name, SCHEMA_NAME(tbl.schema_id) as schema_name, CAST(p.rows AS bigint) as row_count FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id
2020-05-13 17:42:13,223Z    tap - INFO Executing query: SELECT kcu.table_name as table_name, kcu.column_name as primary_key, kcu.table_schema as table_schema, kcu.table_catalog as table_catalog FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON tc.CONSTRAINT_TYPE = 'PRIMARY KEY' AND tc.CONSTRAINT_NAME = kcu.CONSTRAINT_NAME
2020-05-13 17:42:13,448Z    tap - INFO Tap Finished
2020-05-13 17:42:13,448Z    tap - INFO Shutting down the nrepl server
2020-05-13 17:42:13,480Z   main - INFO Tap exited normally.
2020-05-13 17:42:13,495Z   main - INFO Saving list of discovered streams
2020-05-13 17:42:13,528Z   main - INFO Saving structure of stream vwExt_ExtChargeType (tap_stream_id: LBLA_dbo_vwExt_ExtChargeType)
2020-05-13 17:42:13,543Z   main - INFO Saving structure of stream vw_DOTFlatRates (tap_stream_id: LBLA_dbo_vw_DOTFlatRates)
2020-05-13 17:42:13,571Z   main - INFO Saving structure of stream vwExt_Station (tap_stream_id: LBLA_dbo_vwExt_Station)
2020-05-13 17:42:13,583Z   main - INFO Saving structure of stream vwExt_PADD (tap_stream_id: LBLA_dbo_vwExt_PADD)
2020-05-13 17:42:13,598Z   main - INFO Saving structure of stream SpecificProduct (tap_stream_id: LBLA_dbo_SpecificProduct)
2020-05-13 17:42:13,610Z   main - INFO Saving structure of stream vw_aspnet_Roles (tap_stream_id: LBLA_Staging_dbo_vw_aspnet_Roles)
2020-05-13 17:42:13,623Z   main - INFO Saving structure of stream vwExt_Commodity (tap_stream_id: LBLA_dbo_vwExt_Commodity)
2020-05-13 17:42:13,642Z   main - INFO Saving structure of stream Form6Report (tap_stream_id: LBLA_Staging_dbo_Form6Report)
2020-05-13 17:42:13,660Z   main - INFO Saving structure of stream PipelineSystemTariff (tap_stream_id: LBLA_Staging_dbo_PipelineSystemTariff)
2020-05-13 17:42:13,673Z   main - INFO Saving structure of stream vw_AllEffectiveCounts (tap_stream_id: LBLA_Staging_dbo_vw_AllEffectiveCounts)
2020-05-13 17:42:13,693Z   main - INFO Saving structure of stream CommodityType (tap_stream_id: LBLA_dbo_CommodityType)
2020-05-13 17:42:13,710Z   main - INFO Saving structure of stream vwExt_RateVolume (tap_stream_id: LBLA_dbo_vwExt_RateVolume)
2020-05-13 17:42:13,726Z   main - INFO Saving structure of stream vw_RatesHistory (tap_stream_id: LBLA_dbo_vw_RatesHistory)
2020-05-13 17:42:13,746Z   main - INFO Saving structure of stream vwExt_Jurisdiction (tap_stream_id: LBLA_dbo_vwExt_Jurisdiction)
2020-05-13 17:42:13,760Z   main - INFO Saving structure of stream vw_FutureEffectiveTariff (tap_stream_id: LBLA_Staging_dbo_vw_FutureEffectiveTariff)
2020-05-13 17:42:13,778Z   main - INFO Saving structure of stream vw_Form6Pipeline (tap_stream_id: LBLA_Staging_dbo_vw_Form6Pipeline)
2020-05-13 17:42:13,795Z   main - INFO Saving structure of stream aspnet_UsersInRoles (tap_stream_id: LBLA_Staging_dbo_aspnet_UsersInRoles)
2020-05-13 17:42:13,808Z   main - INFO Saving structure of stream RelatedTariff (tap_stream_id: LBLA_Staging_dbo_RelatedTariff)
2020-05-13 17:42:13,823Z   main - INFO Saving structure of stream vw_AllIS (tap_stream_id: LBLA_dbo_vw_AllIS)
2020-05-13 17:42:13,841Z   main - INFO Saving structure of stream vw_DistanceReport (tap_stream_id: LBLA_Staging_dbo_vw_DistanceReport)
2020-05-13 17:42:13,858Z   main - INFO Saving structure of stream TariffCommodity (tap_stream_id: LBLA_Staging_dbo_TariffCommodity)
2020-05-13 17:42:13,873Z   main - INFO Saving structure of stream vwExt_AERExtCharge (tap_stream_id: LBLA_Staging_dbo_vwExt_AERExtCharge)
2020-05-13 17:42:13,888Z   main - INFO Saving structure of stream vw_EffectiveTariff (tap_stream_id: LBLA_Staging_dbo_vw_EffectiveTariff)
2020-05-13 17:42:13,909Z   main - INFO Saving structure of stream RelatedTariff (tap_stream_id: LBLA_dbo_RelatedTariff)
2020-05-13 17:42:13,920Z   main - INFO Saving structure of stream MarketStation (tap_stream_id: LBLA_dbo_MarketStation)
2020-05-13 17:42:13,942Z   main - INFO Saving structure of stream vwExt_SpecProduct (tap_stream_id: LBLA_Staging_dbo_vwExt_SpecProduct)
2020-05-13 17:42:13,954Z   main - INFO Saving structure of stream vw_AllEffectiveCounts (tap_stream_id: LBLA_dbo_vw_AllEffectiveCounts)
2020-05-13 17:42:13,966Z   main - INFO Saving structure of stream vw_UserActivity (tap_stream_id: LBLA_dbo_vw_UserActivity)
2020-05-13 17:42:13,980Z   main - INFO Saving structure of stream vwExt_ExtChargeType (tap_stream_id: LBLA_Staging_dbo_vwExt_ExtChargeType)
2020-05-13 17:42:13,994Z   main - INFO Saving structure of stream vwExt_Commodity (tap_stream_id: LBLA_Staging_dbo_vwExt_Commodity)
2020-05-13 17:42:14,010Z   main - INFO Saving structure of stream aspnet_Applications (tap_stream_id: LBLA_dbo_aspnet_Applications)
2020-05-13 17:42:14,022Z   main - INFO Saving structure of stream vwExt_PipelineSystemTariff (tap_stream_id: LBLA_dbo_vwExt_PipelineSystemTariff)
2020-05-13 17:42:14,038Z   main - INFO Saving structure of stream PADD (tap_stream_id: LBLA_Staging_dbo_PADD)
2020-05-13 17:42:14,057Z   main - INFO Saving structure of stream vw_RatesHistory_Trafigura (tap_stream_id: LBLA_dbo_vw_RatesHistory_Trafigura)
2020-05-13 17:42:14,071Z   main - INFO Saving structure of stream Market (tap_stream_id: LBLA_dbo_Market)
2020-05-13 17:42:14,083Z   main - INFO Saving structure of stream vwExt_AllEffectiveRates (tap_stream_id: LBLA_Staging_dbo_vwExt_AllEffectiveRates)
2020-05-13 17:42:14,098Z   main - INFO Saving structure of stream vw_ExplorerRates (tap_stream_id: LBLA_Staging_dbo_vw_ExplorerRates)
2020-05-13 17:42:14,116Z   main - INFO Saving structure of stream vwExt_Tariff (tap_stream_id: LBLA_dbo_vwExt_Tariff)
2020-05-13 17:42:14,131Z   main - INFO Saving structure of stream vw_AllTariffsWithRateAuthorities (tap_stream_id: LBLA_Staging_dbo_vw_AllTariffsWithRateAuthorities)
2020-05-13 17:42:14,149Z   main - INFO Saving structure of stream vw_AllTariffsWithIds (tap_stream_id: LBLA_dbo_vw_AllTariffsWithIds)
2020-05-13 17:42:14,170Z   main - INFO Saving structure of stream RateAuthorityType (tap_stream_id: LBLA_dbo_RateAuthorityType)
2020-05-13 17:42:14,184Z   main - INFO Saving structure of stream vw_RatesHistory_Trafigura (tap_stream_id: LBLA_Staging_dbo_vw_RatesHistory_Trafigura)
2020-05-13 17:42:14,198Z   main - INFO Saving structure of stream CommodityType (tap_stream_id: LBLA_Staging_dbo_CommodityType)
2020-05-13 17:42:14,214Z   main - INFO Saving structure of stream vw_ExplorerRates (tap_stream_id: LBLA_dbo_vw_ExplorerRates)
2020-05-13 17:42:14,238Z   main - INFO Saving structure of stream vw_PipelineStation (tap_stream_id: LBLA_dbo_vw_PipelineStation)
2020-05-13 17:42:14,257Z   main - INFO Saving structure of stream TariffRateAuthority (tap_stream_id: LBLA_Staging_dbo_TariffRateAuthority)
2020-05-13 17:42:14,272Z   main - INFO Saving structure of stream vwExt_Jurisdiction (tap_stream_id: LBLA_Staging_dbo_vwExt_Jurisdiction)
2020-05-13 17:42:14,289Z   main - INFO Saving structure of stream MarketStation (tap_stream_id: LBLA_Staging_dbo_MarketStation)
2020-05-13 17:42:14,301Z   main - INFO Saving structure of stream vw_aspnet_UsersInRoles (tap_stream_id: LBLA_dbo_vw_aspnet_UsersInRoles)
2020-05-13 17:42:14,317Z   main - INFO Saving structure of stream vw_aspnet_Users (tap_stream_id: LBLA_Staging_dbo_vw_aspnet_Users)
2020-05-13 17:42:14,331Z   main - INFO Saving structure of stream VolumeMeasurement (tap_stream_id: LBLA_Staging_dbo_VolumeMeasurement)
2020-05-13 17:42:14,346Z   main - INFO Saving structure of stream Tariff (tap_stream_id: LBLA_Staging_dbo_Tariff)
2020-05-13 17:42:14,362Z   main - INFO Saving structure of stream RefineryStation (tap_stream_id: LBLA_dbo_RefineryStation)
2020-05-13 17:42:14,373Z   main - INFO Saving structure of stream vw_DOTFlatRates (tap_stream_id: LBLA_Staging_dbo_vw_DOTFlatRates)
2020-05-13 17:42:14,390Z   main - INFO Saving structure of stream vw_DistanceReport2 (tap_stream_id: LBLA_Staging_dbo_vw_DistanceReport2)
2020-05-13 17:42:14,410Z   main - INFO Saving structure of stream BaseRateAuthorityType (tap_stream_id: LBLA_dbo_BaseRateAuthorityType)
2020-05-13 17:42:14,424Z   main - INFO Saving structure of stream vw_RoutesRatesCancel (tap_stream_id: LBLA_Staging_dbo_vw_RoutesRatesCancel)
2020-05-13 17:42:14,449Z   main - INFO Saving structure of stream RelatedTariffType (tap_stream_id: LBLA_Staging_dbo_RelatedTariffType)
2020-05-13 17:42:14,464Z   main - INFO Saving structure of stream vw_FutureEffectiveTariff (tap_stream_id: LBLA_dbo_vw_FutureEffectiveTariff)
2020-05-13 17:42:14,482Z   main - INFO Saving structure of stream TariffRate (tap_stream_id: LBLA_dbo_TariffRate)
2020-05-13 17:42:14,501Z   main - INFO Saving structure of stream PeriodType (tap_stream_id: LBLA_dbo_PeriodType)
2020-05-13 17:42:14,513Z   main - INFO Saving structure of stream vw_UserActivity (tap_stream_id: LBLA_Staging_dbo_vw_UserActivity)
2020-05-13 17:42:14,527Z   main - INFO Saving structure of stream vwExt_AllEffectiveRates (tap_stream_id: LBLA_dbo_vwExt_AllEffectiveRates)
2020-05-13 17:42:14,543Z   main - INFO Saving structure of stream Jurisdiction (tap_stream_id: LBLA_Staging_dbo_Jurisdiction)
2020-05-13 17:42:14,567Z   main - INFO Saving structure of stream Holiday (tap_stream_id: LBLA_Staging_dbo_Holiday)
2020-05-13 17:42:14,584Z   main - INFO Saving structure of stream vw_RatesHistory_Marathon (tap_stream_id: LBLA_dbo_vw_RatesHistory_Marathon)
2020-05-13 17:42:14,599Z   main - INFO Saving structure of stream vw_AllIS (tap_stream_id: LBLA_Staging_dbo_vw_AllIS)
2020-05-13 17:42:14,612Z   main - INFO Saving structure of stream SpecificProduct (tap_stream_id: LBLA_Staging_dbo_SpecificProduct)
2020-05-13 17:42:14,627Z   main - INFO Saving structure of stream vw_RatesHistory_FrereHall (tap_stream_id: LBLA_dbo_vw_RatesHistory_FrereHall)
2020-05-13 17:42:14,638Z   main - INFO Saving structure of stream ExtraChargeType (tap_stream_id: LBLA_dbo_ExtraChargeType)
2020-05-13 17:42:14,651Z   main - INFO Saving structure of stream vwExt_RateCurrency (tap_stream_id: LBLA_dbo_vwExt_RateCurrency)
2020-05-13 17:42:14,664Z   main - INFO Saving structure of stream aspnet_Users (tap_stream_id: LBLA_dbo_aspnet_Users)
2020-05-13 17:42:14,679Z   main - INFO Saving structure of stream vwExt_RateCurrency (tap_stream_id: LBLA_Staging_dbo_vwExt_RateCurrency)
2020-05-13 17:42:14,692Z   main - INFO Saving structure of stream RateAuthorityType (tap_stream_id: LBLA_Staging_dbo_RateAuthorityType)
2020-05-13 17:42:14,706Z   main - INFO Saving structure of stream BaseRateAuthorityType (tap_stream_id: LBLA_Staging_dbo_BaseRateAuthorityType)
2020-05-13 17:42:14,717Z   main - INFO Saving structure of stream Tariff (tap_stream_id: LBLA_dbo_Tariff)
2020-05-13 17:42:14,733Z   main - INFO Saving structure of stream vw_aspnet_Applications (tap_stream_id: LBLA_Staging_dbo_vw_aspnet_Applications)
2020-05-13 17:42:14,747Z   main - INFO Saving structure of stream vw_AllEffectiveCancelTariffRatesRules (tap_stream_id: LBLA_Staging_dbo_vw_AllEffectiveCancelTariffRatesRules)
2020-05-13 17:42:14,767Z   main - INFO Saving structure of stream Filing (tap_stream_id: LBLA_dbo_Filing)
2020-05-13 17:42:14,780Z   main - INFO Saving structure of stream vwExt_State (tap_stream_id: LBLA_Staging_dbo_vwExt_State)
2020-05-13 17:42:14,793Z   main - INFO Saving structure of stream Station (tap_stream_id: LBLA_Staging_dbo_Station)
2020-05-13 17:42:14,808Z   main - INFO Saving structure of stream vw_AllEffectiveTariffRatesRules (tap_stream_id: LBLA_Staging_dbo_vw_AllEffectiveTariffRatesRules)
2020-05-13 17:42:14,827Z   main - INFO Saving structure of stream Commodity (tap_stream_id: LBLA_Staging_dbo_Commodity)
2020-05-13 17:42:14,844Z   main - INFO Saving structure of stream vw_aspnet_MembershipUsers (tap_stream_id: LBLA_dbo_vw_aspnet_MembershipUsers)
2020-05-13 17:42:14,859Z   main - INFO Saving structure of stream UserActivity (tap_stream_id: LBLA_dbo_UserActivity)
2020-05-13 17:42:14,871Z   main - INFO Saving structure of stream TariffStatus (tap_stream_id: LBLA_Staging_dbo_TariffStatus)
2020-05-13 17:42:14,886Z   main - INFO Saving structure of stream VolumeMeasurement (tap_stream_id: LBLA_dbo_VolumeMeasurement)
2020-05-13 17:42:14,901Z   main - INFO Saving structure of stream PipelineSystem (tap_stream_id: LBLA_dbo_PipelineSystem)
2020-05-13 17:42:14,916Z   main - INFO Saving structure of stream vw_Stations (tap_stream_id: LBLA_Staging_dbo_vw_Stations)
2020-05-13 17:42:14,934Z   main - INFO Saving structure of stream State (tap_stream_id: LBLA_Staging_dbo_State)
2020-05-13 17:42:14,946Z   main - INFO Saving structure of stream TariffHeading (tap_stream_id: LBLA_Staging_dbo_TariffHeading)
2020-05-13 17:42:14,959Z   main - INFO Saving structure of stream LinkType (tap_stream_id: LBLA_Staging_dbo_LinkType)
2020-05-13 17:42:14,974Z   main - INFO Saving structure of stream Pipeline (tap_stream_id: LBLA_dbo_Pipeline)
2020-05-13 17:42:14,989Z   main - INFO Saving structure of stream LinkType (tap_stream_id: LBLA_dbo_LinkType)
2020-05-13 17:42:15,003Z   main - INFO Saving structure of stream vw_RoutesRatesCancel (tap_stream_id: LBLA_dbo_vw_RoutesRatesCancel)
2020-05-13 17:42:15,031Z   main - INFO Saving structure of stream vwExt_Station (tap_stream_id: LBLA_Staging_dbo_vwExt_Station)
2020-05-13 17:42:15,045Z   main - INFO Saving structure of stream aspnet_UsersInRoles (tap_stream_id: LBLA_dbo_aspnet_UsersInRoles)
2020-05-13 17:42:15,060Z   main - INFO Saving structure of stream TariffHeadingType (tap_stream_id: LBLA_Staging_dbo_TariffHeadingType)
2020-05-13 17:42:15,077Z   main - INFO Saving structure of stream vwExt_GoverningTariff (tap_stream_id: LBLA_dbo_vwExt_GoverningTariff)
2020-05-13 17:42:15,092Z   main - INFO Saving structure of stream Station (tap_stream_id: LBLA_dbo_Station)
2020-05-13 17:42:15,152Z   main - INFO Saving structure of stream UserActivity (tap_stream_id: LBLA_Staging_dbo_UserActivity)
2020-05-13 17:42:15,165Z   main - INFO Saving structure of stream PADD (tap_stream_id: LBLA_dbo_PADD)
2020-05-13 17:42:15,176Z   main - INFO Saving structure of stream vw_Stations (tap_stream_id: LBLA_dbo_vw_Stations)
2020-05-13 17:42:15,191Z   main - INFO Saving structure of stream vwExt_JointCarrier (tap_stream_id: LBLA_dbo_vwExt_JointCarrier)
2020-05-13 17:42:15,205Z   main - INFO Saving structure of stream vwExt_CommodityType (tap_stream_id: LBLA_Staging_dbo_vwExt_CommodityType)
2020-05-13 17:42:15,217Z   main - INFO Saving structure of stream TariffType (tap_stream_id: LBLA_dbo_TariffType)
2020-05-13 17:42:15,231Z   main - INFO Saving structure of stream sysdiagrams (tap_stream_id: LBLA_dbo_sysdiagrams)
2020-05-13 17:42:15,244Z   main - INFO Saving structure of stream StationAlias (tap_stream_id: LBLA_dbo_StationAlias)
2020-05-13 17:42:15,258Z   main - INFO Saving structure of stream Refinery (tap_stream_id: LBLA_dbo_Refinery)
2020-05-13 17:42:15,272Z   main - INFO Saving structure of stream vwExt_PipelineSystemTariff (tap_stream_id: LBLA_Staging_dbo_vwExt_PipelineSystemTariff)
2020-05-13 17:42:15,284Z   main - INFO Saving structure of stream sysdiagrams (tap_stream_id: LBLA_Staging_dbo_sysdiagrams)
2020-05-13 17:42:15,300Z   main - INFO Saving structure of stream aspnet_Roles (tap_stream_id: LBLA_dbo_aspnet_Roles)
2020-05-13 17:42:15,312Z   main - INFO Saving structure of stream aspnet_Profile (tap_stream_id: LBLA_dbo_aspnet_Profile)
2020-05-13 17:42:15,330Z   main - INFO Saving structure of stream aspnet_Membership (tap_stream_id: LBLA_dbo_aspnet_Membership)
2020-05-13 17:42:15,355Z   main - INFO Saving structure of stream aspnet_Membership (tap_stream_id: LBLA_Staging_dbo_aspnet_Membership)
2020-05-13 17:42:15,376Z   main - INFO Saving structure of stream vw_EffectiveTariff (tap_stream_id: LBLA_dbo_vw_EffectiveTariff)
2020-05-13 17:42:15,393Z   main - INFO Saving structure of stream vw_DistanceReport2 (tap_stream_id: LBLA_dbo_vw_DistanceReport2)
2020-05-13 17:42:15,411Z   main - INFO Saving structure of stream vwExt_PADD (tap_stream_id: LBLA_Staging_dbo_vwExt_PADD)
2020-05-13 17:42:15,424Z   main - INFO Saving structure of stream PeriodType (tap_stream_id: LBLA_Staging_dbo_PeriodType)
2020-05-13 17:42:15,437Z   main - INFO Saving structure of stream aspnet_Users (tap_stream_id: LBLA_Staging_dbo_aspnet_Users)
2020-05-13 17:42:15,452Z   main - INFO Saving structure of stream Currency (tap_stream_id: LBLA_Staging_dbo_Currency)
2020-05-13 17:42:15,465Z   main - INFO Saving structure of stream vwExt_Tariff (tap_stream_id: LBLA_Staging_dbo_vwExt_Tariff)
2020-05-13 17:42:15,483Z   main - INFO Saving structure of stream vw_FourthSection (tap_stream_id: LBLA_Staging_dbo_vw_FourthSection)
2020-05-13 17:42:15,499Z   main - INFO Saving structure of stream vw_AllTariffsWithIds (tap_stream_id: LBLA_Staging_dbo_vw_AllTariffsWithIds)
2020-05-13 17:42:15,516Z   main - INFO Saving structure of stream vw_RatesHistory_Marathon (tap_stream_id: LBLA_Staging_dbo_vw_RatesHistory_Marathon)
2020-05-13 17:42:15,529Z   main - INFO Saving structure of stream vw_DistanceReport (tap_stream_id: LBLA_dbo_vw_DistanceReport)
2020-05-13 17:42:15,547Z   main - INFO Saving structure of stream aspnet_Profile (tap_stream_id: LBLA_Staging_dbo_aspnet_Profile)
2020-05-13 17:42:15,560Z   main - INFO Saving structure of stream TariffRateDiscount (tap_stream_id: LBLA_Staging_dbo_TariffRateDiscount)
2020-05-13 17:42:15,577Z   main - INFO Saving structure of stream Form6Report (tap_stream_id: LBLA_dbo_Form6Report)
2020-05-13 17:42:15,593Z   main - INFO Saving structure of stream vw_PipelineStation (tap_stream_id: LBLA_Staging_dbo_vw_PipelineStation)
2020-05-13 17:42:15,611Z   main - INFO Saving structure of stream PipelineSystemTariff (tap_stream_id: LBLA_dbo_PipelineSystemTariff)
2020-05-13 17:42:15,621Z   main - INFO Saving structure of stream vw_Form6Pipeline (tap_stream_id: LBLA_dbo_vw_Form6Pipeline)
2020-05-13 17:42:15,635Z   main - INFO Saving structure of stream vw_aspnet_UsersInRoles (tap_stream_id: LBLA_Staging_dbo_vw_aspnet_UsersInRoles)
2020-05-13 17:42:15,648Z   main - INFO Saving structure of stream ExtraChargeTypeGroup (tap_stream_id: LBLA_dbo_ExtraChargeTypeGroup)
2020-05-13 17:42:15,660Z   main - INFO Saving structure of stream vw_RatesHistory_FrereHall (tap_stream_id: LBLA_Staging_dbo_vw_RatesHistory_FrereHall)
2020-05-13 17:42:15,677Z   main - INFO Saving structure of stream Pipeline (tap_stream_id: LBLA_Staging_dbo_Pipeline)
2020-05-13 17:42:15,688Z   main - INFO Saving structure of stream vw_aspnet_Profiles (tap_stream_id: LBLA_Staging_dbo_vw_aspnet_Profiles)
2020-05-13 17:42:15,704Z   main - INFO Saving structure of stream vwExt_RateVolume (tap_stream_id: LBLA_Staging_dbo_vwExt_RateVolume)
2020-05-13 17:42:15,720Z   main - INFO Saving structure of stream ExtraChargeTypeGroup (tap_stream_id: LBLA_Staging_dbo_ExtraChargeTypeGroup)
2020-05-13 17:42:15,738Z   main - INFO Saving structure of stream ExtraChargeType (tap_stream_id: LBLA_Staging_dbo_ExtraChargeType)
2020-05-13 17:42:15,750Z   main - INFO Saving structure of stream aspnet_Applications (tap_stream_id: LBLA_Staging_dbo_aspnet_Applications)
2020-05-13 17:42:15,764Z   main - INFO Saving structure of stream PipelineSystem (tap_stream_id: LBLA_Staging_dbo_PipelineSystem)
2020-05-13 17:42:15,778Z   main - INFO Saving structure of stream ReportType (tap_stream_id: LBLA_dbo_ReportType)
2020-05-13 17:42:15,791Z   main - INFO Saving structure of stream RelatedTariffType (tap_stream_id: LBLA_dbo_RelatedTariffType)
2020-05-13 17:42:15,805Z   main - INFO Saving structure of stream JointCarrier (tap_stream_id: LBLA_Staging_dbo_JointCarrier)
2020-05-13 17:42:15,822Z   main - INFO Saving structure of stream TariffTariffType (tap_stream_id: LBLA_dbo_TariffTariffType)
2020-05-13 17:42:15,836Z   main - INFO Saving structure of stream vwExt_PipelineSystem (tap_stream_id: LBLA_dbo_vwExt_PipelineSystem)
2020-05-13 17:42:15,850Z   main - INFO Saving structure of stream vwExt_TariffCommodity (tap_stream_id: LBLA_dbo_vwExt_TariffCommodity)
2020-05-13 17:42:15,862Z   main - INFO Saving structure of stream vw_FourthSection (tap_stream_id: LBLA_dbo_vw_FourthSection)
2020-05-13 17:42:15,876Z   main - INFO Saving structure of stream Currency (tap_stream_id: LBLA_dbo_Currency)
2020-05-13 17:42:15,894Z   main - INFO Saving structure of stream aspnet_SchemaVersions (tap_stream_id: LBLA_Staging_dbo_aspnet_SchemaVersions)
2020-05-13 17:42:15,908Z   main - INFO Saving structure of stream vwExt_PipelineSystem (tap_stream_id: LBLA_Staging_dbo_vwExt_PipelineSystem)
2020-05-13 17:42:15,923Z   main - INFO Saving structure of stream vwExt_Pipeline (tap_stream_id: LBLA_dbo_vwExt_Pipeline)
2020-05-13 17:42:15,938Z   main - INFO Saving structure of stream TariffType (tap_stream_id: LBLA_Staging_dbo_TariffType)
2020-05-13 17:42:15,949Z   main - INFO Saving structure of stream TariffCommodity (tap_stream_id: LBLA_dbo_TariffCommodity)
2020-05-13 17:42:15,961Z   main - INFO Saving structure of stream TariffRateDiscount (tap_stream_id: LBLA_dbo_TariffRateDiscount)
2020-05-13 17:42:15,975Z   main - INFO Saving structure of stream vw_aspnet_Applications (tap_stream_id: LBLA_dbo_vw_aspnet_Applications)
2020-05-13 17:42:15,992Z   main - INFO Saving structure of stream ExtraCharge (tap_stream_id: LBLA_dbo_ExtraCharge)
2020-05-13 17:42:16,004Z   main - INFO Saving structure of stream State (tap_stream_id: LBLA_dbo_State)
2020-05-13 17:42:16,016Z   main - INFO Saving structure of stream JointCarrier (tap_stream_id: LBLA_dbo_JointCarrier)
2020-05-13 17:42:16,031Z   main - INFO Saving structure of stream Commodity (tap_stream_id: LBLA_dbo_Commodity)
2020-05-13 17:42:16,046Z   main - INFO Saving structure of stream Market (tap_stream_id: LBLA_Staging_dbo_Market)
2020-05-13 17:42:16,057Z   main - INFO Saving structure of stream TariffHeading (tap_stream_id: LBLA_dbo_TariffHeading)
2020-05-13 17:42:16,074Z   main - INFO Saving structure of stream vwExt_Pipeline (tap_stream_id: LBLA_Staging_dbo_vwExt_Pipeline)
2020-05-13 17:42:16,090Z   main - INFO Saving structure of stream aspnet_Roles (tap_stream_id: LBLA_Staging_dbo_aspnet_Roles)
2020-05-13 17:42:16,105Z   main - INFO Saving structure of stream vwExt_CommodityType (tap_stream_id: LBLA_dbo_vwExt_CommodityType)
2020-05-13 17:42:16,120Z   main - INFO Saving structure of stream vwExt_SpecProduct (tap_stream_id: LBLA_dbo_vwExt_SpecProduct)
2020-05-13 17:42:16,134Z   main - INFO Saving structure of stream vwExt_TariffCommodity (tap_stream_id: LBLA_Staging_dbo_vwExt_TariffCommodity)
2020-05-13 17:42:16,146Z   main - INFO Saving structure of stream ReportType (tap_stream_id: LBLA_Staging_dbo_ReportType)
2020-05-13 17:42:16,162Z   main - INFO Saving structure of stream ExtraCharge (tap_stream_id: LBLA_Staging_dbo_ExtraCharge)
2020-05-13 17:42:16,179Z   main - INFO Saving structure of stream TariffRateAuthority (tap_stream_id: LBLA_dbo_TariffRateAuthority)
2020-05-13 17:42:16,193Z   main - INFO Saving structure of stream vw_AllTariffsWithRateAuthorities (tap_stream_id: LBLA_dbo_vw_AllTariffsWithRateAuthorities)
2020-05-13 17:42:16,210Z   main - INFO Saving structure of stream vwExt_AERDiscount (tap_stream_id: LBLA_dbo_vwExt_AERDiscount)
2020-05-13 17:42:16,223Z   main - INFO Saving structure of stream vw_aspnet_Profiles (tap_stream_id: LBLA_dbo_vw_aspnet_Profiles)
2020-05-13 17:42:16,235Z   main - INFO Saving structure of stream TariffHeadingType (tap_stream_id: LBLA_dbo_TariffHeadingType)
2020-05-13 17:42:16,248Z   main - INFO Saving structure of stream Filing (tap_stream_id: LBLA_Staging_dbo_Filing)
2020-05-13 17:42:16,266Z   main - INFO Saving structure of stream Holiday (tap_stream_id: LBLA_dbo_Holiday)
2020-05-13 17:42:16,279Z   main - INFO Saving structure of stream vw_AllEffectiveTariffRatesRules (tap_stream_id: LBLA_dbo_vw_AllEffectiveTariffRatesRules)
2020-05-13 17:42:16,297Z   main - INFO Saving structure of stream Refinery (tap_stream_id: LBLA_Staging_dbo_Refinery)
2020-05-13 17:42:16,311Z   main - INFO Saving structure of stream vw_aspnet_MembershipUsers (tap_stream_id: LBLA_Staging_dbo_vw_aspnet_MembershipUsers)
2020-05-13 17:42:16,330Z   main - INFO Saving structure of stream vwExt_AERDiscount (tap_stream_id: LBLA_Staging_dbo_vwExt_AERDiscount)
2020-05-13 17:42:16,347Z   main - INFO Saving structure of stream RefineryStation (tap_stream_id: LBLA_Staging_dbo_RefineryStation)
2020-05-13 17:42:16,362Z   main - INFO Saving structure of stream TariffRate (tap_stream_id: LBLA_Staging_dbo_TariffRate)
2020-05-13 17:42:16,376Z   main - INFO Saving structure of stream vw_AllEffectiveCancelTariffRatesRules (tap_stream_id: LBLA_dbo_vw_AllEffectiveCancelTariffRatesRules)
2020-05-13 17:42:16,394Z   main - INFO Saving structure of stream TariffTariffType (tap_stream_id: LBLA_Staging_dbo_TariffTariffType)
2020-05-13 17:42:16,409Z   main - INFO Saving structure of stream Jurisdiction (tap_stream_id: LBLA_dbo_Jurisdiction)
2020-05-13 17:42:16,425Z   main - INFO Saving structure of stream aspnet_SchemaVersions (tap_stream_id: LBLA_dbo_aspnet_SchemaVersions)
2020-05-13 17:42:16,439Z   main - INFO Saving structure of stream vwExt_GoverningTariff (tap_stream_id: LBLA_Staging_dbo_vwExt_GoverningTariff)
2020-05-13 17:42:16,452Z   main - INFO Saving structure of stream vw_RatesHistory (tap_stream_id: LBLA_Staging_dbo_vw_RatesHistory)
2020-05-13 17:42:16,472Z   main - INFO Saving structure of stream vw_aspnet_Roles (tap_stream_id: LBLA_dbo_vw_aspnet_Roles)
2020-05-13 17:42:16,484Z   main - INFO Saving structure of stream StationAlias (tap_stream_id: LBLA_Staging_dbo_StationAlias)
2020-05-13 17:42:16,500Z   main - INFO Saving structure of stream vw_aspnet_Users (tap_stream_id: LBLA_dbo_vw_aspnet_Users)
2020-05-13 17:42:16,515Z   main - INFO Saving structure of stream vwExt_AERExtCharge (tap_stream_id: LBLA_dbo_vwExt_AERExtCharge)
2020-05-13 17:42:16,528Z   main - INFO Saving structure of stream TariffStatus (tap_stream_id: LBLA_dbo_TariffStatus)
2020-05-13 17:42:16,541Z   main - INFO Saving structure of stream vwExt_JointCarrier (tap_stream_id: LBLA_Staging_dbo_vwExt_JointCarrier)
2020-05-13 17:42:16,556Z   main - INFO Saving structure of stream vwExt_State (tap_stream_id: LBLA_dbo_vwExt_State)
2020-05-13 17:42:16,573Z   main - INFO Writing catalog to file
2020-05-13 17:42:16,774Z   main - INFO Selected fields for stream LBLA_dbo_TariffHeadingType: Name, Id
2020-05-13 17:42:16,774Z   main - INFO Selected fields for stream LBLA_dbo_TariffTariffType: TariffId, TariffTypeId
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_ExtraCharge: TariffRateId, Value, Id, ExtraChargeTypeId, IsCancelled, Note, LastUpdated
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_Form6Report: Period, Url, ReportVerifiedDate, Id, Description, LinkTypeId, PeriodTypeId, IsReportVerified, ReportTypeId, Year, Notes, FiledDate, PipelineId
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_ExtraChargeType: Name, EnforceNumericValue, FormatString, ExtraChargeTypeGroupId, Id, IsSubjectToIndexAdjustment
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_Refinery: Name, Id, Site, StateId
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_sysdiagrams: definition, principal_id, diagram_id, version
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_UserActivity: Url, Id, UserId, Date, Activity
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_PipelineSystemTariff: TariffId, PipelineSystemId
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_Jurisdiction: Id, ShortName, LongName, StateId, IsDefault
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_RelatedTariff: TariffId, RelatedTariffId, RelatedTariffTypeId
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_aspnet_Applications: ApplicationName, Description, ApplicationId, LoweredApplicationName
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_CommodityType: Name, Id, IsSearchable
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_ExtraChargeTypeGroup: Name, Id
2020-05-13 17:42:16,775Z   main - INFO Selected fields for stream LBLA_dbo_TariffType: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_TariffStatus: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_Pipeline: Name, Id, Url, Notes, MapUrl
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_Filing: DocketYear, DocketSubNumber, DocketSheetUrl, Id, IsApproved, FiledDate, DocketNumber, AmendmentProtestSummary, CommissionAction, DocketPrefix, Notes, FilingUrl, DataNotes, DocketMainNumber, LastUpdated, Summary, PipelineId
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_State: Name, Id, PaddId, Abbreviation
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_Commodity: Name, Id, CommodityTypeId, Description
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_PipelineSystem: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_BaseRateAuthorityType: Name, Id, Description
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_TariffHeading: TariffId, TariffHeadingTypeId
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_PeriodType: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_Currency: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_ReportType: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_RateAuthorityType: Name, Id, Description, IsNewRate, IsChangedRate
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_VolumeMeasurement: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_aspnet_Membership: PasswordAnswer, ApplicationId, PasswordSalt, FailedPasswordAttemptCount, FailedPasswordAttemptWindowStart, IsApproved, MobilePIN, FailedPasswordAnswerAttemptCount, LoweredEmail, PasswordFormat, Password, PasswordQuestion, LastPasswordChangedDate, Email, CreateDate, FailedPasswordAnswerAttemptWindowStart, LastLockoutDate, UserId, IsLockedOut, LastLoginDate
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_RelatedTariffType: Name, Id
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_aspnet_Profile: UserId, LastUpdatedDate
2020-05-13 17:42:16,776Z   main - INFO Selected fields for stream LBLA_dbo_TariffCommodity: TariffId, CommodityId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_TariffRateAuthority: TariffId, RateAuthorityTypeId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_aspnet_UsersInRoles: RoleId, UserId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_Holiday: Name, Id, JurisdictionId, HolidayDate
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_MarketStation: StationId, MarketId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_aspnet_Roles: Description, RoleId, ApplicationId, LoweredRoleName, RoleName
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_StationAlias: AliasStationId, StationId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_PADD: Name, Id, DistrictName, SubdistrictName
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_JointCarrier: TariffId, PipelineId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_LinkType: Name, Id
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_TariffRate: DestinationStationId, OriginStationId, CeilingRate, Id, IsCancelled, BaseRateNote, SpecificProductId, TariffId, RouteNumber, RouteNote, BaseRate, BaseRateAuthorityTypeId, LastUpdated
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_aspnet_SchemaVersions: IsCurrentVersion, Feature, CompatibleSchemaVersion
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_RefineryStation: RefineryId, StationId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_SpecificProduct: Name, Id
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_TariffRateDiscount: RateNote, IsCommitted, RateValue, MinVolume, Id, LastUpdated, IsCancelled, TermLength, MaxVolume, TariffRateId
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_Market: Name, Id, Description
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_aspnet_Users: UserName, ApplicationId, LoweredUserName, IsAnonymous, MobileAlias, UserId, LastActivityDate
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_vwExt_TariffCommodity: CommodityID, TariffID
2020-05-13 17:42:16,777Z   main - INFO Selected fields for stream LBLA_dbo_Station: ONRRId, Name, Diameter, SubstationName, Id, Latitude, ONRRNote, Longitude, EnviromentCanadaNote, FRSId, FRSNote, DiameterNote, IRSId, AddressNumber, DEQId, EnviromentCanadaId, DEQNote, AddressStreet, CountyName, City, AlternateCity, AddressNote, IRSNote, StateId
2020-05-13 17:42:16,778Z   main - INFO Selected fields for stream LBLA_dbo_Tariff: ExpireDate, Name, EffectiveDate, DataNotes, Id, TariffStatusId, TariffVerifiedDate, Url, Summary, PipelineId, IsRateDecrease, CurrencyId, IsInitialRate, IsHighValue, IsApproved, IssueDate, FilingId, TariffNumber, VolumeMeasurementId, CommodityDescription, SupplementNumber, JurisdictionId, IsRateIncrease, LastUpdated
2020-05-13 17:42:16,782Z   main - INFO Current state: {}
2020-05-13 17:42:16,783Z   main - INFO Starting tap: tap-env/bin/tap-mssql --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2020-05-13 17:42:16,788Z   main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2020-05-13 17:42:17,489Z target - INFO Using batch_size_prefernces of {'batch_size_preference': None, 'user_batch_size_preference': None, 'full_table_streams': ['vwExt_TariffCommodity']}
2020-05-13 17:42:17,490Z target - INFO Using turbo_boost_factor of 5
2020-05-13 17:42:20,147Z    tap - INFO Started nrepl server at /0.0.0.0:32925
2020-05-13 17:42:20,315Z    tap - INFO Starting sync mode
2020-05-13 17:42:20,316Z    tap - INFO Skipping stream LBLA_dbo_vwExt_ExtChargeType
2020-05-13 17:42:20,316Z    tap - INFO Skipping stream LBLA_dbo_vw_DOTFlatRates
2020-05-13 17:42:20,316Z    tap - INFO Skipping stream LBLA_dbo_vwExt_Station
2020-05-13 17:42:20,316Z    tap - INFO Skipping stream LBLA_dbo_vwExt_PADD
2020-05-13 17:42:20,317Z    tap - INFO Syncing stream LBLA_dbo_SpecificProduct using replication method null
2020-05-13 17:42:20,321Z    tap - java.lang.IllegalArgumentException: Replication Method for stream LBLA_dbo_SpecificProduct is invalid: null
2020-05-13 17:42:20,321Z    tap - 	at tap_mssql.singer.messages$maybe_write_activate_version_BANG_.invokeStatic(messages.clj:144)
2020-05-13 17:42:20,321Z    tap - 	at tap_mssql.singer.messages$maybe_write_activate_version_BANG_.invoke(messages.clj:123)
2020-05-13 17:42:20,322Z    tap - 	at tap_mssql.core$sync_stream_BANG_.invokeStatic(core.clj:122)
2020-05-13 17:42:20,322Z    tap - 	at tap_mssql.core$sync_stream_BANG_.invoke(core.clj:116)
2020-05-13 17:42:20,322Z    tap - 	at tap_mssql.core$maybe_sync_stream_BANG_.invokeStatic(core.clj:133)
2020-05-13 17:42:20,322Z    tap - 	at tap_mssql.core$maybe_sync_stream_BANG_.invoke(core.clj:129)
2020-05-13 17:42:20,322Z    tap - 	at clojure.core$partial$fn__5563.invoke(core.clj:2624)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core.protocols$fn__7852.invokeStatic(protocols.clj:168)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core.protocols$fn__7852.invoke(protocols.clj:124)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core.protocols$fn__7807$G__7802__7816.invoke(protocols.clj:19)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core.protocols$fn__7835.invokeStatic(protocols.clj:75)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core.protocols$fn__7835.invoke(protocols.clj:75)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core.protocols$fn__7781$G__7776__7794.invoke(protocols.clj:13)
2020-05-13 17:42:20,323Z    tap - 	at clojure.core$reduce.invokeStatic(core.clj:6748)
2020-05-13 17:42:20,324Z    tap - 	at clojure.core$reduce.invoke(core.clj:6730)
2020-05-13 17:42:20,324Z    tap - 	at tap_mssql.core$do_sync.invokeStatic(core.clj:143)
2020-05-13 17:42:20,324Z    tap - 	at tap_mssql.core$do_sync.invoke(core.clj:139)
2020-05-13 17:42:20,324Z    tap - 	at tap_mssql.core$_main.invokeStatic(core.clj:169)
2020-05-13 17:42:20,324Z    tap - 	at tap_mssql.core$_main.doInvoke(core.clj:154)
2020-05-13 17:42:20,324Z    tap - 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
2020-05-13 17:42:20,324Z    tap - 	at tap_mssql.core.main(Unknown Source)
2020-05-13 17:42:20,325Z    tap - FATAL Replication Method for stream LBLA_dbo_SpecificProduct is invalid: null
2020-05-13 17:42:20,325Z    tap - INFO Shutting down the nrepl server
2020-05-13 17:42:20,352Z target - INFO Requests complete, stopping loop
2020-05-13 17:42:20,420Z   main - INFO Target exited normally with status 0
2020-05-13 17:42:20,422Z   main - INFO Closed tunnel and deleted temporary keyfile
2020-05-13 17:42:20,422Z   main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "Replication Method for stream LBLA_dbo_SpecificProduct is invalid: null". Target succeeded.

Errors when replicating tables with spaces in column names

I'm trying to replicate data from a source where some column names contain spaces. This causes the extraction to fail with the following output:

2021-02-01 05:47:56,107Z    tap - 	at clojure.core$reduce.invokeStatic(core.clj:6747)
2021-02-01 05:47:56,107Z    tap - 	at clojure.core$reduce.invoke(core.clj:6730)
2021-02-01 05:47:56,107Z    tap - 	at tap_mssql.sync_strategies.incremental$sync_and_write_messages_BANG_.invokeStatic(incremental.clj:50)
2021-02-01 05:47:56,107Z    tap - 	at tap_mssql.sync_strategies.incremental$sync_and_write_messages_BANG_.invoke(incremental.clj:33)
2021-02-01 05:47:56,107Z    tap - 	at tap_mssql.sync_strategies.incremental$sync_BANG_.invokeStatic(incremental.clj:66)
2021-02-01 05:47:56,107Z    tap - 	at tap_mssql.sync_strategies.incremental$sync_BANG_.invoke(incremental.clj:61)
2021-02-01 05:47:56,107Z    tap - 	at tap_mssql.core$dispatch_sync_by_strategy.invokeStatic(core.clj:109)
2021-02-01 05:47:56,107Z    tap - 	at tap_mssql.core$dispatch_sync_by_strategy.invoke(core.clj:99)
2021-02-01 05:47:56,108Z    tap - 	at tap_mssql.core$sync_stream_BANG_.invokeStatic(core.clj:123)
2021-02-01 05:47:56,108Z    tap - 	at tap_mssql.core$sync_stream_BANG_.invoke(core.clj:116)
2021-02-01 05:47:56,108Z    tap - 	at tap_mssql.core$maybe_sync_stream_BANG_.invokeStatic(core.clj:133)
2021-02-01 05:47:56,108Z    tap - 	at tap_mssql.core$maybe_sync_stream_BANG_.invoke(core.clj:129)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core$partial$fn__5563.invoke(core.clj:2624)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core.protocols$fn__7852.invokeStatic(protocols.clj:168)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core.protocols$fn__7852.invoke(protocols.clj:124)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core.protocols$fn__7807$G__7802__7816.invoke(protocols.clj:19)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core.protocols$fn__7835.invokeStatic(protocols.clj:75)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core.protocols$fn__7835.invoke(protocols.clj:75)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core.protocols$fn__7781$G__7776__7794.invoke(protocols.clj:13)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core$reduce.invokeStatic(core.clj:6748)
2021-02-01 05:47:56,108Z    tap - 	at clojure.core$reduce.invoke(core.clj:6730)
2021-02-01 05:47:56,108Z    tap - 	at tap_mssql.core$do_sync.invokeStatic(core.clj:143)
2021-02-01 05:47:56,108Z    tap - 	at tap_mssql.core$do_sync.invoke(core.clj:139)
2021-02-01 05:47:56,108Z    tap - 	at tap_mssql.core$_main.invokeStatic(core.clj:169)
2021-02-01 05:47:56,109Z    tap - 	at tap_mssql.core$_main.doInvoke(core.clj:154)
2021-02-01 05:47:56,109Z    tap - 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
2021-02-01 05:47:56,109Z    tap - 	at tap_mssql.core.main(Unknown Source)
2021-02-01 05:47:56,109Z    tap - FATAL Incorrect syntax near 'No_'.

Is there a workaround or any plans to address this in a fix?

Allow overwriting the java path

The project.clj specifies the jdk path directly, meaning it will only work on ubuntu installations. For example, in alpine, the default jdk path is /usr/lib/jvm/java-1.8-openjdk/ and not /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java. I am not familiar with Leiningen and Clojure, but there should be a better way then either overriding the project.clj or creating a profiles.clj and moving that in the project directory.

Incremental sync strategy fails for timestamp/rowversion replication keys

Using a timestamp / rowversion column as a replication key causes the incremental strategy to error due to the byte array that's being written to STATE being passed to the where clause.

I tested that converting the replication-key-value back into a number allows it to complete successfully.

Example state:

{"bookmarks": {"TEST_dbo_person": {"replication_key_value": [0, 0, 0, 0, 0, 0, 7, -25], "replication_key_name": "RowVers", "version": 1585608487075}}}

Test Schema:

{
    "streams": [{
        "stream": "person",
        "tap_stream_id": "TEST_dbo_person",
        "table_name": "person",
        "schema": {
            "type": "object",
            "properties": {
                "person_id": {
                    "type": ["integer"],
                    "minimum": -2147483648,
                    "maximum": 2147483647
                },
                "first_name": {
                    "type": ["string", "null"],
                    "maxLength": 50
                },
                "last_name": {
                    "type": ["string", "null"],
                    "maxLength": 50
                },
                "gender": {
                    "type": ["string", "null"],
                    "maxLength": 1
                },
                "RowVers": {
                    "type": ["string", "null"]
                },
                "Edit_DtTm": {
                    "type": ["string", "null"],
                    "format": "date-time"
                },
                "Create_DtTm": {
                    "type": ["string", "null"],
                    "format": "date-time"
                }
            }
        },
        "metadata": [{
            "metadata": {
                "database-name": "TEST",
                "schema-name": "dbo",
                "table-key-properties": ["person_id"],
                "is-view": false,
                "row-count": 1,
                "selected": true,
                "replication-method": "INCREMENTAL",
                "replication-key": "RowVers"
            },
            "breadcrumb": []
        }, {
            "metadata": {
                "inclusion": "automatic",
                "sql-datatype": "int identity",
                "selected-by-default": true
            },
            "breadcrumb": ["properties", "person_id"]
        }, {
            "metadata": {
                "inclusion": "available",
                "sql-datatype": "varchar",
                "selected-by-default": true
            },
            "breadcrumb": ["properties", "first_name"]
        }, {
            "metadata": {
                "inclusion": "available",
                "sql-datatype": "varchar",
                "selected-by-default": true
            },
            "breadcrumb": ["properties", "last_name"]
        }, {
            "metadata": {
                "inclusion": "available",
                "sql-datatype": "char",
                "selected-by-default": true
            },
            "breadcrumb": ["properties", "gender"]
        }, {
            "metadata": {
                "inclusion": "available",
                "sql-datatype": "timestamp",
                "selected-by-default": true
            },
            "breadcrumb": ["properties", "RowVers"]
        }, {
            "metadata": {
                "inclusion": "available",
                "sql-datatype": "datetime",
                "selected-by-default": true
            },
            "breadcrumb": ["properties", "Edit_DtTm"]
        }, {
            "metadata": {
                "inclusion": "available",
                "sql-datatype": "datetime",
                "selected-by-default": true
            },
            "breadcrumb": ["properties", "Create_DtTm"]
        }]
    }]
}

Example stack trace:

INFO Successfully connected to the instance
INFO Syncing stream TEST_dbo_person using replication method INCREMENTAL
INFO Executing query: ("SELECT [person_id], [first_name], [last_name], [gender], [RowVers], [Edit_DtTm], [Create_DtTm] FROM [dbo].[person] WHERE RowVers >= ? ORDER BY RowVers" [0 0 0 0 0 0 7 -25])
INFO Mapping: customer to ['customer']
INFO Mapping: tp_customer_person_id__sdc_sequence_idx to None
INFO Mapping: tp_person__1585608487075_person_id__sdc_sequence_idx to None
INFO Mapping: person to ['person']
INFO Stream customer (customer) with max_version None targetting None
INFO Root table name customer
INFO Writing batch with 0 records for `customer` with `key_properties`: `['person_id']`
INFO Writing table batch schema for `('customer',)`...
com.microsoft.sqlserver.jdbc.SQLServerException: The conversion from UNKNOWN to UNKNOWN is unsupported.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:228)
	at com.microsoft.sqlserver.jdbc.DataTypes.throwConversionError(DataTypes.java:1094)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.setObject(SQLServerPreparedStatement.java:1595)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.setObjectNoType(SQLServerPreparedStatement.java:1493)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.setObject(SQLServerPreparedStatement.java:1502)
	at clojure.java.jdbc$eval705$fn__706.invokePrim(jdbc.clj:484)
	at clojure.java.jdbc$eval705$fn__706.invoke(jdbc.clj)
	at clojure.java.jdbc$eval684$fn__685$G__675__694.invoke(jdbc.clj:471)
	at clojure.java.jdbc$dft_set_parameters.invokeStatic(jdbc.clj:495)
	at clojure.java.jdbc$dft_set_parameters.invoke(jdbc.clj:490)
	at clojure.java.jdbc$execute_query_with_params.invokeStatic(jdbc.clj:1071)
	at clojure.java.jdbc$execute_query_with_params.invoke(jdbc.clj:1066)
	at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1095)
	at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:1075)
	at clojure.java.jdbc$reducible_query$reify__944.reduce(jdbc.clj:1339)
	at clojure.core$reduce.invokeStatic(core.clj:6747)
	at clojure.core$reduce.invoke(core.clj:6730)
	at tap_mssql.sync_strategies.incremental$sync_and_write_messages_BANG_.invokeStatic(incremental.clj:50)
	at tap_mssql.sync_strategies.incremental$sync_and_write_messages_BANG_.invoke(incremental.clj:33)
	at tap_mssql.sync_strategies.incremental$sync_BANG_.invokeStatic(incremental.clj:66)
	at tap_mssql.sync_strategies.incremental$sync_BANG_.invoke(incremental.clj:61)
	at tap_mssql.core$dispatch_sync_by_strategy.invokeStatic(core.clj:109)
	at tap_mssql.core$dispatch_sync_by_strategy.invoke(core.clj:99)
	at tap_mssql.core$sync_stream_BANG_.invokeStatic(core.clj:123)
	at tap_mssql.core$sync_stream_BANG_.invoke(core.clj:116)
	at tap_mssql.core$maybe_sync_stream_BANG_.invokeStatic(core.clj:133)
	at tap_mssql.core$maybe_sync_stream_BANG_.invoke(core.clj:129)
	at clojure.core$partial$fn__5563.invoke(core.clj:2624)
	at clojure.core.protocols$fn__7852.invokeStatic(protocols.clj:168)
	at clojure.core.protocols$fn__7852.invoke(protocols.clj:124)
	at clojure.core.protocols$fn__7807$G__7802__7816.invoke(protocols.clj:19)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
	at clojure.core.protocols$fn__7835.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__7835.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__7781$G__7776__7794.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6748)
	at clojure.core$reduce.invoke(core.clj:6730)
	at tap_mssql.core$do_sync.invokeStatic(core.clj:143)
	at tap_mssql.core$do_sync.invoke(core.clj:139)
	at tap_mssql.core$_main.invokeStatic(core.clj:169)
	at tap_mssql.core$_main.doInvoke(core.clj:154)
	at clojure.lang.RestFn.invoke(RestFn.java:512)
	at clojure.lang.Var.invoke(Var.java:411)
	at user$eval149.invokeStatic(form-init8490039590763633355.clj:1)
	at user$eval149.invoke(form-init8490039590763633355.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:7062)
	at clojure.lang.Compiler.eval(Compiler.java:7052)
	at clojure.lang.Compiler.load(Compiler.java:7514)
	at clojure.lang.Compiler.loadFile(Compiler.java:7452)
	at clojure.main$load_script.invokeStatic(main.clj:278)
	at clojure.main$init_opt.invokeStatic(main.clj:280)
	at clojure.main$init_opt.invoke(main.clj:280)
	at clojure.main$initialize.invokeStatic(main.clj:311)
	at clojure.main$null_opt.invokeStatic(main.clj:345)
	at clojure.main$null_opt.invoke(main.clj:342)
	at clojure.main$main.invokeStatic(main.clj:424)
	at clojure.main$main.doInvoke(main.clj:387)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:702)
	at clojure.main.main(main.java:37)
FATAL The conversion from UNKNOWN to UNKNOWN is unsupported.
INFO Shutting down the nrepl server

Re-fetching duplicate rows, decremented CHANGE_TRACKING_CURRENT_VERSION()

tap-mssql/src/tap_mssql/sync_strategies/logical.clj

Line 166 in 81cce9b

(dec current-log-version)

This spawned from a discussion with Carlos at Stitch. I believe there may be an off-by-one error with the CHANGE_TRACKING_CURRENT_VERSION being passed into SQL Server's CHANGETABLE function. The SQL Server documentation recommends directly passing the version number retrieved by CHANGE_TRACKING_CURRENT_VERSION() into the CHANGETABLE function: https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/work-with-change-tracking-sql-server?view=sql-server-2017 . If this version number is passed as-is, tables with no new updates should return zero rows (at least, I believe this is what SQL Server documentation says).

But the implementation shown in line 166 above decrements the retrieved number by one. This results in lots of duplicated copying of data, and seems to disagree with the SQL Server documentation.

Is there a reason for the decrement-by-one behavior?

Custom catalog `table-key-properties` omitted in initial SCHEMA message `key_properties`

Context:

For tables which do not have logical keys set on the underlying upstream database, we create a custom catalog file with table-key-properties set. However, this metadata appears to be missing in some schema messages, and importantly the first SCHEMA message, which is used in downstream table creation. (Missing primary key is an error by default with pipelinewise-target-snowflake, for example.)

Log excerpt:

(Note the first SCHEMA message has incorrect/missing key_properties, but the second message on line 9 is correct.)

{"type":"SCHEMA","stream":"Resources","key_properties":[],"schema":{...}}
{"type":"ACTIVATE_VERSION","stream":"Resources","version":1600290773862}
{"type":"STATE","stream":"Changepoint_Integration_Resources","value":{"bookmarks":{"Changepoint_Integration_Resources":{"version":1600290773862}}}}
{"type":"RECORD","stream":"Resources","record":{...},"version":1600290773862}
...
{"type":"RECORD","stream":"Resources","record":{...},"version":1600290773862}
{"type":"ACTIVATE_VERSION","stream":"Resources","version":1600290773862}
{"type":"STATE","stream":"Changepoint_Integration_Resources","value":{"bookmarks":{"Changepoint_Integration_Resources":{"version":1600290773862}}}}
{"type":"SCHEMA","stream":"Resources","key_properties":["ResourceId"],"schema":{...}}
{"type":"ACTIVATE_VERSION","stream":"Resources","version":1600290801173}
{"type":"STATE","stream":"Changepoint_dbo_Resources","value":{"bookmarks":{"Changepoint_Integration_Resources":{"version":1600290773862},"Changepoint_dbo_Resources":{"version":1600290801173,"max_pk_values":{"ResourceId":"09C7314D-97AD-4460-9209-FFFF83AC6CB8"}}}}}
{"type":"RECORD","stream":"Resources","record":{...},"version":1600290801173}
...

Caveat:

This is using a slightly stale version (by a few months at most?). Please let me know if this was known to be an issue and should be fixed by upgrading to the latest version.

Incremental sync strategy fails on Azure SQL when db_id is different to sys.change_tracking_databases entry

I had change tracking enabled on an Azure SQL database and wanted to do an incremental sync using Stitch, it was blowing up with Change Tracking is not enabled for database. Long story short, in the code here I see the trouble stems from SELECT DB_NAME(database_id) AS db_name FROM sys.change_tracking_databases.

For some bizarre reason, in Azure SQL, the database ID numbers in sys.* tables can fall out of sync with the database ID numbers used by DB_NAME() and DB_ID() (see note in Microsoft docs), and this query directly compares the two, so my valid entry in sys.change_tracking_databases was put through DB_NAME() which output NULL.

The MS-recommended solution is to compare like for like, i.e. to look up the name from sys.databases instead.

PR to follow!

Support for deprecated data types `text` and `ntext`

Background:

The tap currently does not support the data types text and ntext. Although these are deprecated data types as noted here, there are still many environments with instances of these column types in the wild.

Our organization has some such columns (type of text) and the resulting generated catalog is omitting the column's type declaration. Since the downstream target (snowflake in our case) is expecting a data type, this causes a hard failure trying to parsing a dict with unknown key.

Proposal:

The proposed solution would handle text as varchar(max) and ntext as nvarchar(max).

Code location:

tap-mssql/src/tap_mssql/catalog.clj

Line 171 in b044ae1

"varchar" {"type" ["string"]

Alternative considered:

An alternative or additional solution would be to fallback to string data types as an else clause in the column type mapping, which would ensure that downstream targets always receive a valid catalog file with some kind of type declaration for each column, rather than having an omitted type field. Since basically all types can be represented as strings in some form, this seems like a good fallback rather than generating a catalog with missing types.

A benefit to this else condition alternative is that there are sometimes other reasons why columns cannot declare their metadata. In rare scenarios we've seen MS SQL Server return a null/missing data type for views that can/t or don't properly inspect their own metdata, for instance in this hypothetical sample scenario:

select null as col_a from table_one
union all
select 123 as col_a from table_two
union all
select '234' as col_b from table_three

The first instance in the union normally declares type but it is null so it may default to int or unknown. The third instance in the union being a string will probably succeed at implicit conversion to integer type, but again, this might not be able to be resolved - or might be resolved inconsistently - at view creation time.

Related article regarding text and ntext data types:

https://docs.microsoft.com/en-us/sql/t-sql/data-types/ntext-text-and-image-transact-sql?view=sql-server-ver15

image

Extraction giving error message about TLS Version

Hello,
We use the Stitch to replicate our SQL Server data and it seems that we are now getting the following errors when running extraction:

INFO Exit status is: Discovery failed with code 1 and error message: "The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "The server selected protocol version TLS10 is not accepted by client preferences [TLS12]"

Has anyone else encountered this or know what the solution is to get our extraction up and running again?
I see knew code was pushed two days ago when our issue first popped up. Any insight @leslievandemark on if this is related?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.