deputyapp / singer-tap-deputy Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 7.0 34 KB

Deputy Stitch Stream

License: MIT License

Python 100.00%

singer-tap-deputy's People

Contributors

Stargazers

Watchers

Forkers

singer-io kilometer8629

singer-tap-deputy's Issues

Datetimeoffset conversion error

All columns of type date-time (created, modified, _sdc_batched_at, etc.) with values between 2024-01-01T00:00:00 and 2024-01-01T23:59:59 got converted to 2025-01-01T__:__:__, adding 1 year to all of them.

This issue was raised with Stitch support and was redirected here.

Submission Review Notes

Adding notes to this issue as I review this tap for acceptance. These should be rather small but important changes to make before we can move this tap forward.

Metadata
It looks like the custom metadata for resource is written has the wrong namespace. It should be tap-deputy.resource.

Del on kwargs
Should this line be calling delete on a parameter passed in? I'm not quite sure how Python behaves in all kwargs situations (e.g., splat, named keywords, etc.), but this seems like a side-effect that would be better avoided.

all_selected
This should be removed from here. I see the usefulness of this feature to enable testing without having to select all streams, however it should be an optional configuration argument, rather than implied by the absence of a catalog. The absence of a catalog should mean no streams are selected, since that's what it is.

current_stream
This isn't being handled correctly (here), it should attempt to run through every stream when it finds a current_stream state entry, rather than skipping earlier streams and only attempting them on the next tap run. We just created a helper function to manage this called get_selected_streams on the Catalog class in singer-python 5.7 (implementation)

Add documentation to the tap

What should go in the config?
What data will be extracted by the tap?

Missing License

Here's the license included in the template for writing a Singer tap:
GNU Affero General Public License v3.0

Update `requirements.txt` to `setup.py`

Here is an example of a setup.py file.

CloudQuery Source Plugin?

Hi Team, hopefully this is right place to ask, if not, I'd appreciate if you can direct me.

I'm the founder of cloudquery.io, a high performance open source ELT framework.

Our users are interested in an Deputy plugin, but as we cannot maintain all the plugins ourselves, I was curious if this would be an interesting collaboration, where we would help implement an initial source plugin, and you will help maintain it.

This will give your users the ability to sync Deputy data to any of their datalakes/data-warehouses/databases easily using any of the growing list of CQ destination plugins.

Best,
Yevgeny

Missing type `Json` in TYPE_MAP

Resource EmployeeAgreement has a newly added field (Config) which has a new type (Json, as reported by the INFO endpoint). This new field type is not accounted for in TYPE_MAP in discover.py, and is throwing an error every time Stitch (or singer) tries to perform an extraction.

singer-tap-deputy/tap_deputy/discover.py

Lines 64 to 71 in 6126aba

 TYPE_MAP = { 

 'Integer': 'integer', 

 'Float': 'number', 

 'VarChar': 'string', 

 'Blob': 'string', 

 'Bit': 'boolean', 

 'Time': 'string' 

 }

Logs

$ tap-deputy -c config.json --dev
WARNING Executing Tap in Dev mode
DEBUG Starting new HTTPS connection (1): [REDACTED].eu.deputy.com:443
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Address/INFO HTTP/1.1" 200 208
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.2956879138946533, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Category/INFO HTTP/1.1" 200 177
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.1277756690979004, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Comment/INFO HTTP/1.1" 200 175
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.11403942108154297, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Company/INFO HTTP/1.1" 200 254
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.11162185668945312, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://276be407021147.eu.deputy.com:443 "GET /api/v1/resource/CompanyPeriod/INFO HTTP/1.1" 200 166
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.09200048446655273, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Contact/INFO HTTP/1.1" 200 215
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.10479259490966797, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Country/INFO HTTP/1.1" 200 184
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.10700106620788574, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/CustomAppData/INFO HTTP/1.1" 200 189
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.16899871826171875, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/CustomField/INFO HTTP/1.1" 200 277
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.09703683853149414, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/CustomFieldData/INFO HTTP/1.1" 200 410
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.1805715560913086, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/Employee/INFO HTTP/1.1" 200 427
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.22699999809265137, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
DEBUG https://[REDACTED].eu.deputy.com:443 "GET /api/v1/resource/EmployeeAgreement/INFO HTTP/1.1" 200 294
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.22404694557189941, "tags": {"endpoint": "resource_info", "http_status_code": 200, "status": "succeeded"}}
CRITICAL 'Json'
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "[REDACTED]\.venv\Scripts\tap-deputy.exe\__main__.py", line 7, in <module>
  File "[REDACTED]\.venv\lib\site-packages\singer\utils.py", line 235, in wrapped
    return fnc(*args, **kwargs)
  File "[REDACTED]\.venv\lib\site-packages\tap_deputy\__init__.py", line 50, in main
    sync(client,
  File "[REDACTED]\.venv\lib\site-packages\tap_deputy\sync.py", line 88, in sync
    catalog = discover(client)
  File "[REDACTED]\.venv\lib\site-packages\tap_deputy\discover.py", line 120, in discover
    schema_dict, metadata = get_schema(client, resource_name)
  File "[REDACTED]\.venv\lib\site-packages\tap_deputy\discover.py", line 96, in get_schema
    'type': ['null', TYPE_MAP[field_type]]
KeyError: 'Json'

replication method: full reload missing

since deputy doesn't have soft delete option.
eg. Once a roaster is deleted, this tap is not useful.

We are using stitchdata

Reorganize repository structure

Following the Singer Tap Template,
the directory should look something like:

/stitch-stream-deputy
  - README.md
  - LICENSE
  - .gitignore
  - MANIFEST.in
  - setup.py
  - stitch_stream_deputy/
    - schemas/
      - ...
    - __init__.py

Dates do not follow the RFC3339 format

Example from state.json:

"2018-11-09T08:52:57+11:00"

should be

2018-11-09T19:52:57+00:00

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	TYPE_MAP = {
	'Integer': 'integer',
	'Float': 'number',
	'VarChar': 'string',
	'Blob': 'string',
	'Bit': 'boolean',
	'Time': 'string'
	}