add support for TIMESTAMPTZ about ibis HOT 4 CLOSED

@djouallah I see there are two different issues here.

The one initially reported, about "TIMESTAMPTZ". To proved a timezone you need to use the Ibis timestamp dtype.

import ibis.expr.datatypes as dt
#then you can do something like
DUNIT = DUNIT.cast({"SETTLEMENTDATE": dt.Timestamp(timezone="UTC")})

The other issue, you are reporting on the second comment might be a bug. What seems to be happening is that at the moment the disk type that we are writing with is backend specific, and there seems to be backend inconsistencies when writing to disk. But the code provided, is not a minimal reproducible example as as it requires downloading some data from the internet. Would you be able to provide a
minimal reproducer (ideally with ibis.memtable()) so we can track this down.

from ibis.

@ncclementi thanks a lot, it was just me not reading the documentation, problem solved all good thanks for your help

from ibis.

Hi @djouallah thanks for opening this issue.

Can you give us more context on what are you trying to achieve? If possible a minimal reproducible example?

What's the original type of the SETLEMENTDATE column, and what are you trying to cast it too?

You can create an example using a memtable, this way we can reproduce it and help you out.

import ibis
t = ibis.memtable([{"a": 1}, {"a": 2}])

from ibis.

alright, I am trying ibis for the first time, my approach, load some data, do the transformation and write back to delta table, which works fine except that there is a subtle difference how arrow write timestamp and pyspark, i endup with two data type one with time zone the other not

see example here : AnalysisException: [DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields 'SETTLEMENTDATE' and 'SETTLEMENTDATE'

https://colab.research.google.com/drive/18JGPcj3VaJ2tcxDZx1XB7UufTnEPjRfu#scrollTo=51jIBhgTXzrS

from ibis.