Pandas 2.0 support about databricks-sql-python HOT 12 CLOSED

Thanks for the ping. We'll have the answer today about the timeline.

from databricks-sql-python.

After discussion with the team we've actually deprioritised this. We won't consider a timeline for it again for at least a few more weeks. I know that's not a great answer ☹️ It's a side-effect of a lot of things in the pipeline for both this connector and dbt-databricks (its biggest user).

Perhaps you can help with this? Long-term we'd like to remove the pandas dependency from the connector entirely. Short-term, do you think it would be acceptable to simply update pyproject.toml to allow pandas==^2.0.0? If so, we would absolutely review a pull request implementing such a change. If it passes our test suite then there's little risk.

This wouldn't work if pandas 2.0 support involves significant syntax changes to the few places where we import pandas as pd and do something with it.

from databricks-sql-python.

This is a duplicate of #91. We'll discuss timelines for this and update that ticket within the next week.

Whoops! Got my wires crossed. This isn’t a duplicate. We’ll still have a timeline for it this week.

from databricks-sql-python.

Hello @susodapop 👋
Sorry to be that person....
Any idea when this will be on the roadmap?

from databricks-sql-python.

@susodapop I had a quick attempt on it, all unit tests pass with pandas@^2.0.0 (it works on my machine 😅).
But I had to update the python version to >=3.8.0. Is this acceptable?

from databricks-sql-python.

from databricks-sql-python.

@susodapop Any thoughts on my comment above about python >=3.8.0

from databricks-sql-python.

Dropping support for Python 3.7 is a significant decision that we need to adequately consider how it will affect other users. I wonder if Poetry will allow us to specify an older version of pandas for Python 3.7 and a newer version of Pandas for >3.8?

from databricks-sql-python.

@PedroBrochado @susodapop, I think either of these approaches would work:

• pandas = ">=1.2.5": https://github.com/cdcadman/databricks-sql-python/commit/5d37969fe5f1adbeb6c925286630ae4b4312173a
• pandas = [
  {version = "^2.0.0", python = ">=3.8"},
  {version = "^1.2.5", python = "<3.8"}
  ]: https://github.com/cdcadman/databricks-sql-python/commit/11c07989451425ba5fbeaface6f3bf5c752b3a46

from databricks-sql-python.

Is there any update for pandas 2.0 support? Thank you

from databricks-sql-python.

bumping for pandas 2.0 support

from databricks-sql-python.

We released version 2.8.0 which allows installation alongside pandas==2.0

https://github.com/databricks/databricks-sql-python/releases/tag/v2.8.0

from databricks-sql-python.