Problem finding datetime_ambiguous on EMR about dagster HOT 5 CLOSED

worth to add, we run the same asset on Databricks, that runs on EC2 instances under the hood, and it does work with no issue in the newer version... so the problem is in when we submit it directly using the dagster_aws

from dagster.

This is very odd. On a new EMR instance, running pip freeze shows:

python-dateutil==2.8.1

But going into python and checking dateutil's version gives

>>> import dateutil
>>> dateutil.__version__
'2.2'

from dagster.

Oh weird, are there two dateutils in the default EMR python environment??

py-dateutil==2.2
python-dateutil==2.8.1

from dagster.

@HPicatto can you try pip uninstalling py-dateutil? That fixed the problem for me. Very odd that the default EMR environment includes both, py-dateutil had a single 2.2 release almost a decade ago: https://pypi.org/project/py-dateutil/#history

from dagster.

Raised this with AWS here: https://repost.aws/questions/QUl4F36LBhQu-TPbgalNJ-sQ/default-emr-spark-python-environment-contains-two-different-versions-of-the-dateutil-package

from dagster.