[BUG] about fugue HOT 5 CLOSED

@guilhermedelyra thanks so much for reporting!

Acutally prefect-fugue needs refactoring because it was designed before a major refactoring of Fugue.

After the fix on the prefect-fugue side, the experience should be even simpler:

import pandas as pd
import fugue.api as fa
from prefect import task, flow

@task
def load_data():
    return pd.read_parquet("https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2022-01.parquet")

@flow
def run_sql(top, engine):
    data = load_data()
    with fa.engine_context(engine):
        fa.fugue_sql_flow("""
        SELECT PULocationID, COUNT(*) AS ct FROM df
        GROUP BY 1 ORDER BY 2 DESC LIMIT 2
        PRINT
        """, data)


run_sql(1, "spark") # your current spark session (not recommended)
run_sql(1, "prefect_fugue/<spark_block>") # using the config in your prefect block to connect to remote spark/databricks

from fugue.

@guilhermedelyra for this one, this is a long due change, and I just made it. Thanks for raising this problem.

fugue-project/prefect-fugue#9

If you are interested in contributing to either the prefect-fugue or the fugue project, please let us know. We can discuss on slack channel.

Thanks!

from fugue.

@guilhermedelyra we have released prefect-fugue 0.0.3, and you can find instruction here: fugue-project/prefect-fugue#10

Please let us know if you have any question or feedbacks.

Thanks

from fugue.

that's really nice to know 🙂

Is there anything I can do to help?

from fugue.

thank you very much! 🙂
I'll be in contact any time soon

from fugue.