Git Product home page Git Product logo

Comments (5)

nicor88 avatar nicor88 commented on June 12, 2024

@dacreify @juliansteger-sc could you try the latest version from the main branch and verify that the fix solves your issue?

from dbt-athena.

dacreify avatar dacreify commented on June 12, 2024

@nicor88 I installed from main and cranked the threads back up on our project. I'm not able to reproduce the throttling now, but I guess it's hard to tell if this is the dbt-athena caching or AWS fixing their regression.

I noticed that the cache key includes the client object:

dbt-athena/dbt/adapters/athena/impl.py

Lines 219 to 224 in 3cdd7ee

@lru_cache()
def _get_work_group(self, client: AthenaClient, work_group: str) -> GetWorkGroupOutputTypeDef:
"""
helper function to cache the result of the get_work_group to avoid APIs throttling
"""
return client.get_work_group(WorkGroup=work_group)

Given the module-level lock here I'm guessing there's only one instance of athena_client in use for the whole run?

dbt-athena/dbt/adapters/athena/impl.py

Lines 232 to 240 in 3cdd7ee

with boto3_client_lock:
athena_client = client.session.client(
"athena",
region_name=client.region_name,
config=get_boto3_config(num_retries=creds.effective_num_retries),
)
if creds.work_group:
work_group = self._get_work_group(athena_client, creds.work_group)

Just confirming that I understand how the caching is working.

from dbt-athena.

nicor88 avatar nicor88 commented on June 12, 2024

@dacreify lru_cache allow you to cache result in case the same inputs are passed multiple times.
Given the fact that client and workgroup don't cange it should work as expected.
Not sure when using more threads if multiple clients are spawn per thread, I needed to check this behavior, in general lru_cache is thread safe.

from dbt-athena.

juliansteger-sc avatar juliansteger-sc commented on June 12, 2024

I'm not able to reproduce the throttling now, but I guess it's hard to tell if this is the dbt-athena caching or AWS fixing their regression.

same for us, but fix looks reasonable. thanks for investigating & fixing

from dbt-athena.

nicor88 avatar nicor88 commented on June 12, 2024

Let's close this issue for now, given the fact that either AWS or the caching behaviour helped to mitigate the issue.

from dbt-athena.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.