HF models are not using key-value caching? about tf-transformers HOT 2 CLOSED

@abhi-mosaic - key- value caching has been enabled by default in HuggingFace .generate function. We don't have to take care of it from our end.

forward pass does not require it while training, thats why you see a None value there. But inside generate it has been taken care of.
I already had a word with HF team 2 years back regarding this and was also a part of internal SLACK group. :)

from tf-transformers.

@legacyai Got it! I think I see it now, the use_cache=True is a bit hard to track down but it looks like it sits in the default GPT2 config, and then your benchmark pulls a pre-trained model (config here) which doesn't specify use_cache so it goes with the default.

Thank you for clarifying, I just wanted to make sure the cacheing was on :)

from tf-transformers.