Caching Strategy about anlessini HOT 1 OPEN

I've thought about caching-as-a-service, but it breaks the scale-to-zero principle which is important for low volume application like ours. When there are zero queries, I want us to incur zero costs.

And yes, correct, container warmup is a pain.

I think the next step is a smarter caching policy - the current implementation is to suck everything in memory, but we can be much smarter... one obvious idea is to implement the equivalent of paging blocks of memory in out...

Not sure if this is the lowest hanging fruit though.

Loading MS MARCO passage in memory is okay-ish, I think - IIRC, lambda warmup is about a minute from cold start. MS MARCO doc latency, I agree, would be unacceptable...

from anlessini.