A cache is generally applied to functions and methods which are either slow or expensive to execute, in order to minimize both caller latency and stress on underlying services.
As it stands today, calling a cachetools cached function multiple times from separate threads with the same key may cause the function body to be evaluated multiple times. This means that a cached, 10 seconds reference data load may be invoked thread count number of times during the first 10 seconds that it's executing, potentially swamping underlying services.
Cachetools today:
For example, setting up a REST (I used FastAPI) server to call the following function per request yields multiple calls even though the function is cached. (Note that each timestamped line represents a call to the FastAPI endpoint)
This is because @ cached only locks on the access to the cache, not on the generation of the value when the key is not present. During the time it takes from the first call for that key to that call (or a subsequent) call completing, the wrapped function will always be evaluated.
cache = TTLCache(maxsize=1024, ttl=600)
@cached(cache)
def test(self):
print("Function body called")
time.sleep(10)
> 2021-09-29 13:29:42,240 [.....
> Function body called
> 2021-09-29 13:29:44,137 [.....
> Function body called
> 2021-09-29 13:29:45,474 [.....
> Function body called
> 2021-09-29 13:29:46,974 [.....
> Function body called
> 2021-09-29 13:29:48,527 [.....
> Function body called
> 2021-09-29 13:29:50,242 [.....
> Function body called
> 2021-09-29 13:29:51,895 [.....
> Function body called
> 2021-09-29 13:29:51,895 [.....
> 2021-09-29 13:29:53.543 [.....
> 2021-09-29 13:29:57.213 [.....
> 2021-09-29 13:29:59.753 [.....
Another, more self contained example is as follows:
from cachetools import TTLCache
from cachetools.decorators import cached
from time import sleep
from concurrent.futures import ThreadPoolExecutor
cache = TTLCache(maxsize=100,ttl=600)
calls=0
@cached(cache)
def method(*args):
global calls
sleep(1)
calls+=1
print("Doing something expensive!")
return args
with ThreadPoolExecutor(max_workers=5) as executor:
executor.map(method, ['arg']*10)
print(calls)
> Doing something expensive!
> Doing something expensive!
> Doing something expensive!Doing something expensive!
> Doing something expensive!
> 5
Cachetools post-fix
After the fixes which I'm proposing, the expensive underlying function is only executed a single time for each unique (per key) call.
For the first example:
cache = TTLCache(maxsize=1024, ttl=600)
@cached(cache)
def test(self):
print("Function body called")
time.sleep(10)
> 2021-09-29 13:59:17,391 [...
> Function body called
> 2021-09-29 13:59:17,996 [.... subsequent calls to the API
> 2021-09-29 13:59:21,140 [.... subsequent calls to the API
> 2021-09-29 13:59:22,758 [.... subsequent calls to the API
> 2021-09-29 13:59:24,222 [.... subsequent calls to the API
> 2021-09-29 13:59:25,740 [.... subsequent calls to the API
> 2021-09-29 13:59:27,289 [.... Original call unblocks
> 2021-09-29 13:59:27,290 [.... All subsequent calls unblock once call 1 finishes
> 2021-09-29 13:59:27,292 [.... All subsequent calls unblock once call 1 finishes
> 2021-09-29 13:59:27,293 [.... All subsequent calls unblock once call 1 finishes
> 2021-09-29 13:59:27,293 [.... All subsequent calls unblock once call 1 finishes
> 2021-09-29 13:59:27,294 [.... All subsequent calls unblock once call 1 finishes
I have manually added some commentary to the log lines. Note how the first call hits our expensive function, while subsequent calls wait for it to complete.
10 seconds after the first call has come in, all other calls instantly return, since the value is now available.
The request at 13:59:25 took only two seconds to respond, whereas it would not only have taken 10 seconds to respond before the bug fix, it would also add more stress to the underlying services called from within test()
In this second, self contained example, note how only one call is logged to the cached function, even though the code is functionally identical to before.
from cachetools import TTLCache # Still using cachetools TTLCache
from cachetools_fixed.decorators import cached # Fixed @ cached decorator
from time import sleep
from concurrent.futures import ThreadPoolExecutor
cache = TTLCache(maxsize=100,ttl=600)
calls=0
@cached(cache)
def method(*args):
global calls
sleep(1)
calls+=1
print("Doing something expensive!")
return args
with ThreadPoolExecutor(max_workers=5) as executor:
executor.map(method, ['arg']*10)
print(calls)
> Doing something expensive!
> 1
I'll also add that key level locking still works as expected - repeated calls with different keys yields no benefit over the previous implementation before this bug fix.
846 Nov 26, 2022
1.7k Jan 5, 2023
2 Dec 22, 2021
2 Aug 14, 2022
11 Feb 3, 2022
364 Jan 8, 2023
1.8k Jan 6, 2023
102 Jan 6, 2023
34 Oct 27, 2022
2 Nov 5, 2021
0 Feb 6, 2022
7 Oct 21, 2022
10 May 2, 2022
326 Dec 15, 2022
684 Jan 6, 2023
1 Nov 2, 2021
6.4k Jan 6, 2023
338 Dec 16, 2022
42 Nov 3, 2022
44 Nov 29, 2022