Caching Graph Adapter
You can use CachingGraphAdapter to cache certain nodes.
This is great for:
- Iterating during development, where you don't want to recompute certain expensive function calls.
- Providing some lightweight means to control recomputation in production, by controlling whether a “cached file” exists or not.
For iterating during development, the general process would be:
- Write your functions.
- Mark them with
tag(cache="SERIALIZATION_FORMAT") - Use the CachingGraphAdapter and pass that to the Driver to turn on caching for these functions. a. If at any point in your development you need to re-run a cached node, you can pass its name to the adapter in the
force_compute argument. Then, this node and its downstream nodes will be computed instead of loaded from cache. - When no longer required, you can just skip (3) and any caching behavior will be skipped.