howdy/cache

An application cache: keep the result of expensive work for a while, and throw it away when the data it came from changes.

import howdy/cache
import howdy/cache/memory

// In main, once:
let notes_cache = cache.new(memory.new(max_entries: 50_000))

// Anywhere:
pub fn find(id: Int) -> service.Result(Note) {
  let entry =
    cache.entry("note:" <> int.to_string(id), ttl: 300)
    |> cache.tags(["notes", "note:" <> int.to_string(id)])
  use <- cache.fetch(notes_cache, entry, note.codec())
  notes.find(repo, id)
}

// When a note changes:
cache.invalidate(notes_cache, ["note:" <> int.to_string(id)])

fetch returns the stored value when there is one, and otherwise runs the function, stores an Ok result and returns it. Errors are never stored, so a failed database read is tried again next time.

Values are kept as JSON through a howdy/codec codec, in every store, so a codec that does not round-trip fails in development the same way it would in production. A stored value that no longer decodes, say because a deploy changed its type, counts as missing and is replaced.

Invalidation

Give each entry the tags of the data it was built from, then call invalidate with a tag when that data changes: every entry carrying the tag is treated as missing from then on. Tags have version numbers. fetch reads its tags’ versions before it computes, and stores them with the value; a value is only served while every one of its tags still has the version it was stored with. A slow computation that read the data before a change therefore cannot store an old value as current after the change was invalidated.

Inside a howdy/database transaction, invalidate and delete wait until the outermost transaction has ended, so nobody can cache the old data again between the invalidation and the commit. put deletes the key then instead of storing, since its value might roll back. fetch inside a transaction neither reads nor writes the cache: it would see the old value after the transaction’s own writes, or store data that might roll back.

Failures

The cache never takes the application down with it. If the store fails, fetch logs a warning and computes the value as if the cache were empty. invalidate, delete, put and clear return the error, since a failed invalidation can leave stale data behind.

Many requests at once

When several requests on one node miss the same key together, one computes and the rest wait for its result, or its error, instead of all running the same query. Each node computes separately. A computation that is itself filling an entry never waits for another, so two entries built from each other cannot wait on each other forever. One that hands work to another process and waits for it still can, if that process fetches the entry being filled.

Types

Cache

opaque </>

A cache over a store. Build one at startup with new and share it.

pub opaque type Cache

Entry

opaque </>

Where to keep a value and for how long.

pub opaque type Entry

Store

opaque </>

Where entries and tag versions are kept. howdy/cache/memory and howdy/cache/database provide stores; build your own with store.

pub opaque type Store

Stored

</>

An entry as a store returns it. expires_at is Unix time in milliseconds. current is whether every tag the entry was written with still has the version it was written with.

pub type Stored {
  Stored(value: String, expires_at: Int, current: Bool)
}

Constructors

Stored(value: String, expires_at: Int, current: Bool)

Values

clear

</>

pub fn clear(cache: Cache) -> Result(Nil, service.Error)

Remove every entry in the store, in every namespace. Tag versions are kept.

default_max_value_bytes

</>

pub const default_max_value_bytes: Int

Values larger than this, in bytes of JSON, are returned but not stored, unless max_value_bytes says otherwise: 1 MiB.

delete

</>

pub fn delete(
  cache: Cache,
  key: String,
) -> Result(Nil, service.Error)

Remove the entry under key. Inside a transaction this happens once the transaction has ended.

Unlike invalidate, this cannot stop a fetch that read the old data just before the change from storing it just after the delete. Tag entries whose data changes, and invalidate the tag.

entry

</>

pub fn entry(key: String, ttl seconds: Int) -> Entry

Keep the value under key for ttl seconds. A ttl of zero or less means the value is computed every time and never stored.

entry_ms

</>

pub fn entry_ms(key: String, ttl milliseconds: Int) -> Entry

Like entry, with the time to keep it in milliseconds.

fetch

</>

pub fn fetch(
  cache: Cache,
  entry: Entry,
  codec: codec.Codec(a),
  compute: fn() -> Result(a, service.Error),
) -> Result(a, service.Error)

The value under the entry’s key if there is a current one, otherwise the result of compute, stored when it is Ok.

use <- cache.fetch(app_cache, cache.entry("stats", ttl: 60), stats.codec())
stats.compute(db)

get

</>

pub fn get(
  cache: Cache,
  key: String,
  codec: codec.Codec(a),
) -> option.Option(a)

The current value under key, if there is one. A store that fails is logged and reads as None. Like fetch, reads nothing inside a transaction.

invalidate

</>

pub fn invalidate(
  cache: Cache,
  tags: List(String),
) -> Result(Nil, service.Error)

Treat every entry carrying any of tags as missing from now on. Inside a transaction this happens once the transaction has ended, whether it committed or not.

max_value_bytes

</>

pub fn max_value_bytes(cache: Cache, bytes: Int) -> Cache

Return but do not store values whose JSON is larger than bytes.

namespace

</>

pub fn namespace(cache: Cache, namespace: String) -> Cache

Keep this cache’s keys and tags apart from other caches sharing the store. Changing the namespace, say from "v1" to "v2", also starts the cache afresh, without deleting anything.

new

</>

pub fn new(store: Store) -> Cache

A cache over store. Its single-flight table belongs to the calling process, like a memory store’s tables, so create it in main.

put

</>

pub fn put(
  cache: Cache,
  entry: Entry,
  codec: codec.Codec(a),
  value: a,
) -> Result(Nil, service.Error)

Store value under the entry’s key, replacing what is there. The tags take their current versions when the value is stored, so only put a value that is current then; fetch is safer.

Inside a transaction the value is not stored: once the transaction has ended the key is deleted instead, and the next fetch computes it. The value might come from changes the transaction, or a savepoint within it, rolls back, and the cache cannot tell.

store

</>

pub fn store(
  named name: String,
  read read: fn(String) -> Result(
    option.Option(Stored),
    service.Error,
  ),
  versions versions: fn(List(String)) -> Result(
    List(#(String, Int)),
    service.Error,
  ),
  write write: fn(String, String, Int, List(#(String, Int))) -> Result(
    Nil,
    service.Error,
  ),
  delete delete: fn(String) -> Result(Nil, service.Error),
  invalidate invalidate: fn(List(String)) -> Result(
    Nil,
    service.Error,
  ),
  clear clear: fn() -> Result(Nil, service.Error),
) -> Store

A store from its operations. Keys and tags arrive already namespaced and at most 250 bytes long; values are JSON text.

read(key): the entry under key, expired or not, or None.
versions(tags): the current version of each tag, 0 for a tag that was never invalidated.
write(key, value, expires_at, versions): replace the entry under key, remembering the tag versions it was computed with, unless the stored entry was written with a newer version of a tag both share. Such a value was computed before an invalidation the stored one already reflects. A store that replaces it anyway is still correct, since the value is never served, but loses the newer entry.
delete(key): remove the entry under key, if there is one.
invalidate(tags): add one to each tag’s version, atomically per tag. The store may also remove the entries carrying them.
clear(): remove every entry. Tag versions must be kept, or an invalidation could be undone.

Work deferred to the end of a transaction calls these while howdy/database still counts the transaction as open. A store on the app’s database should use howdy/database functions, which work then, rather than open a pool transaction of its own.

store_name

</>

pub fn store_name(store: Store) -> String

The name the store was built with, such as "memory".

with_clock

</>

pub fn with_clock(cache: Cache, clock: fn() -> Int) -> Cache

Read the time, in Unix milliseconds, from clock instead of the system clock, so tests can expire entries without sleeping. Stores that sweep expired entries in the background still use the system clock for that.