Docs

Store AI trace history with SlateDB

In this example, we are going to store AI traces in a customer's cloud.

The familiar answer is to deploy a database such as ClickHouse. That is a good answer when you need fast aggregations across billions of events. But software you put in a customer's environment should be as small and boring as possible. Every database server becomes another thing to size, upgrade, monitor, back up, and explain during a security review.

SlateDB gives us a smaller option. It is an embedded key-value database that stores its durable files in object storage. Your application opens it as a library; there is no separate database server to deploy. One process writes, while any number of API replicas can read.

Here, we use SlateDB to save traces, fetch one by ID, and browse by agent, status, model, and time.

For writes, an AI application sends a trace to replicated API containers, which stage the body and enqueue a pointer. One writer commits the trace and indexes to SlateDB on object storage. For reads, replicated API readers query SlateDB directly.

The API scales from two to four replicas. The writer stays at one replica because SlateDB accepts writes through a single writer. S3, Google Cloud Storage, or Azure Blob Storage holds everything durable.

Why object storage is a useful foundation

Object storage is already one of the easiest resources for a customer to own. It is durable across zones, inexpensive, encrypted by default, and does not have database nodes that need patching. Separating it from the containers also means you can replace every running process without replacing the data.

Step 1: declare the four pieces

Start with Storage and Queue. They are frozen resources, so customer setup owns them and an ordinary application release cannot replace them.

alien.ts
const data = new alien.Storage("data")
  .lifecycleRules([{ prefix: "staging/v1/", days: 7 }])
  .build()

const ingestion = new alien.Queue("ingestion").build()

The API is ordinary stateless compute, so it can have several replicas. The writer is explicitly fixed at one.

alien.ts
const api = new alien.Container("api")
  .code({
    type: "source",
    src: ".",
    toolchain: { type: "rust", binaryName: "slatedb-trace-store" },
  })
  .autoScale({ min: 2, desired: 2, max: 4, targetHttpInFlightPerReplica: 100 })
  .publicEndpoint("api", 8080, "http")
  .link(data)
  .link(ingestion)
  .build()

const writer = new alien.Container("writer")
  .code({
    type: "source",
    src: ".",
    toolchain: { type: "rust", binaryName: "slatedb-trace-store" },
  })
  .replicas(1)
  .link(data)
  .link(ingestion)
  .build()

Both containers are live. Alien can ship new code and replace them without taking ownership of the customer's stored traces.

Step 2: accept a trace quickly

A trace can be larger than the portable 64 KiB Queue limit. The API therefore writes the body to Storage first, then puts a small pointer on Queue:

src/service.rs
let encoded = serde_json::to_vec(&trace)?;
let content_hash = hex::encode(Sha256::digest(&encoded));
let staging_path = format!(
    "staging/v1/{}/{}.json",
    hex::encode(&trace.trace_id),
    content_hash,
);

storage
    .put(&Path::from(staging_path.as_str()), Bytes::from(encoded).into())
    .await?;

let pointer = IngestionPointer {
    trace_id: trace.trace_id.clone(),
    content_hash: content_hash.clone(),
    staging_path,
};
queue
    .send(MessagePayload::Json(serde_json::to_value(pointer)?))
    .await?;

The body is durable before the pointer is published. Keeping the queue message small also avoids copying a large trace through the queue service.

Here is a complete request:

curl -i http://localhost:8080/v1/traces \
  -H 'content-type: application/json' \
  -d '{
    "traceId": "run-01",
    "agent": "researcher",
    "status": "completed",
    "model": "claude-sonnet",
    "startedAt": "2026-08-26T18:00:00Z",
    "finishedAt": "2026-08-26T18:00:04Z",
    "payload": {"events": [{"type": "tool", "name": "search"}]}
  }'

202 Accepted has a precise meaning: the trace body and its queue pointer are durable. The background writer may not have made the trace queryable yet.

Step 3: commit once, even after redelivery

The writer receives the pointer, loads the staged body, and verifies its hash. Only then does it commit the trace to SlateDB.

The primary key is simple:

t/{encodedTraceId}

The list endpoint allows any combination of agent, status, and model. There are only eight combinations, so every trace gets eight small index entries ordered by startedAt.

i/{filterCombination}/{filterValues}/{startedAt}/{traceId}

That is the central tradeoff: a little more storage and write work in exchange for fast, predictable reads. We are choosing the questions up front instead of building a general-purpose analytics database.

Generating those eight entries is just a loop over the three filter bits:

src/keys.rs
const AGENT_BIT: u8 = 1;
const STATUS_BIT: u8 = 2;
const MODEL_BIT: u8 = 4;

pub fn index_keys(trace: &Trace) -> impl Iterator<Item = String> + '_ {
    (0..8).map(|mask| {
        let prefix = index_prefix(
            mask,
            (mask & AGENT_BIT != 0).then_some(trace.agent.as_str()),
            (mask & STATUS_BIT != 0).then_some(trace.status.as_str()),
            (mask & MODEL_BIT != 0).then_some(trace.model.as_str()),
        );
        format!(
            "{}{time:016x}/{}",
            prefix,
            hex::encode(&trace.trace_id),
            time = trace.started_at.timestamp_millis() as u64,
        )
    })
}

The primary value and every index entry go into one SlateDB transaction. await_durable waits until the commit is safe in object storage before we acknowledge the queue message.

src/store.rs
let transaction = db
    .begin(IsolationLevel::SerializableSnapshot)
    .await?;

if let Some(existing) = transaction.get(primary_key.as_bytes()).await? {
    let existing: StoredTrace = serde_json::from_slice(&existing)?;
    if existing.content_hash == content_hash {
        return Ok(CommitResult::AlreadyExists);
    }
    return Err(ApiError::conflict(&trace.trace_id).into());
}

let encoded_trace = serde_json::to_vec(&stored_trace)?;
transaction.put(primary_key.as_bytes(), &encoded_trace)?;
for index_key in index_keys(&trace) {
    transaction.put(index_key.as_bytes(), primary_key.as_bytes())?;
}

transaction
    .commit_with_options(&WriteOptions {
        await_durable: true,
        ..WriteOptions::default()
    })
    .await?;

After that returns, the writer acknowledges the queue message and deletes the staging object.

Queues can deliver a message more than once. The trace ID and content hash make that safe:

  • same ID and same content: already committed, acknowledge it;
  • same ID and different content: record a conflict and acknowledge it;
  • temporary Storage or SlateDB failure: release the message for another attempt.

Step 4: read from any API replica

API replicas open SlateDB in read-only mode. They poll for new database files once per second and cache frequently used blocks in memory.

src/store.rs
let db = DbReader::builder("db/v1", object_store)
    .with_options(DbReaderOptions {
        manifest_poll_interval: Duration::from_secs(1),
        ..DbReaderOptions::default()
    })
    .build()
    .await?;

// Direct lookup
let trace = db.get(primary_key(trace_id)).await?;

// Ordered, paginated listing
let prefix = query_prefix(&query);
let mut entries = db.scan_prefix(prefix.as_bytes()).await?;

query_prefix selects the index matching the supplied filters. The timestamp and trace ID at the end of each key keep results ordered and provide a stable pagination cursor.

# Fetch one trace
curl http://localhost:8080/v1/traces/run-01

# Browse a known index
curl 'http://localhost:8080/v1/traces?agent=researcher&status=completed&limit=25'

New writes are eventually visible, normally within the one-second poll interval. Already committed traces remain readable while the writer is restarting.

When this is—and is not—a good fit

This design works well when traces are append-heavy and the ways you read them are known ahead of time: look up an ID, filter on a few fields, and paginate through history.

It is not a good fit for arbitrary SQL, joins, large aggregations, full-text search, or many concurrent writers. If those become core product features, keep object storage as the durable archive and add ClickHouse, OpenSearch, or another query system on top. You do not need to migrate the original trace data out of the customer's storage.

What this example guarantees

PropertyBehavior
Durable acceptance202 means the staged body and queue pointer were written
Atomic commitThe trace and all indexes appear together
IdempotencyIdentical redeliveries do not create duplicate traces
Read freshnessReaders normally observe commits within about one second
Compute replacementAPI and writer containers can be replaced without moving data

Malformed messages, hash mismatches, and ID conflicts are recorded under failures/v1/ without copying the rejected payload. Successfully committed staging objects are deleted; the seven-day lifecycle rule cleans up anything left behind.

Run it locally

The example includes a deployed test, not only unit tests:

cd examples/slatedb-trace-store
alien dev

Then submit the sample trace, wait briefly, and fetch it with the commands above. To run the automated version of the same flow:

cargo nextest run -p slatedb-trace-store
pnpm test

Before exposing this in production, add authentication at the deployment boundary.

What Alien provides

At this point, you have built a small data plane for AI trace history: object storage holds the durable data, a queue absorbs writes, one SlateDB writer commits them, and replicated API containers serve reads.

Alien helps you deploy, monitor, and update that data plane inside each customer's AWS, Google Cloud, or Azure account. The same alien.ts creates the object storage, queue, API, and writer; keeps durable resources under customer setup ownership; and gives each container only the permissions it declares.

The application code uses the same Storage and Queue bindings on every cloud. You can run the complete topology locally with alien dev, then ship it to customers without maintaining separate bucket SDKs, queue integrations, IAM policies, and deployment templates for every provider.

Complete source: examples/slatedb-trace-store.

Next: Storage, Queue, and Frozen and live resources.

Complete multi-service application

See how a larger application maps services, databases, queues, and storage to a stack.

Build an object-storage-backed vector database

Run stateless Rust readers and writers over durable object storage.

On this page

Why object storage is a useful foundationStep 1: declare the four piecesStep 2: accept a trace quicklyStep 3: commit once, even after redeliveryStep 4: read from any API replicaWhen this is—and is not—a good fitWhat this example guaranteesRun it locallyWhat Alien provides