Skip to main content
This is the page to read before you ship keeps to production. It explains exactly what guarantees the SDK gives you (and which it deliberately doesn’t), and the single pattern that prevents silent data loss in serverless, queue workers, CLI jobs, and agentic steps. The short version: keeps never breaks your app, and delivery is best-effort. If your process can exit or be frozen quickly, you must call keeps.flush() before the unit of work returns.

​
The two design principles

​
Fail-open, always

No public keeps call ever raises into your code. capture(), context(), flush(), and shutdown() swallow their own errors. If the network is down, if the buffer is full, if the endpoint is misconfigured — your generation code runs exactly as it would without keeps installed. Observability is never allowed to take down the thing it observes.

​
Best-effort delivery, not at-least-once

keeps trades durability for staying out of your request path. There is no local disk queue, no write-ahead log, no acknowledgement protocol surfaced to your code. Events live in an in-memory buffer and are flushed in the background. If the process dies with events still buffered or in-flight, those events are gone. This is a deliberate tradeoff: keeps is a neutral observer, never a dependency your generations wait on. The cost is that you own the flush boundary in short-lived processes. The rest of this page is about getting that boundary right.

​
What happens when you call capture()

capture() is a non-blocking, O(1) append to an in-memory buffer. It mints (or reuses) the asset_id, builds the event envelope, appends it, and returns immediately — it never does network I/O on your thread. A background daemon thread does the actual sending:

Time-based flush

The daemon wakes every flush_interval seconds (default 1.0s) and sends whatever is buffered.

Size-based flush

When the buffer reaches 200 events (max_batch), the daemon is woken immediately rather than waiting for the next tick.

Bounded buffer

The buffer is capped at 10,000 events (max_buffer). On overflow the newest (incoming) event is dropped and dropped_overflow increments.

atexit flush

On a normal interpreter exit, an atexit handler flushes remaining events. Crashes, SIGKILL, and frozen workers skip this.

​
HTTP delivery and retries

When the daemon sends a batch:
  • It retries up to 3 times with exponential backoff (with jitter). Timeouts and 5xx responses count as transient; the server’s one deliberate transient signal is 503 (its storage is down) — everything else it has to say about your data comes back inside a 202.
  • It honors 429 responses, waiting for the Retry-After header before retrying.
  • It does not retry 4xx responses (other than 429). A 400 or 401 means the batch is malformed or unauthorized — retrying won’t help, so the batch is dropped, dropped_send increments, and a one-time warning is logged.
  • A 413 (batch over the server’s size cap) makes the transport bisect: it splits the batch in half and sends each half separately, recursively — so one oversized event costs only itself, never the batch it rode in.
  • A 202 response can still reject individual events (bad timestamp, over-cap field, …). The SDK parses the response body: confirmed events add to stats()['accepted'], per-event rejects add to stats()['rejected_by_server'] and log a one-time warning with the reason codes. These rejects are terminal — the server will never accept those bytes, so they are not retried.

​
The flush boundary: read this if your process exits fast

In any process that can exit or be frozen quickly, call keeps.flush() before the unit of work returns.The background daemon flushes on a ~1s tick, and atexit flushes on a clean exit — but neither is guaranteed to run in time when:
  • a serverless function returns and the platform freezes or recycles the container before the next daemon tick,
  • a queue/worker finishes a job and the process is killed or the worker is reaped,
  • a CLI job exits via os._exit, a signal, or a hard SIGKILL,
  • an agentic step / forked worker is torn down between steps.
In all of these, events that are still buffered (or mid-flight) are lost. A capture() that “succeeded” only means it made it into memory — not that it was delivered. flush() blocks the calling thread until the buffer is drained (sent or dropped) or it times out.

​
The correct pattern: flush in finally

Put the flush in a finally: so it runs whether the work succeeded or raised. flush() itself never raises, so it’s safe there. 
import keeps

keeps.init(api_key="kp_live_...", endpoint="https://ingest.yourco.com")

def handle_job(job):
    asset_id = None
    try:
        # First capture mints the asset_id; reuse it to correlate the rest.
        asset_id = keeps.capture(
            model="fal-ai/flux/dev",
            status="submitted",
            inputs={"prompt": job["prompt"], "num_images": 1},
        )

        result = call_provider(job)  # your fal/Replicate/OpenAI call

        keeps.capture(
            asset_id,
            status="completed",
            outputs=[{"url": result.url, "width": 1024, "height": 1024}],
        )
        return result
    except Exception:
        if asset_id is not None:
            keeps.capture(asset_id, status="failed", error=repr_provider_error())
        raise  # never swallow your own errors for the sake of telemetry
    finally:
        # The one line that prevents silent loss before this process is frozen.
        keeps.flush()
flush(timeout=5.0) returns True if the buffer drained and False if it timed out. In a latency-sensitive serverless handler you can tighten the timeout — but remember a False means some events may not have been delivered. 
if not keeps.flush(timeout=2.0):
    # Flush timed out — some events may not have reached the backend.
    # Best-effort: log it, don't fail the job over telemetry.
    ...
For a long-lived server (a web app, a daemon) you generally don’t need a per-request flush — the 1s daemon tick keeps up, and atexit catches the rest on a clean shutdown. The flush boundary matters specifically when the process is short-lived or can be killed/frozen out from under the daemon.

​
Detecting silent loss with stats()

Because delivery is best-effort and fail-open, the SDK won’t shout when it drops events — that’s the whole point. keeps.stats() is how you observe the observer. 
import keeps

s = keeps.stats()
# {
#   'initialized': True, 'disabled': False,
#   'emitted': 1287,           # capture() events that made it into the buffer
#   'sent': 1240,              # events delivered to the backend
#   'batches': 9, 'retries': 2,
#   'dropped_overflow': 0,     # buffer was full (max_buffer=10_000) at append time
#   'dropped_send': 0,         # batches dropped after retries / non-429 4xx
#   'accepted': 1238,          # events the server confirmed it stored (202 body)
#   'rejected_by_server': 2,   # per-event rejects from the 202 body — terminal
# }
What to watch for:
You’re producing events faster than the daemon can ship them, and the buffer was already at its 10,000-event cap when a new event arrived. Usual causes: a misconfigured or unreachable endpoint (nothing is draining), or a genuine burst. Check connectivity first; if the endpoint is healthy and it’s pure volume, raise max_buffer in init() or reduce capture volume.
Batches failed permanently — either repeated transient failures that exhausted the 3-attempt retry budget, or a non-429 4xx (commonly a bad API key → 401, or a malformed/too-large batch → 400). Verify your API key and endpoint.
Events reached the server but were rejected one-by-one in the 202 response (a bad_ts, an over-cap field, …). These are terminal — the server will never accept those bytes — so they are dropped, not retried. The first occurrence logs a warning listing the reason codes; fix the producer that builds those events. See the HTTP API reference for every reason code.
Some events are stuck in the buffer or never delivered. Right before a short-lived process exits, sent < emitted is exactly the symptom of a missing flush() at the work boundary. A persistent gap on a long-lived process points at delivery failures — cross-check dropped_send.
A cheap production health check: after a successful flush(), the events that made it into the buffer should be fully accounted for — that is, sent + dropped_send should equal emitted (minus anything still buffered or in-flight). Note dropped_overflow counts events that were rejected before they entered the buffer, so they are not part of emitted — track that counter on its own. A growing, unexplained gap means events are leaking. On the server side of the ledger, accepted + rejected_by_server accounts for what happened to sent: sent is what reached the endpoint, accepted is what it stored.

​
Configuration gotchas that look like “nothing is happening”

Both of these fail open and quiet by design — so the only signal is that no data shows up in your dashboard. Check them first when delivery seems broken.

​
The default endpoint never sends

Unconfigured endpoints are no longer a silent-drop hazard: with nothing set, the SDK delivers to the hosted ingest (https://ingest.keeps.dev). KEEPS_INGEST_URL/endpoint= exist to redirect delivery, not to enable it. You must point keeps at your ingest URL via KEEPS_INGEST_URL or the endpoint= argument: 
import keeps

# Either:
keeps.init(api_key="kp_live_...", endpoint="https://ingest.yourco.com")
# Or set KEEPS_INGEST_URL in the environment and call keeps.init(api_key=...).
init() auto-appends /v1/events if your endpoint lacks that path, so https://ingest.yourco.com and https://ingest.yourco.com/v1/events are equivalent.

​
A missing API key disables the SDK

If no api_key is provided (neither the api_key= argument nor the KEEPS_API_KEY environment variable) and the SDK isn’t explicitly disabled, keeps logs a warning and disables itself. Every subsequent capture() becomes a no-op (it still returns an asset_id, but nothing is buffered or sent). Check your logs for: 
keeps: no api_key provided (arg or KEEPS_API_KEY) — SDK disabled
You can also confirm at runtime: keeps.stats()['disabled'] is True when the SDK has disabled itself (or when you set KEEPS_DISABLED=1 / disabled=True on purpose).
Your project_id is derived server-side from the API key — the client never sends it and it is never a capture() argument. The key is the only thing that binds your events to your project, which is why a missing key disables delivery entirely.

​
Unit-of-work helper: keeps.session()

keeps.session() is the built-in form of the flush-in-finally pattern: it applies ambient tags like keeps.context() and guarantees a flush on exit. Reach for it at every boundary where the process may exit or be frozen fast — serverless handlers, queue jobs, CLI runs, agentic steps.
import keeps

def handler(event):
    with keeps.session(user_id=event["user"], worker="image-gen"):
        asset_id = keeps.capture(model="fal-ai/flux/dev", status="submitted",
                                inputs={"prompt": event["prompt"]})
        result = call_provider(event)
        keeps.capture(asset_id, status="completed",
                     outputs=[{"url": result.url}])
    # buffered events are flushed here, before the handler returns
session() is fail-open and does not swallow exceptions raised inside the block — it flushes first, then the exception propagates, so your error handling is unaffected. Pass flush_timeout= to bound the exit-flush (default 5s); keyword args beyond user_id/flush_timeout become ambient properties tags.

​
Quick reference

BehaviorValue / rule
capture()Non-blocking O(1) in-memory append; never raises; returns the asset_id
Time flushEvery flush_interval (default 1.0s)
Size flushAt 200 buffered events (max_batch)
Buffer capmax_buffer (default 10_000); overflow drops the newest (incoming) event
RetriesUp to 3x exponential backoff; honors 429 Retry-After; no retry on other 4xx; 413 bisects the batch
Server rejectsPer-event, in the 202 body → stats()['rejected_by_server'] + one-time warning; terminal, never retried
Clean exitatexit flush (skipped on crash / kill / freeze)
Loss countersstats()['dropped_overflow'], stats()['dropped_send'], stats()['rejected_by_server']
Delivery guaranteeBest-effort, not at-least-once
Your responsibilitykeeps.flush() before a short-lived process returns

Recording provider calls

Where the finally: keeps.flush() boundary fits into a full submit → complete capture flow.

API reference

Full signatures for init, capture, context, flush, shutdown, and stats.

Core concepts

Events, correlation by reused asset_id, and properties as group-by tags.

Privacy & safety

References-not-bytes, verbatim capture, and how project_id derives from your key.