Skip to content
LACE
  • v0.1 Current
  • Python
  • TypeScript Soon
GitHub Book a demo

API & Platform

Observability — logs, traces & metrics

Structured logs, OpenTelemetry traces, and per-call model attribution for every AI operation — not as an add-on, but as a property of the control plane.

Logs

Every service emits JSON structured logs (via structlog) to stdout. In local dev, read them with docker compose logs -f api or lace-app logs <app_id> --lines 500 for a live sidecar. In cloud, logs ship to CloudWatch (or your configured sink) with tenant, request, and trace ids attached.

Traces — OpenTelemetry

The API and workers are instrumented with OpenTelemetry. Each pipeline step, retrieval leg, and agent turn creates a span with:

  • tenant_id, request_id, trace_id
  • model_alias and token counts for every LLM call (see governance)
  • Step / tool id, input hash, latency, and error code

In dev, bring the full stack with make dev-observability (adds Tempo + Prometheus + Grafana). In cloud, traces flow to your OTEL collector.

Per-call model attribution

Every AI operation is attributed: which model, which step/tool/agent, which tenant, how many tokens, and what it cost. This is what powers the cost dashboards and budget caps — not an estimate, the actual accounting record for that call.

Pipeline replay & traces

Pipelines persist a trace per run (src/lace/pipeline/trace.py & trace_report.py). You can inspect a run's step graph, retry history, and dead-letter reasons from the API (GET /v1/pipelines/{id}/runs/{run_id}) or from Workflow Studio.

Agent sessions

Agent sessions are durable (Postgres-backed). Inspect a session with GET /v1/agents/{agent_id}/sessions/{session_id} — turns, tool calls, approval state, and the full prompt trace. Steering (POST /v1/agents/.../steer) and cancellation (POST .../cancel) are control-plane operations, not client-side conventions.

Sidecars & apps

  • GET /health/ready on every sidecar — what lace-app status checks.
  • GET /v1/apps/{app_id} + GET /v1/apps/{app_id}/releases — release history and health.
  • Platform metrics: pipeline success rate, retrieval latency, reranker GPU queue depth, ingest lag — exposed via Prometheus in dev, CloudWatch in cloud.

Cost dashboards

Token and cost accounting is per tenant, per model, per day. Admins see it in the console; operators can pull it from GET /v1/billing/usage. Budgets are enforced — a hard cap stops dispatch before the overrun, not after.

Next: governance & security or publishing & releases.