Free lesson

Structured logs without secrets

Structured logs without secrets — Run AI workflows reliably. A stable event schema; An allowlist of fields; Correlation without personal data. Practical assignment: Structured logs without secrets.

Demo edition. The materials are undergoing editorial review.

Topics covered

  • A stable event schema
  • An allowlist of fields
  • Correlation without personal data

A stable event schema

JSON logs become useful when field names and types stay consistent. If one worker writes duration, another elapsedSeconds and a third says “slow,” aggregation becomes unreliable. Define schemaVersion, UTC timestamp, level, event, jobId, attempt, durationMs and errorCode. A successful step uses event=artifact.persisted rather than an arbitrary sentence. The snippet is our teaching schema, not OTLP export format; durationMs=240 is fictional. Validate before saving: durationMs is nonnegative, timestamp is valid, and job events include jobId. Do not change a field’s meaning without migrating consumers. A shared duration field with inconsistent units can create a false thousandfold slowdown. An explicit Ms suffix makes unit mistakes easier to spot. Do not add the model’s entire payload to this operational event.

{"schemaVersion":1,"timestamp":"2026-11-18T10:00:00Z","level":"info","event":"artifact.persisted","jobId":"job-7","attempt":1,"durationMs":240,"errorCode":null}

An allowlist of fields

Avoid leaks by not copying an entire error object. Select fields the log actually needs and drop unknown ones. A training error contains code=TIMEOUT, request.headers.Authorization=TRAINING_SECRET and a body with meeting text. Keep code and operationName, not the header or body. Regex masking is a useful extra layer but may miss a new token shape or nested array. An allowlist constrains data before storage. Check canaries inside nested fields, URL parameters and exception messages, never using a real secret. Set log-reading permissions, retention and disk-full behavior: unlimited debug logs can stop the workflow. Old-log removal should follow an explicit policy rather than improvised incident cleanup. Ensure the safe error code remains useful after unnecessary data is removed.

Correlation without personal data

A stable ID is enough to find all job-7 events; a person’s name or meeting text is unnecessary. Carry jobId through every stage and include traceId when tracing is configured. Preserve those IDs as queue metadata rather than generating a new jobId for retries. An ID does not authorize artifact reading: access to private summary.json needs separate enforcement. Do not use jobId as a metric label. Every job adds a new value, rapidly multiplying time series. Metrics can use stage, status and service; locate individual job IDs in logs or traces. This division supports aggregate counters without losing investigation of one run. Test with two concurrent training jobs to ensure their events remain distinct without requiring personal data to tell them apart.

Practical assignment

Structured logs without secrets

Continue to learning

Course outline

  1. What “the workflow works” means
  2. Structured logs without secrets
  3. Tracing and latency without misleading arithmetic
  4. Retries, idempotency and cost boundaries
  5. Signals that lead to a concrete action
  6. Checkpoints, backups and restoration evidence