ServConf Workshop: Observability and Incident Response

ServConf Workshop: Observability and Incident Response

Overview

This workshop teaches teams how to design observable systems and run effective incident response. It covers telemetry strategies, signal prioritization, alerting practices, post-incident workflows, and hands-on exercises using common open-source tools.

Objectives

• Understand the three pillars of observability: metrics, logs, and traces.
• Design telemetry that supports rapid troubleshooting and SLO-driven operations.
• Implement effective alerting and on-call practices that reduce noise and pager fatigue.
• Run structured incident response and post-incident reviews that lead to durable fixes.
• Practice these skills in guided, realistic exercises.

Target audience

• Site Reliability Engineers (SREs) and on-call engineers
• Backend and platform engineers responsible for production services
• Engineering managers seeking better operational practices

Workshop agenda (4 hours)

1. Introduction & goals (15 min) — scope, success criteria, and baseline surveys.
2. Observability fundamentals (30 min) — metrics, logs, traces, context propagation, and instrumentation patterns.
3. Telemetry design patterns (30 min) — cardinality control, semantic conventions, labels/tags strategy, and cost trade-offs.
4. Break (10 min)
5. Alerting & SLOs (40 min) — defining useful alerts, alert routing, burn rates, error budgets, and reducing noisy alerts.
6. Incident response playbooks (30 min) — incident command, runbooks, comms, severity levels, and escalation paths.
7. Hands-on exercise: Simulated outage (50 min) — teams diagnose and mitigate a realistic failure using supplied dashboards, traces, and logs.
8. Post-incident review & action planning (20 min) — blameless RCA, corrective actions, and tracking.
9. Wrap-up & resources (5 min)

Key practices taught

• Instrument for action: Emit the minimal set of high-signal metrics and structured logs that answer operational questions.
• Correlate across signals: Use trace IDs and consistent labeling so logs, traces, and metrics link for faster root cause analysis.
• SLO-driven alerting: Alert on SLO breaches or burn-rate changes rather than low-level symptoms to focus on user impact.
• Tiered alerts: Separate pages from notifications; page only high-severity incidents with clear remediation steps.
• Runbooks and playbooks: Maintain concise runbooks with command snippets and mitigation steps for frequent failure modes.
• Blameless postmortems: Capture timeline, contributing factors, and measurable action items; track until verified.

Tools and tech covered

• Observability: Prometheus/OpenTelemetry, Grafana, Jaeger/Tempo, Loki/Elastic.
• Alerting: Alertmanager, PagerDuty, Opsgenie, or native platform alerting.
• Incident management: Status pages, runbook tooling, postmortem templates (Markdown).
• Optional demos: OpenTelemetry SDK examples (tracing and metrics), alert rule examples, and a sample Dockerized app for exercises.

Deliverables for attendees

• A checklist for telemetry readiness and instrumentation.
• Sample alerting rules and SLO templates.
• A starter incident response playbook and postmortem template.
• Access to workshop exercise artifacts and slides.

Expected outcomes

• Faster mean time to detection and recovery (MTTD/MTTR) through better signal design.
• Reduced alert noise and clearer on-call responsibilities.
• Improved learning from incidents with actionable, tracked remediation items.

Follow-up recommendations

• Run a quarterly simulated outage for teams to practice.
• Adopt SLOs for critical user journeys and iterate alert thresholds with historical data.
• Invest in consistent trace and log correlation across services.
• Track implementation of postmortem action items and review their effectiveness after 30–90 days.

For hands-on materials (sample configs, runbooks, and exercise repo), tell me your preferred stack (e.g., Prometheus+Grafana+Jaeger) and I’ll provide tailored artifacts.