GPU Runtime Incident Response Checklist

Documentation Index

Fetch the complete documentation index at: https://docs.impulselabs.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Severity Definitions

​

Phase 1: Containment

• Acknowledge the alert in PagerDuty and mark yourself as responder.
• Open an incident ticket in Linear with severity, symptoms, and time of detection.
• Join the incident channel (#incident-gpu-runtime in Slack) and post the Linear ticket link.
• Assess scope — how many jobs are affected? Which region(s)?
  # Check failure rate
  gcloud ai custom-jobs list \
    --project=impulse-gpu-runtime \
    --region=us-central1 \
    --filter="state=JOB_STATE_FAILED AND updateTime>$(date -d '15 minutes ago' -Iseconds)" \
    --format="value(name)" | wc -l
  
• Pause job intake if failure rate > 25 % or budget is exhausted:
  kubectl set env deployment/gpu-scheduler -n gpu-runtime PAUSE_JOB_INTAKE=true
  
• Notify customers (via status page) if P1/P2 with customer-visible impact. CL owns this step.
• Verify containment — confirm no new failures are being created.

​

Phase 2: Triage

• Check scheduler pod health:
  kubectl get pods -n gpu-runtime -l app=gpu-scheduler
  kubectl logs -n gpu-runtime -l app=gpu-scheduler --tail=100
  
• Check Pub/Sub subscription backlog:
  gcloud pubsub subscriptions describe gpu-job-request-sub \
    --project=impulse-gpu-runtime \
    --format="yaml(numUndeliveredMessages,oldestUnackedMessageAge)"
  
• Check DLQ message count:
  gcloud pubsub subscriptions describe gpu-job-request-dlq-sub \
    --project=impulse-gpu-runtime \
    --format="yaml(numUndeliveredMessages)"
  
• Check Vertex AI quota utilization:
  gcloud compute regions describe us-central1 \
    --project=impulse-gpu-runtime \
    --format="yaml(quotas)" | grep -A3 "NVIDIA_T4_GPUS"
  
• Check GPU cost and budget status:
  # Review current vs. monthly budget in GCP Console → Billing
  
• Check for GCP incidents at status.cloud.google.com (Vertex AI, Pub/Sub).
• Determine failure category and select the appropriate runbook:
  • Scheduler crash → Failure Recovery Runbook § 1
  • Quota exhaustion → Quota Exhaustion Runbook
  • DLQ backlog → DLQ Recovery Runbook
  • Cost anomaly → Cost Anomaly Runbook
  • Artifact upload failure → Failure Recovery Runbook § 3
  • Preemption storm → Failure Recovery Runbook § 4
  • Pub/Sub consumer failure → Failure Recovery Runbook § 5
• Post triage summary to incident channel with root cause hypothesis and recovery plan.

​

Phase 3: Service Recovery

• Execute the relevant runbook from the triage phase above.
• Validate scheduler restart (if applicable):
  kubectl rollout status deployment/gpu-scheduler -n gpu-runtime
  kubectl exec -n gpu-runtime deploy/gpu-scheduler -- curl -s http://localhost:8080/healthz
  
• Resume job intake once the root cause is resolved and health checks pass:
  kubectl set env deployment/gpu-scheduler -n gpu-runtime PAUSE_JOB_INTAKE=false
  
• Monitor job submission success rate for 10 minutes after resuming:
  watch -n30 "gcloud ai custom-jobs list \
    --project=impulse-gpu-runtime --region=us-central1 \
    --filter='state=JOB_STATE_FAILED AND updateTime>$(date -d \"5 minutes ago\" -Iseconds)' \
    --format='value(name)' | wc -l"
  
• Replay DLQ messages once the scheduler is healthy. See DLQ Recovery Runbook.
• Update status page to “Monitoring” and then “Resolved” as service recovers.
• Notify customers of resolution (CL step).

​

Phase 4: Artifact Validation

• Identify jobs that ran during the incident window:
  gcloud ai custom-jobs list \
    --project=impulse-gpu-runtime \
    --region=us-central1 \
    --filter="updateTime>INCIDENT_START AND updateTime<INCIDENT_END" \
    --format="table(displayName,state,createTime,updateTime)"
  
• Check for jobs in terminal states without GCS artifacts:
  # For each SUCCEEDED job, verify artifacts exist
  gsutil ls gs://impulse-gpu-artifacts/<session_id>/<job_id>/
  
• Identify jobs that completed successfully but lack artifacts (potential upload failures):
  curl -s "https://api.impulselabs.ai/internal/gpu/jobs?status=SUCCEEDED&artifact_missing=true&since=INCIDENT_START" \
    -H "Authorization: Bearer $IMPULSE_SERVICE_TOKEN" | jq length
  
• Trigger artifact re-upload for any jobs with missing artifacts:
  curl -s "https://api.impulselabs.ai/internal/gpu/jobs?status=SUCCEEDED&artifact_missing=true" \
    -H "Authorization: Bearer $IMPULSE_SERVICE_TOKEN" \
    | jq -r '.[].job_id' \
    | while read -r job_id; do
        curl -s -X POST "https://api.impulselabs.ai/internal/gpu/jobs/$job_id/retry-upload" \
          -H "Authorization: Bearer $IMPULSE_SERVICE_TOKEN"
      done
  
• Confirm artifact counts match expected volumes (jobs × output files per job).
• Document any permanently lost artifacts in the incident ticket.

​

Phase 5: Postmortem Requirements

​

Postmortem must be completed within:

• P1: 48 hours
• P2: 5 business days

​

Required sections

• Incident summary — brief description, duration, severity, customer impact
• Timeline — minute-by-minute log from first alert to resolution (use incident ticket history)
• Root cause analysis — technical root cause (not “human error”)
• Contributing factors — monitoring gaps, deployment process issues, etc.
• Impact quantification — number of affected jobs, customers, estimated cost, data loss (if any)
• Action items — specific, owner-assigned, time-bound remediation tasks:
  • At least one action item to detect the issue faster
  • At least one action item to reduce impact
  • At least one action item to prevent recurrence
• Postmortem reviewed by IC, CL, and Engineering Lead
• Postmortem linked in the Linear incident ticket

​

Postmortem template location

​

Quick Reference

​

Key Contacts