Production Degradation - Delays For Large Batches

Started
Lasted 48min
Degraded performance

Affected components

  1. Machine Production (Prod 1)

Updates

RESOLVED

lasted 48min

The code rollback is completed. Production has returned to normal.

IDENTIFIED

13min earlier

A recent code change has been identified as the source of the production latency. A rollback has been initiated.

INVESTIGATING

20min earlier

Software team received reports of production delays. We are investigating the issue.

INVESTIGATING

28min earlier

A code change was deployed that was later identified as the root cause of the issue. This deploy introduced latency to production that disproportionately impacted customers that produce in large batches.