Production Degradation - Import Latency

Started
Lasted 2h 7min
Degraded performance

Affected components

  1. User Interface (Prod 1)
  2. Import & Job Creation (Prod 1)

Updates

RESOLVED

lasted 2h 7min

The PackNet issue is now resolved.

RESOLVED

51s earlier

Root Cause Analysis: On Monday, July 20, PackNet Software Platform experienced an unexpected increase in import volume following the receipt of a large batch of work through a customer file drop. That work went through an import pipeline that checked for duplicates using the Audit system. This elevated processing demand caused delays within the Audit service, which is used as part of the import workflow. To restore normal processing, Packsize temporarily limited a portion of incoming workload while the backlog was cleared. Packsize is taking the following actions to strengthen the resiliency of PackNet Software Platform and reduce the likelihood of a similar incident: load testing to validate system performance under high-volume conditions, retry behavior to better handle delays and timeouts, import workflow resiliency to reduce dependency on the Audit service, and request management to prevent stalled or no-longer-needed requests from continuing to consume system resources.

MONITORING

12min earlier

PackNet services are recovering.

MONITORING

20min earlier

A fix has been implemented and we are monitoring the results.

IDENTIFIED

27min earlier

The issue has been identified and we are working on a fix.

INVESTIGATING

1h 40min earlier

We have received reports that PackNet is slow to process jobs and are investigating the issue.