Activity

Production Degradation - Import Latency

Started
Lasted 2h 8min
MAJORaffected
Import & Job Creation (Prod 1)User Interface (Prod 1)

RESOLVED

lasted 2h 8min

Root Cause Analysis: On Monday, July 20, PackNet Software Platform experienced an unexpected increase in import volume following the receipt of a large batch of work through a customer file drop. That work went through an import pipeline that checked for duplicates using the Audit system. This elevated processing demand caused delays within the Audit service, which is used as part of the import workflow. To restore normal processing, Packsize temporarily limited a portion of incoming workload while the backlog was cleared. Packsize is taking the following actions to strengthen the resiliency of PackNet Software Platform and reduce the likelihood of a similar incident: load testing to validate system performance under high-volume conditions, retry behavior to better handle delays and timeouts, import workflow resiliency to reduce dependency on the Audit service, and request management to prevent stalled or no-longer-needed requests from continuing to consume system resources.

RESOLVED

3d 4h earlier

The PackNet issue is now resolved.

MONITORING

12min earlier

PackNet services are recovering.

MONITORING

7min earlier

A fix has been implemented and we are monitoring the results.

IDENTIFIED

7min earlier

The issue has been identified and we are working on a fix.

INVESTIGATING

1h 40min earlier

We have received reports that PackNet is slow to process jobs and are investigating the issue.