Production Degradation - Import Latency
RESOLVED
Root Cause Analysis: On Monday, July 20, PackNet Software Platform experienced an unexpected increase in import volume following the receipt of a large batch of work through a customer file drop. That work went through an import pipeline that checked for duplicates using the Audit system. This elevated processing demand caused delays within the Audit service, which is used as part of the import workflow. To restore normal processing, Packsize temporarily limited a portion of incoming workload while the backlog was cleared. Packsize is taking the following actions to strengthen the resiliency of PackNet Software Platform and reduce the likelihood of a similar incident: load testing to validate system performance under high-volume conditions, retry behavior to better handle delays and timeouts, import workflow resiliency to reduce dependency on the Audit service, and request management to prevent stalled or no-longer-needed requests from continuing to consume system resources.
RESOLVED
The PackNet issue is now resolved.
MONITORING
PackNet services are recovering.
MONITORING
A fix has been implemented and we are monitoring the results.
IDENTIFIED
The issue has been identified and we are working on a fix.
INVESTIGATING
We have received reports that PackNet is slow to process jobs and are investigating the issue.