The engineering team was able to isolate the cause of the problem.  A new version of a third party library recently introduced into the filtering process had begun consuming all available memory on some of the filtering servers, periodically causing the delivery process to stall.  This resulted in some messages being queued and delivered in a subsequent attempt.

To correct the issue, we have reverted to the prior version of that library, and we are no longer seeing the memory issues nor the stalled delivery processes.  Over the next several days, the engineering team will also be assessing why the issue did not surface in testing or at our other datacenters.

We are continuing to monitor the systems at the German facility but believe that the issue has been mitigated.