11:00 AM Pacific — In the last several days and particularly in the last 24 hours, we were seeing intermittent network issues between our primary German datacenter and our California datacenter.  The intermittent latency and slow transfer times were impacting database replication over the WAN, which in some cases led to various reports (including statistics, logs, and quarantine/digest data) being temporarily understated.  Message flow was not affected.

Working with the network providers at each of the datacenters, our engineers determined that the problem was with a part of the Level 3 network, an intermediary path between the network providers.  The routing from the California datacenter to the German datacenter has been updated with a shunt to bypass Level 3, and that appears to have resolved the network transfer speed and database replication performance.