We have fixed an error related to our Database replicas and everything is now operating normally. We're still monitoring the system for overall health.
Posted Dec 11, 2020 - 13:18 PST
We are seeing error rates spiking again. We are investigating
Posted Dec 11, 2020 - 13:07 PST
Our fix has been deployed and all systems are showing as currently healthy. New jobs should be able to be created, existing jobs should now be progressing normally, and webhook updates should all be firing. Queued courier and job updates are now being drained and sent.
Posted Dec 11, 2020 - 12:31 PST
We're performing another code rollback now which should then allow new jobs to progress normally. Existing job updates (courier updates and status updates) are being queued and will be sent once the system is healthy. Existing jobs have not been able to advance, but will do so once this rollback is complete.
Posted Dec 11, 2020 - 12:25 PST
We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Dec 11, 2020 - 11:42 PST
This incident affected: API and Partner Dashboard.