The sequencer uses a cloud DB for storage. An internal GCP fault - verified with Google - caused one of the resources storing our DB to crash. Consequently, the sequencer wasn’t unable to read from or write to storage. Hence it was unable to execute transactions and produce new blocks.

The storage nodes did not automatically recover after the GCP fault was resolved due to enforcement of a strong consistency condition: a manual process must be initiated after confirming there was no data corruption. The nodes were revived after ensuring the DB is in order.

The total sequencer downtime was roughly 2 hours. The only effect was a break in block production.

Improvements to the response protocol will be discussed in the upcoming days.