Partial outage on Golive Rest API

Incident Report for Apwide

Resolved

Following an operational/infrastructure change (update of our API gateway), Golive Rest API was in partial outage.

Impact: Golive frontend was up and running, but some generated Rest API tokens were considered revoked. This resulted in some failing API calls (HTTP 500).

Actions (UTC time):
- 3:00am: rollback operational change
- 6:32am: re-apply change with first attempt of resolution + monitoring
- 8:00am: new occurrences identified on production
- 9:32am: push fix for the second attempt of resolution
- 10:30am: still occurrences of error found on production
- 1:10pm: apply new fix on API gateway + monitoring
- 4pm: no more occurrences, problem seems fixed

We apologize for the inconvenience caused.

Posted Jan 17, 2023 - 03:00 CET