Home / Security updates / Incident Report
Incident Report
Security updates detail rendered from /security-updates/upd_191312edada22b2e.
Overview
| ID | upd_191312edada22b2e |
| Collection | Security Updates |
| Provider | TrustShare (TrustCloud / Kintent) |
| Company | EarnTrust |
| URL | - |
| Counts | - |
| Updated | - |
Raw record
| Field | Value |
|---|---|
| id | upd_191312edada22b2e |
| providerId | trustshare |
| organizationId | org_781b17177999b69c |
| trustCenterId | tc_cba6746d90ed27e3 |
| title | Incident Report |
| type | incident |
| message | A performance and resilience optimization to the authorization microservice contained a memory leak that was exposed under high traffic. This resulted in a number of pages returning 404’s that should not have. Testing the build in our canary ring did not expose the service to sufficient traffic to discover the leak, allowing it to graduate to production at 6:37 PM UTC. The memory leak under high load caused pods to crash repeatedly starting at 6:42 PM UTC, failing authorization checks. These failures triggered alerts at 6:44 PM UTC. Rolling back the authorization service change was delayed as parts of the deployment infrastructure relied on the authorization service and required manual intervention to complete. Rollback completed at 7:08 PM UTC and all impacted GitHub features recovered after pods came back online. We are evaluating changes to our rollout strategy to better detect this sooner and with less impact, changes to remove the dependency between authorization services and deployment rollback, and incident response improvements to reduce the overall time to recover. This incident is unrelated to the Slack integration incident. The combined status updates are a limitation of our status reporting tooling and we recognize the confusion this creates. Work to address this was already in progress and will be complete this month. |
| publishedAt | 2025-04-28 |
| gated | no |
| source | {
"field": "trustshare/notifications[]"
} |
| company | {
"id": "org_781b17177999b69c",
"name": "EarnTrust",
"domains": [
"earntrust.trustshare.com",
"www.kintent.com"
]
} |
| trust_center | {
"id": "tc_cba6746d90ed27e3",
"name": "EarnTrust",
"url": "https://earntrust.trustshare.com",
"host": "earntrust.trustshare.com"
} |
| provider | {
"id": "trustshare",
"name": "TrustShare (TrustCloud / Kintent)"
} |
| links | {
"self": "/v1/security-updates/upd_191312edada22b2e",
"company": "/v1/companies/org_781b17177999b69c",
"trust_center": "/v1/trust-centers/tc_cba6746d90ed27e3",
"provider": "/v1/providers/trustshare"
} |
Get this page with API
Rendered from the bluedoor Trust Centers API. Reproduce it:
GET https://api.bluedoor.sh/trust-centers/v1/security-updates/upd_191312edada22b2eJSON