This post outlines a recent production issue on the Platform as a Service for government (PaaS) and how it was resolved.
What happened
At 3pm on Friday 16 September 2016, the ssl private key was published in a location accessible to all PaaS team members. It was also made available to a small number of people with ‘read access’ to our terraform repository on our private github enterprise installation.
Only users with access to the team’s encrypted credential store should have access to the ssl private key.
People who have ‘read access’ to the aws-account-wide-terraform repository on github.gds, or those with github.gds admin access, could have got access to the SSL private key.
No users were affected
Although no users were affected, users of live applications hosted on the PaaS could have been at a slightly greater risk of being directed to a ‘fake’ app, although this didn’t happen.
An attacker would have had to reconfigure the GOV.UK infrastructure to send traffic to the spoofed site to affect end users.
Potentially the certificate could have been used to spoof other applications running on the PaaS, but Trade Tariff is the only ‘live’ application.
How we responded
We removed the certificate from aws-account-wide-terraform, and rotated the ssl private key.
Why it occurred
The terraform provider for the api gateway did not hide (sha1) the contents of the private key when storing in the state file.
State files are only checked by the reviewer of a pull-request on aws-account-wide terraform.
What we’re doing to prevent this from happening again
For now, we’ll check the state file contents before committing it to github.
In the future we’ll stop storing the state file in the github repository.
When we kick off stories we’ll flag up the need to take extra care when working with terraform and sensitive data, and when reviewing state files.
We’ll move our support manual, which includes our incident process, to our team manual which is publicly accessible.
We’ll speed up and document our process for rotating and revoking SSL certificates.
Detailed timeline for developers
16 September 10:10am
State file containing the private key commited into aws-account-wide-terraform
12 October 3pm
Code pair notice this while working on adding other SSL private key to aws-account-wide terraform
12 October 4pm
Discussion on how and when the fix can be scheduled as the next priority story, and triggering the incident process.
Decision made not to be fixed straight away and can be played as next highest priority story
13 October 9am
Work begins on removing the certificate from aws-account-wide-terraform