Somebody broke my AWS Landing Zone?
Last week I was updating an AWS Landing Zone environment to the latest version. It was late on a Thursday and I was running the upgrade in my own environment and having a few minor issues so I decided to come back in the morning to finish it off.
Come the next morning, I re-ran it. All of a sudden I’m getting a new error in the CodeBuild step that I’ve never seen before. It has run this step successfully just the night before!
It turns out that an update to the Python botocore package was release overnight that removed docevents from botocore, as documented here.
Of course, AWS Landing Zone (all the way up to 2.4) uses the AWS CLI in the CodeBuild steps which relies on botocore.
Fear not, the answer is simple. It turns out docevents has been moved to the AWS CLI in version 1.18.140 (aws/aws-cli#5538). CodeBuild images are not using the latest AWS CLI today. The fix to this problem in an update to the CodeBuild spec for AWS Landing Zone so that it also updates the AWS CLI.
The proper way is to update the build spec for Landing Zone via the CloudFormation initiation template, aws-landing-zone-initation.template. In the template you used for Landing Zone
- Search for
- pip install --upgrade boto3\n(there are 2 occurrences in all version before 2.4 and 3 occurrences in v2.4);
- pip install --upgrade awscli\nbefore the boto3 update above so that it now reads
- pip install --upgrade awscli\n - pip install --upgrade boto3\n;
- Update the stack with the new template.
This will update the build spec so that it looks like the below. Hey presto, your Landing Zone is ready to go again!
What about v2.4.1?
AWS has just released v2.4.1 of Landing Zone to fix this problem. The AWS fix is to remove the update of boto3 from the build spec. I’m not sure this is the best move as any other dependency on botocore versions could break this dependency until CodeBuild is updated to the latest AWS CLI. Caveat Emptor.
Update: the CodeBuild image version used in v2.4.1 has been updated so that is uses the latest AWS CLI and the concern above no longer applies! If you want to update to v2.4.1, go ahead. If not, then you can follow the steps above to update the CodeBuild build spec.
My other workflows are broken!
Fear not, the same technique can be applied to any CodeBuild step you have that relies on botocore!