Sunday, January 28, 2024

[SOLVED] secretsmanager rotate-secret race conditions checking AWSPENDING is complete

Issue

I am trying to implement some type of polling mechanism that checks if an AWS Lambda password rotation has actually completed:

From what I understand, a password rotation is completed if:

The AWSPENDING and AWSCURRENT staging labels are attached to the same version of the secret, or The AWSPENDING staging label is not attached to any version of the secret.

This is what I have so far to check if the rotation is completed:

while [ "$rotationComplete" == false ]; do
    versionStages=$(aws secretsmanager get-secret-value --secret-id awsSecret --query 'VersionStages' | jq -r .)

    # Check if the rotation is complete
    if [[ " ${versionStages[@]} " =~ "AWSPENDING" && " ${versionStages[@]} " =~ "AWSCURRENT" ]] || [[ ! " ${versionStages[@]} " =~ "AWSPENDING" ]]; then
        echo "Secret rotation completed."
        rotationComplete=true
    else
        echo "Secret rotation has not completed yet. Waiting..."
        attempt=$((attempt + 1)) 
        sleep 30  #   
    fi  
done

Is there anything else I should check like versionId to check if the password rotation has actually completed?


Solution

Here's what I have found to be the most reliable way to check that an AWS secret password has actually rotated. This is a method of "polling" the AWS secret to verify that the password has successfully been rotated.

Some points:

  • You want to verify that there is a NEW version ID
  • Check that versionStages has the tags AWSPENDING and AWSCURRENT OR is not equal to AWSPENDING
  • I have a maxAttempts check so that I don't get stuck in an infinite loop. There may be a case where the label gets stuck on AWSPENDING in which you will have to call update-secret-version-stage to remove that label.
versionIdBeforeRotateSecret=$(aws secretsmanager get-secret-value --secret-id <YOUR_AWS_SECRET_ID> --profile <YOUR_AWS_PROFILE>--query 'VersionId' | jq -r .)
# rotate secret
aws --profile <YOUR_AWS_PROFILE> secretsmanager rotate-secret --secret-id <YOUR_AWS_SECRET_ID>
rotationComplete=false
attempt=1
maxAttempts=5
secret_info=$(aws secretsmanager get-secret-value --secret-id <YOUR_AWS_SECRET_ID> --profile <YOUR_AWS_PROFILE>)
versionId=$(echo "$secret_info" | jq -r '.VersionId')
versionStages=($(echo "$secret_info" | jq -r '.VersionStages[]'))
while [ "$rotationComplete" == false ] && [ "$attempt" -le "$maxAttempts" ]; do        
  if [[ " ${versionStages[@]} " =~ "AWSPENDING" && " ${versionStages[@]} " =~ "AWSCURRENT" ]] || [[ ! " ${versionStages[@]} " =~ "AWSPENDING" ]] && [ "$versionId" != "$versionIdBeforeRotateSecret" ]; then
      echo "Secret rotation completed."
      rotationComplete=true
  else
        echo "Secret rotation has not completed yet. Waiting..."
        attempt=$((attempt + 1)) 
        sleep 60  # 
  fi  
  done
done


Answered By - jipot
Answer Checked By - Marie Seifert (WPSolving Admin)
at