Wednesday, October 27, 2021

[SOLVED] Boto3 Custom waiter rejected for not having a resource permission

Issue

I am trying to create a custom waiter to resume a boto3 script when an rds db cluster is restored to a point in time. (I'm trying to adapt this methodology to my needs: https://medium.com/@Kentzo/customizing-botocore-waiters-83badbfd6399) Aside from the thin documentation on custom waiters this seems like it should be straightforward, but I'm having a permissions issue. The EC2 container where I'm running the script has permissions to run rds:DescribeDBClusters and I can make use of the permission in the script like so:

# Check on the cluster
response = rds.describe_db_clusters(
    DBClusterIdentifier=db_cluster_identifier,
)
status = response['DBClusters'][0]['Status']
print(status)
available

But when I set up a custom waiter to monitor this I get the following error:

botocore.exceptions.WaiterError: Waiter DbClusterRestored failed: User: arn:aws:sts::123456789012:assumed-role/OrgIamRole/i-1234567890abcdef is not authorized to perform: rds:DescribeDBClusters

Perhaps I'm missing something obvious, but I don't understand why the waiter is missing permissions to do something that the script that created the waiter is allowed to do.

The container permissions look like this:

"OrgIamPolicy": {
  "Type": "AWS::IAM::Policy",
  "Properties": {
    "PolicyName": "OrgIamPolicy",
    "Roles": [
      {
        "Ref": "OrgIamRole"
      }
    ],
    "PolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "rds:DescribeDBClusters"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:rds:us-east-1:123456789012:*"
          ]
        }
      ]
    }
  }
}

And here is my code for restoring the cluster and setting up the waiter:

import boto3
import botocore
import os
import subprocess


rds = boto3.client('rds')

db_cluster_target_instance = 'orgstagingrdsinstance'
db_instance_identifier = 'backupinstance'
db_instance_class = 'db.t2.medium'
target_db_cluster_identifier = "org-backup-cluster"
source_db_cluster_identifier = "org-staging-rds-cluster"


# Create the cluster
response = rds.restore_db_cluster_to_point_in_time(
  DBClusterIdentifier=target_db_cluster_identifier,
  RestoreType='copy-on-write',
  SourceDBClusterIdentifier=source_db_cluster_identifier,
  UseLatestRestorableTime=True
)


# Check on the cluster
response = rds.describe_db_clusters(
    DBClusterIdentifier=db_cluster_identifier,
)
status = response['DBClusters'][0]['Status']
print(status)


# Create waiter
delay = 10
max_attempts = 30
waiter_name = "DbClusterRestored"

model = botocore.waiter.WaiterModel({
  "version": 2,
  "waiters": {
    "DbClusterRestored": {
      "operation": "DescribeDBClusters",
      "delay": delay,
      "maxAttempts": max_attempts,
      "acceptors": [
        {
          "matcher": "pathAll",
          "expected": "available",
          "state": "success",
          "argument": "DBClusters[].Status"
        },
        {
          "matcher": "pathAll",
          "expected": "deleting",
          "state": "failure",
          "argument": "DBClusters[].Status"
        },
        {
          "matcher": "pathAll",
          "expected": "creating",
          "state": "failure",
          "argument": "DBClusters[].Status"
        },
      ]
    }
  }
})

waiter = botocore.waiter.create_waiter_with_client(waiter_name, model, rds)
waiter.wait()

Obviously I have this code trimmed and I have obfuscated personal data. Sorry for any errors this might have introduced.

Any help you might give is appreciated.


Solution

Okay, the answer to this seems to be pretty simple. The issue is with the scope of the request. The user has permission to run this on the following resource:

"Resource": [
  "arn:aws:rds:us-east-1:123456789012:*"
]

When I ran

response = rds.describe_db_clusters(
    DBClusterIdentifier=db_cluster_identifier,
)

I was constraining the scope to a cluster that was in arn:aws:rds:us-east-1:123456789012:*. When I ran

waiter = botocore.waiter.create_waiter_with_client(waiter_name, model, rds)
waiter.wait()

I was not passing in that constraint. What I needed to run was

waiter = botocore.waiter.create_waiter_with_client(waiter_name, model, rds)
waiter.wait(DBClusterIdentifier=db_cluster_identifier)

This passed the necessary constraint in and made sure that the permission scope matched the request.

I hope this helps someone in a similar situation.



Answered By - greendemiurge
at