Fix Anonymous Owned Objects in S3

I recently solved an issue with ownership of objects in one of our publicly facing S3 buckets that I’d like to share with you.

I work at Equalum, a clustered, highly available Big Data integration Software aimed at enterprise customers.

Our customers use Equalum for running Streaming Big Data pipelines at blazing fast speeds with high velocity, and debugging any small issue with millions of events being generated and processed per second is not a simple matter.

For any incident, we need many logs from all over the cluster and from the many components that we run under the hood such as Apache Spark, Kafka and Zookeeper, and many others.

This means our customers create and upload a very large file when there is an issue available for our Support and Developers to analyze and debug.

Equalum is typically installed by our customers in their own environments near their data stores, so most of the log packages are coming from within our customer networks, this forced us to open up the bucket we use for receiving log packages for public upload of files from virtually any source.

Technically, this means we allowed S3 access to the AllUsers Principal, which is denoted in the bucket policy as follows:

"Principal": "*"

So for some time, the developers just used the public access download URL, which was allowed.

However, recently, as part of the security-enhancing of our AWS accounts, we closed access for public downloads.

This forced me to solve the issue of allowing our developers to use their own credentials to properly download the support files instead of using the public URLs they were using up to now.

Now to complicate things a bit, we use two AWS accounts, ACCT_1 and ACCT_2

The bucket we use for uploading the objects resides in ACCT_1 which is the older one, but our main use and activity are in ACCT_2
So we try to limit our access to ACCT_1 as much as possible to the point that some developers don’t even have credentials there.

The bucket itself has the following policy set on it to allow full S3 access from both accountsACCT_1 and ACCT_2 to eliminate the need for credentials in the old account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicPutObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
        },
        {
            "Sid": "AllowAccessOurAccounts",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<ACCT_1_ID>:root",
                    "arn:aws:iam::<ACCT_2_ID>:root"
                ]
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>",
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}

USEFUL NOTE: To test all below examples yourself, all you need to do is set the following environment variables in your shell before you run the command

export BUCKET=<YOUR_BUCKET_NAME>
export OBJ_KEY=<YOUR_OBJECT_KEY>
export ACCT_1=<ACCT_1_PROFILE>

* object key must be the full path in the bucket
* Account 1 is the old account
* Account 2 is the default profile for the AWS cli

Now that we allowed all S3 actions on this bucket to both accounts, it should work, no?
Ok, let’s check and run the commands our developers are using for trying to download files from the bucket:

aws s3 cp s3://${BUCKET}/${OBJ_KEY} .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Okay, maybe that is because they are using the wrong credentials. Let’s use the correct account credentials then.

aws --profile ${ACCT_1} s3 cp s3://${BUCKET}/${OBJ_KEY} .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Now that’s weird since this is definitely the correct profile from the bucket account and definitely no IAM permissions issue, as there areS3FullAccess permissions on this bucket.

Let’s try getting the object ACL

aws s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied

No luck! Let’s try the profile from the same account as the bucket.

➜ aws --profile ${ACCT_1} s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied

Still no luck!
Okay then, let’s trylist-objects on the bucket instead

aws s3api list-objects --bucket ${BUCKET} --query 'Contents[*].{Name:"Key",ID:Owner.ID,Class:StorageClass}'| grep -B1 -A3 ${OBJ_KEY}
    {
        "Name": "<OBJ_KEY_NAME>",
        "ID": null,
        "Class": "STANDARD"
    },

Okay, that's a bit more info, but what is Owner ID null?

Let’s try again with the bucket owner profile

aws --profile ${ACCT_1} s3api list-objects --bucket ${BUCKET} --query 'Contents[*].{Name:"Key",ID:Owner.ID,Class:StorageClass}'| grep -B1 -A3 ${OBJ_KEY}
    {
        "Name": "<OBJ_KEY_NAME>",
        "ID": "65a011a29cdf8ec533ec3d1ccaae921c",
        "Class": "STANDARD"
    },

What is this ID?

After some digging, I found out that when an object gets uploaded to the bucket with Public permissions, by default, the object is not owned by the bucket owner!

It is owned by the Anonymous Principal, which when queried from an external account owner, shows the ID of null but when queried by the bucket owner, shows the AWS default anonymous ID: 65a011a29cdf8ec533ec3d1ccaae921c

The next step should be to change the object ownership to the bucket account owner.

I ran the following command to transfer the object ownership to be owned by the bucket owner.

aws s3api put-object-acl --bucket ${BUCKET} --acl bucket-owner-full-control --key ${OBJ_KEY}
An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied

No luck changing ownership!

Let’s try the other profile again.

aws --profile ${ACCT_1} s3api put-object-acl --bucket ${BUCKET} --acl bucket-owner-full-control --key ${OBJ_KEY}
An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied

Still not working! What’s going on here?
At this point, I found out that there is an important flag that needs to be added to the put-object-acl command as follows:

aws s3api put-object-acl --bucket ${BUCKET} --acl bucket-owner-full-control --key ${OBJ_KEY} --no-sign-request

Finally! it worked!

Now let’s try to view the object’s ACL, first via the second account.

aws s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied

Okay, no success here. Let’s try the first account again

➜ aws --profile ${ACCT_1} s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}
{
    "Owner": {
        "ID": "65a011a29cdf8ec533ec3d1ccaae921c"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "65a011a29cdf8ec533ec3d1ccaae921c",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "DisplayName": "<YOUR_ACCOUNT_NAME>",
                "ID": "<YOUR_ACCOUNT_ID>",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

It works!
What happens at this stage is that AWS adds the account owner as an additional grantee of the object but not the primary owner, which is still the AWS default anonymous ID.

Seems ok. Let’s try to download the file now.

aws s3 cp s3://${BUCKET}/2258/scripts-2.17.0.11-RELEASE.zip .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Account 2 is still not working. Let’s try Account 1.

aws --profile ${ACCT_1} s3 cp s3://${BUCKET}/${OBJ_KEY} .
download: s3://<BUCKET>/<OBJ_KEY> to ./<FILENAME>

Okay, that’s good, but remember, our main account being used by the developers is the second account, and we must have the credentials from Account 2 working. Why is it not working if the bucket’s IAM Policy permits full access?

For this, I learned that to complete and fully transfer the object to the bucket ownership. I had to copy the object over itself!

aws --profile ${ACCT_1} s3 cp s3://${BUCKET}/${OBJ_KEY} s3://${BUCKET}/${OBJ_KEY} --storage-class STANDARD

Only then did we have all the objects changed to effectively work with our account IAM permissions since we transferred the ownership to us.

And now, the credentials for account 2 work just fine for all operations
Copy:

aws s3 cp s3://${BUCKET}/${OBJ_KEY} .
download: s3://<BUCKET>/<OBJ_KEY> to ./<FILENAME>

Listing object details and permissions both through the bucket and directly through the object key.

aws s3api list-objects --bucket ${BUCKET} --query 'Contents[*].{Name:"Key",ID:Owner.ID,Class:StorageClass}'| grep -B1 -A3 ${OBJ_KEY}
    {
        "Name": "<OBJ_KEY>",
        "ID": "<ACCT_1_OWNER_ID>",
        "Class": "STANDARD"
    },
aws s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}
{
    "Owner": {
        "DisplayName": "<ACCT_1_OWNER_NAME>",
        "ID": "<ACCT_1_OWNER_ID>"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "<ACCT_1_OWNER_NAME>",
                "ID": "<ACCT_1_OWNER_ID>",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

Note that the AWS Anonymous ID is now gone from the ACL!

Admittedly, it is possible to configure the bucket setting itself to enforce bucket ownership on newly uploaded objects as follows:

However, as can be seen in the screenshot, this will only serve to deny upload requests unless the proper ACL bucket-owner-full-control is specified in the upload API request.

In my case, I could not change the tool being used by the customer to upload, so in any case, it would not have solved my issue.

Hope this helps you!