Fixing S3 anonymously owned objects for full cross account access

I recently solved an issue with ownership of objects in one of our publicly facing S3 buckets that I’d like to share with you.

I work at Equalum which is a clustered, highly available Big Data integration Software aimed at enterprise customers.

Our customers use Equalum for running Streaming Big Data pipelines at blazing fast speeds with high velocity, and debugging any small issue with millions of events being generated and processed per second is not a simple matter.

For any incident, we need many logs from all over the cluster and from the many components that we run under the hood such as Apache Spark, Kafka and Zookeeper and many others.

This means a very large file is created and uploaded by our customers when there is an issue to be available for our Support and Developers to analyze and debug.

Equalum is typically installed by our customer in their own environments near their data stores, so most of the log packages are coming from within our customer networks, this forced us to open up the bucket we use for receiving log packages for public upload of files from virtually any source.

Technically, this means we allowed S3 access to the AllUsers Principal, which is denoted in the bucket policy as follows:

"Principal": "*"

So for some time, the developers just used the public access download URL which was allowed.

However, recently, as part of security enhancing of our AWS accounts, we closed access for public downloads.

This forced me to solve the issue allowing our developers to use their own credentials to properly download the support files instead of using the public URLs they were using up to now.

Now to complicate things a bit, we use two AWS accounts, and

The bucket we use for uploading the objects resides in which is the older one but our main use and activity is in
So we try to limit our access to as much as possible to the point that some developers don’t even have credentials there.

The bucket itself has the following policy set on it to allow full S3 access from both accounts and to eliminate the need for credentials in the old account.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicPutObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
},
{
"Sid": "AllowAccessOurAccounts",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<ACCT_1_ID>:root",
"arn:aws:iam::<ACCT_2_ID>:root"
]
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>",
"arn:aws:s3:::<BUCKET_NAME>/*"
]
}
]
}

USEFUL NOTE: To test all below examples yourself, all you need to do is set the following environment variables in your shell before you run the command

* object key must be the full path in the bucket
* Account 1 is the old account
* Account 2 is the default profile for the AWS cli

Now that we allowed all S3 actions on this bucket to both accounts, it should work no?
Ok, let’s check and run the commands our developers are using for trying to download files from the bucket:

aws s3 cp s3://${BUCKET}/${OBJ_KEY} .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Okay, maybe that is because they are using the wrong credentials, let’s use the correct account credentials then.

aws --profile ${ACCT_1} s3 cp s3://${BUCKET}/${OBJ_KEY} .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Now that’s weird, since this is definitely the correct profile from the bucket account and definitely no IAM permissions issue, as there is permissions on this bucket.

Let’s try getting the object ACL

aws s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied

No luck! let’s try the profile from the same account as the bucket.

➜ aws --profile ${ACCT_1} s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied

Still no luck!
Okay then, let’s try on the bucket instead

aws s3api list-objects --bucket ${BUCKET} --query 'Contents[*].{Name:"Key",ID:Owner.ID,Class:StorageClass}'| grep -B1 -A3 ${OBJ_KEY}    {
"Name": "<OBJ_KEY_NAME>",
"ID": null,
"Class": "STANDARD"
},

Okay, thats a bit more info, but what is Owner ID ?

Let’s try again with the bucket owner profile

aws --profile ${ACCT_1} s3api list-objects --bucket ${BUCKET} --query 'Contents[*].{Name:"Key",ID:Owner.ID,Class:StorageClass}'| grep -B1 -A3 ${OBJ_KEY}
{
"Name": "<OBJ_KEY_NAME>",
"ID": "65a011a29cdf8ec533ec3d1ccaae921c",
"Class": "STANDARD"
},

What is this ID?

After some digging I found out that when an object gets uploaded to the bucket with Public permissions, by default the object is not owned by the bucket owner!

It is owned by the Anonymous Principal which when queried from an external account owner, shows the ID of but when queried by the bucket owner shows the AWS default anonymous ID:

Next step should be to changing the object ownership to the bucket account owner, right?

I ran the following command to transfer the object ownership to be owned by the bucket owner.

aws s3api put-object-acl --bucket ${BUCKET} --acl bucket-owner-full-control --key ${OBJ_KEY}An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied

No luck changing ownership!

Let’s try the other profile again.

aws --profile ${ACCT_1} s3api put-object-acl --bucket ${BUCKET} --acl bucket-owner-full-control --key ${OBJ_KEY}An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied

Still not working! what’s going on here?
At this point I found out that there is an important flag that needs to be added to the command as follows:

aws s3api put-object-acl --bucket ${BUCKET} --acl bucket-owner-full-control --key ${OBJ_KEY} --no-sign-request

Finally! it worked!

Now let’s try to view the object’s ACL, first via the second account.

aws s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied

Okay, no success here, let’s try the first account again

➜ aws --profile ${ACCT_1} s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}
{
"Owner": {
"ID": "65a011a29cdf8ec533ec3d1ccaae921c"
},
"Grants": [
{
"Grantee": {
"ID": "65a011a29cdf8ec533ec3d1ccaae921c",
"Type": "CanonicalUser"
},
"Permission": "FULL_CONTROL"
},
{
"Grantee": {
"DisplayName": "<YOUR_ACCOUNT_NAME>",
"ID": "<YOUR_ACCOUNT_ID>",
"Type": "CanonicalUser"
},
"Permission": "FULL_CONTROL"
}
]
}

It works!
What happens at this stage, is that AWS adds the account owner as an additional grantee of the object, but not the primary owner which is still the AWS default anonymous ID.

Seems ok, let’s try to download the file now.

aws s3 cp s3://${BUCKET}/2258/scripts-2.17.0.11-RELEASE.zip .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Account 2 still not working, let’s try Account 1.

aws --profile ${ACCT_1} s3 cp s3://${BUCKET}/${OBJ_KEY} .
download: s3://<BUCKET>/<OBJ_KEY> to ./<FILENAME>

Okay that’s good, but remember, our main account being used by the developers is the second account and we must have the credentials from Account 2 working, why is it not working if the bucket’s IAM Policy permits full access?

For this I learned, that to complete and fully transfer the object to the bucket ownership, I had to copy the object over itself!

aws --profile ${ACCT_1} s3 cp s3://${BUCKET}/${OBJ_KEY} s3://${BUCKET}/${OBJ_KEY} --storage-class STANDARD

Only then did we have all the objects changed to effectively work with our account IAM permissions since we transferred the ownership over to us.

And now, the credentials for account 2 work just fine for all operations
Copy:

aws s3 cp s3://${BUCKET}/${OBJ_KEY} .
download: s3://<BUCKET>/<OBJ_KEY> to ./<FILENAME>

Listing object details and permissions both through the bucket and directly through the object key.

aws s3api list-objects --bucket ${BUCKET} --query 'Contents[*].{Name:"Key",ID:Owner.ID,Class:StorageClass}'| grep -B1 -A3 ${OBJ_KEY}
{
"Name": "<OBJ_KEY>",
"ID": "<ACCT_1_OWNER_ID>",
"Class": "STANDARD"
},
aws s3api get-object-acl --bucket ${BUCKET} --key ${OBJ_KEY}
{
"Owner": {
"DisplayName": "<ACCT_1_OWNER_NAME>",
"ID": "<ACCT_1_OWNER_ID>"
},
"Grants": [
{
"Grantee": {
"DisplayName": "<ACCT_1_OWNER_NAME>",
"ID": "<ACCT_1_OWNER_ID>",
"Type": "CanonicalUser"
},
"Permission": "FULL_CONTROL"
}
]
}

Note that the AWS Anonymous ID is now gone from the ACL!

Admittedly, it is possible to configure the bucket setting itself to enforce bucket ownership on newly uploaded objects as follows:

However, as can be seen in the screenshot, this will only serve to deny upload requests unless the proper ACL is specified in the upload API request.

In my case, I could not change the tool being used at the customer to upload, so in any case it would not have solved my issue .

Hope this helps you!

DevOps