Avatar of Michael Lam
Michael Lam
 asked on

How to use IAM role to access S3 bucket

I want to upload a file to S3 bucket, but my company want to use IAM role as opposed to access keys.  This is AWS documentation on how to upload to S3 asynchronously:

S3AsyncClient client = S3AsyncClient.create();
		CompletableFuture<PutObjectResponse> future = client.putObject(
				PutObjectRequest.builder().bucket(BUCKET)
						.key(fileName)
						.build(),
				AsyncRequestBody
						.fromFile(fromFile.toPath()));
		future.whenComplete((resp, err) -> {
			try {
				if (resp != null) {
					System.out.println("my response: " + resp);
				} else {
					// Handle error
					err.printStackTrace();
				}
			} finally {
				// Lets the application shut down. Only close the client when
				// you are completely done with it.
				client.close();
				
			}
		});

		future.join();
	}

Open in new window


I don't see anywhere to put in IAM role info.  I tried to put it in ~/.aws/credentials in this form:

[useraccount]
aws_access_key_id=<key>
aws_secret_access_key=<secret>

[somerole]
role_arn=<the ARN of the role you want to assume>
source_profile=useraccount

but so far haven't gotten it to work.  I read somewhere you need to use STSAssumeRoleSessionCredentialsProvider but didn't see any good examples.  My main question is do I even need to do anything if I already assigned the IAM role to an ECS instance.  Can someone help me?   Thanks.
JavaAWS

Avatar of undefined
Last Comment
Michael Lam

8/22/2022 - Mon
Phil Phillips

If you've assigned an IAM role to the instance, you shouldn't need to configure anything additional for AWS credentials. If AWS credentials are missing, the client will default to attempting to use IAM role.

Some debugging steps you can try:
  • Remove AWS credential files
  • Use AWS CLI to see if you're able to do simple s3 commands (ie: aws s3 ls)
Michael Lam

ASKER
Yes you are right, I found that after I put in the config and credentials files, it works, except for the first time.  For the first time my code tries to upload to S3, it will get:

Failed to upload to S3: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), ProfileCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., ProfileCredentialsProvider(): java.nio.channels.ClosedByInterruptException, WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ContainerCredentialsProvider(): Interrupted waiting to refresh the value., InstanceProfileCredentialsProvider(): Interrupted waiting to refresh the value.]

but subsequent attempts would work, until I log in the next time, then the same issue occurs.
Michael Lam

ASKER
This is how I generate the credential files in ECS

mkdir -p $ORACLE_HOME/.aws
echo "[default]" > $ORACLE_HOME/.aws/config
echo "region=${AWS_REGION}" >> $ORACLE_HOME/.aws/config
echo "output=json" >> $ORACLE_HOME/.aws/config

echo "[${AWS_PROFILE}]" > $ORACLE_HOME/.aws/credentials
echo "role_arn=${AWS_ROLE_ARN}" >> $ORACLE_HOME/.aws/credentials
echo "source_profile=[default]" >> $ORACLE_HOME/.aws/credentials
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Phil Phillips

Hmm, this part is interesting: 

InstanceProfileCredentialsProvider(): Interrupted waiting to refresh the value

Open in new window

I wonder if it helps at all to set up the credentials provider to get the IAM credentials asynchronously. Maybe try replacing the first line of your code with something like:

AwsCredentialsProvider awsCredentialsProvider = InstanceProfileCredentialsProvider.builder().asyncCredentialUpdateEnabled(Boolean.TRUE).build();
S3AsyncClient client = S3AsyncClient.builder().credentialsProvider(awsCredentialsProvider).build();

Open in new window


Other than that, I'm not too sure - haven't worked too much with Java + AWS/S3 recently.

Michael Lam

ASKER
I implemented you code, and got this:

Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:256:t@1997049922:ExportPPTSingleBudgetBean:Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:256:t@1997049922:ExportPPTSingleBudgetBean:com.fasttrack.utilities.FTException: Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.lambda$uploadS3Async$1(ExportBudgetBase.java:1570)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.uploadS3Async(ExportBudgetBase.java:1562)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.saveWorkBook(ExportBudgetBase.java:1520)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.processExportRequest(BudgetBean.java:7748)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.access$300(BudgetBean.java:92)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:122)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:118)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jan 24, 2020 02:49 (UTC-8):       at java.lang.Thread.run(Thread.java:748)
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:258:t@1997049922:ExportPPTSingleBudgetBean:Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:258:t@1997049922:ExportPPTSingleBudgetBean:com.fasttrack.utilities.FTException: Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.uploadS3Async(ExportBudgetBase.java:1586)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.saveWorkBook(ExportBudgetBase.java:1520)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.processExportRequest(BudgetBean.java:7748)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.access$300(BudgetBean.java:92)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:122)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:118)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jan 24, 2020 02:49 (UTC-8):       at java.lang.Thread.run(Thread.java:748)
Michael Lam

ASKER
Jan 24, 2020 02:52 (UTC-8): Jan 24 10:52:02:585:t@1738448475:ExportPPTSingleBudgetBean:Failed to upload to S3: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Michael Lam

ASKER
I think the issue is not about credentials but why the S3Client is getting interrupted. I will post another question.
ASKER CERTIFIED SOLUTION
Michael Lam

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.