We help IT Professionals succeed at work.

How to use IAM role to access S3 bucket

mmingfeilam
mmingfeilam asked
on
I want to upload a file to S3 bucket, but my company want to use IAM role as opposed to access keys.  This is AWS documentation on how to upload to S3 asynchronously:

S3AsyncClient client = S3AsyncClient.create();
		CompletableFuture<PutObjectResponse> future = client.putObject(
				PutObjectRequest.builder().bucket(BUCKET)
						.key(fileName)
						.build(),
				AsyncRequestBody
						.fromFile(fromFile.toPath()));
		future.whenComplete((resp, err) -> {
			try {
				if (resp != null) {
					System.out.println("my response: " + resp);
				} else {
					// Handle error
					err.printStackTrace();
				}
			} finally {
				// Lets the application shut down. Only close the client when
				// you are completely done with it.
				client.close();
				
			}
		});

		future.join();
	}

Open in new window


I don't see anywhere to put in IAM role info.  I tried to put it in ~/.aws/credentials in this form:

[useraccount]
aws_access_key_id=<key>
aws_secret_access_key=<secret>

[somerole]
role_arn=<the ARN of the role you want to assume>
source_profile=useraccount

but so far haven't gotten it to work.  I read somewhere you need to use STSAssumeRoleSessionCredentialsProvider but didn't see any good examples.  My main question is do I even need to do anything if I already assigned the IAM role to an ECS instance.  Can someone help me?   Thanks.
Comment
Watch Question

View Solution Only

Phil PhillipsSenior Platform Engineer

Commented:
If you've assigned an IAM role to the instance, you shouldn't need to configure anything additional for AWS credentials. If AWS credentials are missing, the client will default to attempting to use IAM role.

Some debugging steps you can try:
  • Remove AWS credential files
  • Use AWS CLI to see if you're able to do simple s3 commands (ie: aws s3 ls)
mmingfeilam

Author

Commented:
Yes you are right, I found that after I put in the config and credentials files, it works, except for the first time.  For the first time my code tries to upload to S3, it will get:

Failed to upload to S3: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), ProfileCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., ProfileCredentialsProvider(): java.nio.channels.ClosedByInterruptException, WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ContainerCredentialsProvider(): Interrupted waiting to refresh the value., InstanceProfileCredentialsProvider(): Interrupted waiting to refresh the value.]

but subsequent attempts would work, until I log in the next time, then the same issue occurs.
mmingfeilam

Author

Commented:
This is how I generate the credential files in ECS

mkdir -p $ORACLE_HOME/.aws
echo "[default]" > $ORACLE_HOME/.aws/config
echo "region=${AWS_REGION}" >> $ORACLE_HOME/.aws/config
echo "output=json" >> $ORACLE_HOME/.aws/config

echo "[${AWS_PROFILE}]" > $ORACLE_HOME/.aws/credentials
echo "role_arn=${AWS_ROLE_ARN}" >> $ORACLE_HOME/.aws/credentials
echo "source_profile=[default]" >> $ORACLE_HOME/.aws/credentials
Phil PhillipsSenior Platform Engineer

Commented:

Hmm, this part is interesting: 

 InstanceProfileCredentialsProvider(): Interrupted waiting to refresh the value

I wonder if it helps at all to set up the credentials provider to get the IAM credentials asynchronously. Maybe try replacing the first line of your code with something like:

AwsCredentialsProvider awsCredentialsProvider = InstanceProfileCredentialsProvider.builder().asyncCredentialUpdateEnabled(Boolean.TRUE).build();
S3AsyncClient client = S3AsyncClient.builder().credentialsProvider(awsCredentialsProvider).build();


Other than that, I'm not too sure - haven't worked too much with Java + AWS/S3 recently.

mmingfeilam

Author

Commented:
I implemented you code, and got this:

Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:256:t@1997049922:ExportPPTSingleBudgetBean:Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:256:t@1997049922:ExportPPTSingleBudgetBean:com.fasttrack.utilities.FTException: Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.lambda$uploadS3Async$1(ExportBudgetBase.java:1570)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.uploadS3Async(ExportBudgetBase.java:1562)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.saveWorkBook(ExportBudgetBase.java:1520)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.processExportRequest(BudgetBean.java:7748)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.access$300(BudgetBean.java:92)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:122)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:118)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jan 24, 2020 02:49 (UTC-8):       at java.lang.Thread.run(Thread.java:748)
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:258:t@1997049922:ExportPPTSingleBudgetBean:Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:258:t@1997049922:ExportPPTSingleBudgetBean:com.fasttrack.utilities.FTException: Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.uploadS3Async(ExportBudgetBase.java:1586)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.saveWorkBook(ExportBudgetBase.java:1520)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.processExportRequest(BudgetBean.java:7748)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.access$300(BudgetBean.java:92)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:122)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:118)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jan 24, 2020 02:49 (UTC-8):       at java.lang.Thread.run(Thread.java:748)
mmingfeilam

Author

Commented:
Jan 24, 2020 02:52 (UTC-8): Jan 24 10:52:02:585:t@1738448475:ExportPPTSingleBudgetBean:Failed to upload to S3: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint.
mmingfeilam

Author

Commented:
I think the issue is not about credentials but why the S3Client is getting interrupted. I will post another question.
Commented:
I think this issue is due to another problem with S3 Client in multi-threaded environments.