Link to home
Start Free TrialLog in
Avatar of Michael Lam
Michael Lam

asked on

How to use IAM role to access S3 bucket

I want to upload a file to S3 bucket, but my company want to use IAM role as opposed to access keys.  This is AWS documentation on how to upload to S3 asynchronously:

S3AsyncClient client = S3AsyncClient.create();
		CompletableFuture<PutObjectResponse> future = client.putObject(
				PutObjectRequest.builder().bucket(BUCKET)
						.key(fileName)
						.build(),
				AsyncRequestBody
						.fromFile(fromFile.toPath()));
		future.whenComplete((resp, err) -> {
			try {
				if (resp != null) {
					System.out.println("my response: " + resp);
				} else {
					// Handle error
					err.printStackTrace();
				}
			} finally {
				// Lets the application shut down. Only close the client when
				// you are completely done with it.
				client.close();
				
			}
		});

		future.join();
	}

Open in new window


I don't see anywhere to put in IAM role info.  I tried to put it in ~/.aws/credentials in this form:

[useraccount]
aws_access_key_id=<key>
aws_secret_access_key=<secret>

[somerole]
role_arn=<the ARN of the role you want to assume>
source_profile=useraccount

but so far haven't gotten it to work.  I read somewhere you need to use STSAssumeRoleSessionCredentialsProvider but didn't see any good examples.  My main question is do I even need to do anything if I already assigned the IAM role to an ECS instance.  Can someone help me?   Thanks.
Avatar of Phil Phillips
Phil Phillips
Flag of United States of America image

If you've assigned an IAM role to the instance, you shouldn't need to configure anything additional for AWS credentials. If AWS credentials are missing, the client will default to attempting to use IAM role.

Some debugging steps you can try:
  • Remove AWS credential files
  • Use AWS CLI to see if you're able to do simple s3 commands (ie: aws s3 ls)
Avatar of Michael Lam
Michael Lam

ASKER

Yes you are right, I found that after I put in the config and credentials files, it works, except for the first time.  For the first time my code tries to upload to S3, it will get:

Failed to upload to S3: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), ProfileCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., ProfileCredentialsProvider(): java.nio.channels.ClosedByInterruptException, WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ContainerCredentialsProvider(): Interrupted waiting to refresh the value., InstanceProfileCredentialsProvider(): Interrupted waiting to refresh the value.]

but subsequent attempts would work, until I log in the next time, then the same issue occurs.
This is how I generate the credential files in ECS

mkdir -p $ORACLE_HOME/.aws
echo "[default]" > $ORACLE_HOME/.aws/config
echo "region=${AWS_REGION}" >> $ORACLE_HOME/.aws/config
echo "output=json" >> $ORACLE_HOME/.aws/config

echo "[${AWS_PROFILE}]" > $ORACLE_HOME/.aws/credentials
echo "role_arn=${AWS_ROLE_ARN}" >> $ORACLE_HOME/.aws/credentials
echo "source_profile=[default]" >> $ORACLE_HOME/.aws/credentials

Hmm, this part is interesting: 

InstanceProfileCredentialsProvider(): Interrupted waiting to refresh the value

Open in new window

I wonder if it helps at all to set up the credentials provider to get the IAM credentials asynchronously. Maybe try replacing the first line of your code with something like:

AwsCredentialsProvider awsCredentialsProvider = InstanceProfileCredentialsProvider.builder().asyncCredentialUpdateEnabled(Boolean.TRUE).build();
S3AsyncClient client = S3AsyncClient.builder().credentialsProvider(awsCredentialsProvider).build();

Open in new window


Other than that, I'm not too sure - haven't worked too much with Java + AWS/S3 recently.

I implemented you code, and got this:

Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:256:t@1997049922:ExportPPTSingleBudgetBean:Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:256:t@1997049922:ExportPPTSingleBudgetBean:com.fasttrack.utilities.FTException: Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.lambda$uploadS3Async$1(ExportBudgetBase.java:1570)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.uploadS3Async(ExportBudgetBase.java:1562)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.saveWorkBook(ExportBudgetBase.java:1520)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.processExportRequest(BudgetBean.java:7748)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.access$300(BudgetBean.java:92)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:122)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:118)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jan 24, 2020 02:49 (UTC-8):       at java.lang.Thread.run(Thread.java:748)
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:258:t@1997049922:ExportPPTSingleBudgetBean:Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8): Jan 24 10:49:42:258:t@1997049922:ExportPPTSingleBudgetBean:com.fasttrack.utilities.FTException: Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.uploadS3Async(ExportBudgetBase.java:1586)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.ExportBudgetBase.saveWorkBook(ExportBudgetBase.java:1520)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.processExportRequest(BudgetBean.java:7748)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean.access$300(BudgetBean.java:92)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:122)
Jan 24, 2020 02:49 (UTC-8):       at com.fasttrack.gm.servlet.BudgetBean$AsyncExporter$1.call(BudgetBean.java:118)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jan 24, 2020 02:49 (UTC-8):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jan 24, 2020 02:49 (UTC-8):       at java.lang.Thread.run(Thread.java:748)
Jan 24, 2020 02:52 (UTC-8): Jan 24 10:52:02:585:t@1738448475:ExportPPTSingleBudgetBean:Failed to upload to S3: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint.
I think the issue is not about credentials but why the S3Client is getting interrupted. I will post another question.
ASKER CERTIFIED SOLUTION
Avatar of Michael Lam
Michael Lam

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial