How to read s3 files from DOcker container

how we can read s3 bucket files form docker container
what are the package needed and what is the command

how we can put data to maria DB form docker container
what are the package needed and what is the command
LVL 20
Sathish David  Kumar NArchitectAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

CEHJCommented:
You might try something like

aws s3 cp s3://my-bucket/test.csv  >(cat) | mysql --local-infile=1 -u user -ppassword mydatabase -e "LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE mytable FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"';"

Open in new window


but as i said before, because of the distributed nature of your data, you should probably use NoSql techniques
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Prabhin MPEngineer-TechOPSCommented:
step1: make sure that aws cli installed in docker container.
if not installed include aws cli in docker file. and Create Environment variables for Access and secret key or move manually from the host machine to Docker container.

aws s3 cp s3://my-bucket/tt.txt . to read the file.
 use pipe to pass the output to mariadb or mysql


Hope these info is enough
0
CEHJCommented:
aws s3 cp s3://my-bucket/tt.txt . to read the file.
 use pipe to pass the output to mariadb or mysql
Again please don't repeat information already stated
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

Prabhin MPEngineer-TechOPSCommented:
I didn't find an alternative for AWS cp command. I also stated the installation of AWS cli . Then you also make sure that  while providing answer all those info are added
0
CEHJCommented:
I also stated the installation of AWS cli .
Not necessary really - it's obvious that those are command line commands
0
Sathish David Kumar NArchitectAuthor Commented:
Thank you both .

But my  docker script will be automatically execute every day
is this possible ?
0
CEHJCommented:
Yes. You could use a cron job
0
Sathish David Kumar NArchitectAuthor Commented:
any ref link ?
0
CEHJCommented:
Hang on - from your new question, i see you're using Lambda. I suggest you confine this issue to that question
0
Sathish David Kumar NArchitectAuthor Commented:
I have different idea. But my target is I need to read file content from s3 and put it in Maria DB

I need to know which one is best
Lambda or directly run cron Job .

For All this i have only Idea .. I dont know how to do .

If any link related to this is very helpful
0
CEHJCommented:
I need to know which one is best
Lambda or directly run cron Job .
That's really a quite different question and should be addressed within the new question you just posted. The current one has been answered
0
Sathish David Kumar NArchitectAuthor Commented:
 aws s3 cp s3://my-bucket/tt.txt . to read the file.  use pipe to pass the output to mariadb

any sample reference for  this ?
0
CEHJCommented:
You don't need one ;) I've given you the actual code (naturally you'll have to change the names) that was just described by someone else
0
Sathish David Kumar NArchitectAuthor Commented:
yes its direct one . But my script will call private s3 bucket ,  So I need to pass  key  to access that

So How to do that ?
0
CEHJCommented:
Not sure i'm afraid - you might need to check the documentation on that issue
0
Sathish David Kumar NArchitectAuthor Commented:
Hi CEHJ... i needto read the file..

cp will copy the file not content!!
is there any cmd available or any docker image for  reading the csv file??
0
CEHJCommented:
The aws s3 command set is extremely limited. The following will show you the contents if the rest of the sql supports it

aws s3 cp s3://my-bucket/test.csv  >(cat) 

Open in new window

0
Sathish David Kumar NArchitectAuthor Commented:
aws s3 cp s3://my-bucket/tt.txt . to read the file.  use pipe to pass the output to mariadb

Can you give me sample code for pipe to pass mariaDB
0
CEHJCommented:
My very first comment gave you sample code
0
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Looks like this question is bouncing back + forth between multiple questions.

CEHJ looks to have answered your original question in his first comment.

Please close this question as answered by CEHJ + then open other questions... one question/post for your other questions.

Likely this will provide more useful info for you.
0
Sathish David Kumar NArchitectAuthor Commented:
Thanks
0
CEHJCommented:
:)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java

From novice to tech pro — start learning today.