Link to home
Create AccountLog in
Microsoft SQL Server

Microsoft SQL Server

--

Questions

--

Followers

Top Experts

Avatar of Richard
Richard🇺🇸

AWS RDS and no Bulk Insert
About once a month we get a CSV file with anywhere between 1,000 and 40,000 records. We have a program which uploads the data to a data table.  When our SQL server was local this upload worked fine.  We have migrated to AWS RDS.  Now the upload time is very painful (to say the least!).  I was planning on changing the upload class to use the SQL Bulk Insert function - only to find out that (as of 2017) AWS doesn't support the function.  I'm looking for ways to improve the upload.  We have a 1Gb Internet connection.  Does anyone have any thoughts on this?
Thank you in advance

Zero AI Policy

We believe in human intelligence. Our moderation policy strictly prohibits the use of LLM content in our Q&A threads.

Avatar of Russell FoxRussell Fox🇺🇸

Are you going directly from the local file to RDS? It would probably be faster to upload the CSV file to an S3 bucket in the same AZ, and then let it import from there.
Avatar of RichardRichard🇺🇸

ASKER

Hi Russell,
Yes, I have to check (and possibly modify) the data before sending it to the data table.  In my local implementation, I would do that record by record - which while not the fastest way was very acceptable.  Sending each record up to AWS (even on a fast connection) takes quite a bit of time.  I'm looking for some automated method to replace the original method.  I could use SSMS and import the table data (which is my fall-back) if I can't find another better way.  I'm not sure how to automate importing from an S3 bucket.
ASKER CERTIFIED SOLUTION
Avatar of Russell FoxRussell Fox🇺🇸

Link to home
membership
Log in or create a free account to see answer.
Signing up is free and takes 30 seconds. No credit card required.
Create Account
Avatar of RichardRichard🇺🇸

ASKER

Hi Russell,
Until AWS supports Bulk Insert, I'm going to use some of your suggestion, create a data file (excel) and use SSMS to import to the RDS instance.  Generally we only have to import every week or so, so the manual stem shouldn't be to much of a burden.
Thanks for the response.
Reward 1Reward 2Reward 3Reward 4Reward 5Reward 6

EARN REWARDS FOR ASKING, ANSWERING, AND MORE.

Earn free swag for participating on the platform.

Microsoft SQL Server

Microsoft SQL Server

--

Questions

--

Followers

Top Experts

Microsoft SQL Server is a suite of relational database management system (RDBMS) products providing multi-user database access functionality.SQL Server is available in multiple versions, typically identified by release year, and versions are subdivided into editions to distinguish between product functionality. Component services include integration (SSIS), reporting (SSRS), analysis (SSAS), data quality, master data, T-SQL and performance tuning.