Solved

Data Ware House - Daily CSV Extracts into Microsoft SQL 2008 R2 Server

Posted on 2014-03-07
3
217 Views
Last Modified: 2016-02-10
Hi,
I've recently been give a task to import daily change files into an Microsoft SQL Server. There are 156 files/tables that are Extracted & downloaded on a daily base that need to be bulk insert into an SQL database. What is the best way to approach this?

There are a few tools that I've been looking at, which are SSIS and BCP.
0
Comment
Question by:johnojohno
  • 2
3 Comments
 
LVL 45

Expert Comment

by:aikimark
Comment Utility
You should include Powershell as another viable tool.  Like BCP, it is a 'push' model.  SSIS would be the pull model.
0
 

Author Comment

by:johnojohno
Comment Utility
Thanks aikmark I'll check it out. I've started to use SSIS to import the data. I've created a foreach loop container for each of my table extracts, which are daily CSV files.
Unfortunately non of the data extracts have a record creation date so I need to come up with a way to understand which records are the most up to date.  

Does anyone have any ideals how I can do this and remove the duplicate records of about 117 tables on a daily bases?
0
 
LVL 45

Accepted Solution

by:
aikimark earned 500 total points
Comment Utility
Files have both a creation date and a modified date properties.  Why couldn't you use that date?
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Dynamics crm 2011 8 43
SLQ View not updating 10 46
The duplicate key value is (<NULL>) 14 43
Data to display differently-SQL Server 4 19
JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now