July 4, 2008 01:05pm pdt

1. chapmandew 101,510
2. acperkins 44,104
3. nmcdermaid 38,814
4. cs97jjm3 30,643
5. brejk 25,500
6. PedroCGD 25,205
7. angelIII 21,210
8. frankytee 19,941
9. dready 14,500
10. bhess1 13,700
11. Zberteoc 13,050
12. mark_wills 13,025
13. matthews... 12,250
14. JimFive 8,375
15. MohammedU 8,025

1. acperkins 312,353
2. angelIII 161,626
3. nmcdermaid 151,531
4. arbert 149,684
5. chapmandew 101,510
6. EugeneZ 57,168
7. matthews... 50,304
8. aneeshat... 41,799
9. ptjcb 38,638
10. cs97jjm3 37,173
11. rboyd56 35,425
12. bhess1 33,280
13. Sirees 27,902
14. twoboats 27,515
15. jltoops 27,275

10.23.2007 at 11:11AM PDT, ID: 22912598

	[x] Attachment Details

[x]

The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

The Grade of the Solution
The Zone Rank of the Expert Providing the Solution
The Number of Author and Expert Comments
The Number of Experts Contributing
The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

8.5

SQL Server DTS importing multiple CSV files.

Zones: MS SQL DTS, MS SQL Server

Tags: dts, sql, csv, server, files

We are using: SQL Server 2000 DTS....

I am trying to import 3 csv files into a table, each spreadsheet contains that years invoices. So there is 2007revenue, 2008revenue.csv, 2009revenue.csv

Each CSV file contains:
- projectID: primary key
- jan_revenue
- feb_revenue
- mar_revenue
and so on.

The table contains:
- projectID: primary key
- yr07_jan
- yr07_feb
- yr07_mar
....
- yr08_jan
- yr08_feb
- yr08_mar

I am able to import the initial CSV file without a problem. But:

1) How do I use multiple connections? I tried to put them on the screen but then they won't attach to the flow, they generate an error.

2) The 2nd and 3rd connections are updates to the projectID line as opposed to insert. I'm not sure how to do an update in a DTS package.

Thanks in advance!

Start your free trial to view this solution

Question Stats

Zone: Microsoft

Question Asked By: prgMan

Solution Provided By: dbbishop

Participating Experts: 1

Solution Grade: B

Translate:

Loading Advertisement...

Microsoft

Internet Protocols
Applications

Development
OS

Hardware
Windows Security

Apple

Operating Systems
Hardware

Programming
Networking

Software

Internet

Search Engines
File Sharing
WebTrends / Stats
Spy / Ad Blockers

Web Browsers
New Net Users
Web Development
Chat / IM

Anti Spam
Web Servers
Anti-Virus
Email Clients

Gamers

Tips
Online / MMORPG
Puzzle
Emulators

Action / Adventure
Role Playing
Consoles
Game Programming

Strategy
Sports
Misc
Computer Games

Digital Living

Hardware
New Net Users
New Users

Software
Digital Music
Gaming World

Home Security
Apple
Networking Hardware

Virus & Spyware

Vulnerabilities
IDS
Encryption
Anti-Virus

Operating Systems Security
Software Firewalls
WebApplications
Cell Phones

Operating Systems
Internet
Hardware Firewalls

Hardware

Handhelds / PDAs
Displays / Monitors
Components
Networking Hardware

Peripherals
Laptops/Notebooks
Storage
Servers

Desktops
New Users
Misc
Apple

Software

System Utilities
Industry Specific
Network Management
Photos / Graphics
Page Layout
VMWare
Misc
Web Development

OS
CYGWIN
Voice Recognition
Message Queue
Quality Assurance
Security
Firewalls
MultiMedia Applications

Development
Database
Office / Productivity
Business Management
OS/2 Apps
Server Software
Internet / Email

ITPro

OS
Storage
Encryption
Operating Systems Security
Apple Hardware
Laptops & Notebooks
Servers
Networking Hardware
Peripherals

Devices
Displays / Monitors
WebTrends / Stats
Search Engines
Firewalls
WebApplications
IDS
Vulnerabilities
Email Clients

File Sharing
Spy / Ad Blockers
Web Browsers
Web Servers
Networking
Anti-Virus
Chat / IM
Anti Spam

Developer

Web Servers
Web Browsers
Game Programming
Dev Tools
Industry Specific
Office / Productivity

Database
CYGWIN
Web Development
Search Engines
File Sharing
WebTrends / Stats

Programming
Content Management
Application Servers
Protocols

Storage

Removable Backup Media
Storage Technology
Servers

Grid
Remote Access
Backup / Restore

Misc
Hard Drives

Miscellaneous
Security
Development
Linux
VMWare

MainFrame OS
Unix
Apple
OS / 2
AS / 400

BeOS
Microsoft
VMS / OpenVMS

Database

Oracle
Miscellaneous
MySQL
Software
Sybase
Contact Management
PostgreSQL
Data Manipulation
Clarion
InterSystems Cache

Siebel
MUMPS
OLAP
SQLBase
SAS
GIS & GPS
4GL
Berkeley DB
DB2
Informix

Interbase / Firebird
FoxPro
Reporting
LDAP
Filemaker Pro
MS SQL Server
dBase
MS Access

Security

Misc
Web Browsers
Software Firewalls
Operating Systems Security
File Sharing

Spy / Ad Blockers
Vulnerabilities
WebApplications
IDS
Anti-Virus

Encryption
Anti Spam
Email Clients
VPN
Chat / IM

Programming

Editors IDEs
Installation
Handhelds / PDAs
Multimedia Programming
System / Kernel

Algorithms
Game
Signal Processing
Project Management
Open Source

Database
Misc
Languages
Processor Platforms
Theory

Web Development

Scripting
Blogs
Web Servers
Software
Search Engines
Web Graphics
Images

Internet Marketing
Images and Photos
Components
Document Imaging
Web Languages/Standards
Illustration
WebApplications

Fonts
WebTrends / Stats
Authoring
Digital Camera Software
Miscellaneous

Networking

Protocols
Apple Networking
Network Management
Message Queue
Application Servers
Content Management
File Servers
Email Servers
Misc
Java Editors & IDEs

Wireless
Networking Hardware
Backup / Restore
System Utilities
ISPs & Hosting
Web Servers
Storage Technology
Removable Backup Media
Servers
Broadband

Grid
OS / 2
Novell Netware
Unix Networking
Windows Networking
Security
Telecommunications
Operating Systems
Linux Networking

Other

Community Advisor
Lounge
Community Support
New Net Users

Philosophy / Religion
Math / Science
Miscellaneous
URLs

Expert Lounge
Politics
Puzzles / Riddles

Community Support

Suggestions
New to EE
New Topics
Community Advisor

CleanUp
Announcements
General

Feedback
Input
EE Bugs

10.23.2007 at 11:40AM PDT, ID: 20133339

Rank: Wizard

dbbishop:

Updating through DTS is a slow and tedious process. You have to use an ActiveX script, open another ADO connection, and update on a record-by-record basis. Depending on the amount of data, it can be extremely slow, because the update is not done as a set of data but individually.

You would not want three asynchrounous connections in this scenario. You would not want to start your update until your initial insert was completed, otherwise you could pull an update record before the initial record was inserted. Once all the data was inserted, you could run your 2nd and third processes simultaneously, as long as you were not trying to update the same columns (wouldn't make much sense).

10.23.2007 at 11:52AM PDT, ID: 20133435

prgMan:

It is not a lot of data but I can see your point.

What would be the best way to handle this? I could import each csv file and then do an update/append as necessary. Is there a way to do this without creating these excess temp tables?

10.23.2007 at 12:04PM PDT, ID: 20133539

Rank: Wizard

dbbishop:

Like I said, the only real way to update a record through DTS is by using an ActiveX script. You would create a connection within the script itself (as opposed to being a connection you would create through one of the DTS connection objects) and communicate to the server through the connection object in the script. Another alternative is to stage the data to a 'staging' table in SQL Server and then do the update from there.

Your staging table could have a status column that defaults to 0. After importing data to it, you run an Execute SQL Task to update your main table from the staging data where the status=0. You can then set the satus to 1.

Alternatively, forget the status column, import the data, update your main table, then truncate data from the staging table. Thus the data only lives in the staging table for the life of the DTS package's executuion.

10.23.2007 at 12:11PM PDT, ID: 20133595

Rank: Wizard

dbbishop:

Your statement, "... without creating these excess temp tables" I do not see where you are creating temp tables. At worst, you'd have the one staging table. It would not be a temp table, but one that existed in the database. This would be a LOT faster scenario than trying to to the update through a DTS ActiveX script.
Once you've loaded data from your first CSV file, you could load the staging table with parallel connections.

BTW, is there something in the CSV file that identifies which year the data pertains to? Are there multiple year columns in the SQL Server table? I would definately question the design of the table. It does not appear to be well designed.

10.23.2007 at 12:37PM PDT, ID: 20133840

prgMan:

I guess what I am asking is what is the alternative to DTS if it is not the best solution.

As to temp table I am assuming that I'll need to import the CSV files somewhere and then manipulate them, a temp table, or derived table?

As to the design of the table it is a requirement to be designed to the customers specs, right or wrong. It is what it is.

10.23.2007 at 12:50PM PDT, ID: 20133975

Rank: Wizard

dbbishop:

:-) Customer is always right, even when they are wrong!

From what you are describing, I think DTS is the best solution. How you implement DTS seems to be the question. You will not be able to use a temp or derived table because they belong to the session and each DTS task is a spearate session, so creating a temp table in a data transformation task would not be available to an Execute SQL task. It would have to be a 'real' physical table. I have several different import processes where I import the data into a staging table in one step and then process it into the final table(s) in another.

10.24.2007 at 05:23AM PDT, ID: 20138093

prgMan:

Could you give me an example or walk through the staging table. Also are you aware of any links to the ActiveX you were referring to. I was doing some reading last night on it and it sounds interesting but the book I was reading kinda of glossed over the details saying it was cool for more custom work but the examples were not very helpful.

Thanks!

10.24.2007 at 10:00AM PDT, ID: 20140740

Rank: Wizard

dbbishop:

The staging table would just be a table with the same columns to match your import file. Add a status column that is just a bit, that defaults to 0. You will not import data into that column, but it will be set to a value of 0 when you insert rows into the table. You can add an indentity column as a PK, but it is not required.

After importing data into it, you can then run DML to insert and/or update your main table. I run two queries, one does an UPDATE based on data that is already in the working table, then an INSERT to add new rows.

Sample below, assume [col1] is the 'primary' key:

UPDATE      myTable
SET      col1 = stg.col1,
      col2 = stg.col2,
      ...
FROM      myTable t1
INNER JOIN myStagingTable stg ON t1.KeyColumn = stg.KeyColumn
WHERE      stg.Status = 0

then:

INSERT INTO      myTable
(
      col1,
      col2,
      ...
)
SELECT      stg.col1,
      stg.col2,
      ...
FROM      myStagingTable stg
LEFT OUTER JOIN
      myTable t1 ON stg.col1 = t1.col1
WHERE      stg.Status = 0
AND      t1.col1 IS NULL

UPDATE myStagingTable SET Status=1 WHERE Status=0

Regarding using ActiveX, it is just VB Script. You always start in Function Main(). Check out the site http://www.sqldts.com for some info/examples.

Accepted Solution

10.25.2007 at 05:18AM PDT, ID: 20146814

prgMan:

Question, I realize a connection is exactly that, so I have connCVS, do I have to dump the data into a table to use it? I've tried to just connect to and apply SQL to it via Execute SQL task but it generates an error.

20080236-EE-VQP-29 / EE_QW_1_20070628