I am looking for a method using DTS to create N export text files of 50,000 rows or less from a growing dataset (there are other requirements that make DTS the correct way to start processing this at least). Here is what I have to work with:
1) A query I can use to determine how many total rows there are to export.
2) A query that extracts data from the total rows in 50,000 row chunks.
a) I will need to keep track of where the previous step's endpoint was (possibly a global variable)
3) A CSV text export connection
a) The name of the exported text file needs to change with each iteration, i.e. the second iteration in January, 2007 would be MyFile-200701-2.csv
What I am hoping to do is to perform all of my initialization steps, then loop through the export step until I have exported all of the rows, then clean up and exit. Unfortunately, I am not *that* good with DTS. Is there anyone who would care to flesh out a framework I could use to do this?
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Actually, quite a lot. What the review in excel is for is (1) spot checking data to ensure validity, and (2) reviewing specific rows when the third-party that processes the data reports an issue.
The output format is modified on a fairly regular basis (about once per two runs recently, as requirements change), and so the consumer wants to be able to review and validate data on a spot check basis. Subdividing the information into files that can be manipulated in Excel allows them to use an application that they are familiar with. This splitting was not an original part of the specification (I added this knowing they wanted to review the data - they were going to try and evaluate the CSV data for the full dump in Word), but has become essential to the consumer since implemented. Thus, the requirement is to split files allowing validation by eye.
acperkins - I am going with one of your solutions (the second link or the full export + split in AX). It's an unusual requirement, but 3rd party consumer requirements combined with local consumer requirements end up with this export method as the most successful.
Business Accounts
Answer for Membership
by: acperkinsPosted on 2007-01-09 at 19:22:44ID: 18281286
Looping in DTS is not trivial. For a start, take a look at these examples in DTS:
How to loop through a global variable Rowset
http://www.sqldts.com/298.
Looping, Importing and Archiving
http://www.sqldts.com/246.
Despite first appearances the second is closer to what you are asking for.
Having said that, I would approach it in an entirely different (and IMHO more efficient) way. I would export all the rows and then use an ActiveX Script Task to do the work of splitting the file into multiple files.