Solved

SSIS help: Process multiple files

Posted on 2008-10-06
2
526 Views
Last Modified: 2013-11-30
Greetings Experts,

I have posted this question before but onobody has responded in a few days and I need to resolve ASAP.  You can see the original post here...

http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/SSIS/Q_23787635.html

I have this SSIS package that takes an Excel file (enrollment.xls) from a server share and imports it into a SQL database, then it deletes it from the server share.  The idea is that someone will create this enrollment-STUDYNAME.xls file and place it on the server share to be processed.  However, the problem with this approach is that there can only be one file in that location at any given time.  How do I go about programming the SSIS package to look for every file in the folder and process each?  The file does not need to be called enrollment.xls.... it can enrollment1.xls, enrollmentB.xls.   This way people can pop as many files in there to be processed and I can then schedule this SSIS package to run at night.

0
Comment
Question by:Pharmica
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 1

Expert Comment

by:MaxLyd
ID: 22655192
Interesting question.  This isn't a perfect solution, but it's what I would do...

Have people save the files on the share drive during the day and have some sort of cut-off when people aren't supposed to save any new files (more of an operations  issue).
 
Then write a small batch file to copy all the .xls (or whatever they are) files to the same drive under a uniform name which can be overwritten each night.  (code snippet)  You can easily schedule this within windows scheduler to run at whatever time you want.  You just need to make sure you have the correct permissions.
You can then have a linked server setup using that file through SQL server management studio.  This will allow you to query that file like a table. Not sure if that is what you had set up already with the existing import...  I find it to be the most versatile way to import data because it allows for a lot of data manipulation prior to insertin the data into tables.  Kind of takes the necessity of temp tables out of the equation a lot of times.
The only thing you will have to be concious of is that the fields will have to be EXACTLY the same in the files that are to be combined.  This is because the batch file with merely mush the data together (yup that means field names and all).  So you will have to take that into account when you are setting up the SSIS package to exclude those rows of data.

COPY \\somedrivename\*.xls "\\somedrivename\Linkedspreadsheet.xls" /Y 

Open in new window

0
 

Accepted Solution

by:
Pharmica earned 0 total points
ID: 22668714
MaxLyd:  Thanks for responding, but I was looking for a way to identify each file using a variable, so I can further use that variable to perform other functions on the file (ie, copy it, delete it, move it...etc).  I managed to get this done by using a Foreach loop sequence in SQL BI studio.    Here is the post that solved the problem.

http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/SSIS/Q_23787635.html

0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction In my previous article (http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/SSIS/A_9150-Loading-XML-Using-SSIS.html) I showed you how the XML Source component can be used to load XML files into a SQL Server database, us…
The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question