Solved

SQL SSIS  issue:  How to Process multiple files with different names

Posted on 2008-10-04
11
1,243 Views
Last Modified: 2013-11-30
Greetings Experts,

I have a 2 questions that I need help with.
First, I have this SSIS package that takes an Excel file (enrollment.xls) from a server share and imports it into a SQL database, then it deletes it from the server share.  The idea is that someone will create this enrollment.xls file and place it on the server share to be processed.  However, the problem with this approach is that there can only be one file in that location at any given time.  How do I go about programming the SSIS package to look for every file in the folder and process each?  The file does not need to be called enrollment.xls.... it can enrollment1.xls, enrollmentB.xls.   This way people can pop as many files in there to be processed and I can then schedule this SSIS package to run at night.

0
Comment
Question by:Pharmica
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
  • 2
  • +1
11 Comments
 
LVL 22

Accepted Solution

by:
PedroCGD earned 300 total points
ID: 22641932
Dear friend,
You need to use ForEach Loop to read each file type inside a specific filesystem path.
Check the example I made in other question:
http://www.experts-exchange.com/Software/Office_Productivity/Office_Suites/MS_Office/Excel/Q_23680131.html

If you have some doubts I can customize it for you!
regards!
Pedro
0
 

Author Comment

by:Pharmica
ID: 22642881
Hi PedroCGD,

I looked at your example in the other post and I sort of understand.    I get the general idea, but I am not sure how to change the data flow to do what I want it to.   I have attached the SSIS I created...be gentle this is my first complex package.

In this package it does a number of things.  This is what happens.
1.  An XLS file is imported into a staging database table.  This file is specfic to one Study so a column called Study  will contain URT456.
2. The stage table is compared to another table, and all records are deleted from the second table where Study = Study.  
3.  The stage table is copied to second table
4.  The stage table is truncated
5. The orginal file is deleted.

The problem is the orginal XLS will be called different things so my code will not work.  Also the XLS will have different studies.  I have attached 2 XLS examples.  

I think if i can get this ForEach loop working it will solve my problem.  I just cant figure out how to determine the name of the XLS file if it isnt hardcoded.
Update-Enrollment-Table-dtsx.txt
EnrollmentTable-INOT11.xls
EnrollmentTable-URT456.xls
0
 
LVL 22

Expert Comment

by:PedroCGD
ID: 22652808
Dear Pharmica,
I'll be on holidays for some days... I will try today or when return I will answer you, ok?
Regards!
Pedro
0
Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

 
LVL 30

Assisted Solution

by:nmcdermaid
nmcdermaid earned 200 total points
ID: 22655209
The for each loop returns the name of each file that it finds. That saves you having to hard code anything.
Read through this and see if it helps:
http://www.sqlis.com/55.aspx
 
.. oh and you should truncate the staging table at the very start not at the end for two reasons:
-If your package fails in the middle, it will leave data in the staging table which will confuse things for the next run
-Yuo can inspect the staging table afterwards to help debug if need be.
 
0
 

Author Comment

by:Pharmica
ID: 22656015
nmcdermaid:

Thanks for the reference.  I looked at the documentation you sent and I was able to get it working with csv files (using flat file connection), but when I try to use an excel connection I get errors.  It seems as soon as I enter the expression for the excel connection manager i get errors when i run the package.

Any ideas?
0
 
LVL 30

Expert Comment

by:nmcdermaid
ID: 22656210
How about this one:

http://msdn.microsoft.com/en-us/library/ms345182.aspx
(If this solves your problem then you should assign points to Pedro and then google.com in that order   :)  )
0
 
LVL 51

Expert Comment

by:Mark Wills
ID: 22660760
attached snippet is an interesting example, it does do the for loop and is using file names, but only the first bit is relevant - it then does a t-sql equivelent, and a dos command equivelent, so do not read too far :)

http://www.simple-talk.com/sql/sql-server-2005/executing-ssis-packages-/
0
 

Author Closing Comment

by:Pharmica
ID: 31503073
Excellent work.  I managed to get it working with both your help.  Thanks again for the tips.
0
 
LVL 51

Expert Comment

by:Mark Wills
ID: 22667723
Hi pharmica : are you still having problems ? Just saw your new posting...

http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/DTS/Q__23790956.html
0
 

Author Comment

by:Pharmica
ID: 22668660
No, I actually resolve and closed the question...i totally forgot about this one though...I will close it now.  I used a combination of all your advice so I will split it three ways.  I dont generally like opening a second question but this was something that cant wait.
0
 
LVL 51

Expert Comment

by:Mark Wills
ID: 22668712
No problems, just a bit worried that things weren't working for you...
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Load balancing is the method of dividing the total amount of work performed by one computer between two or more computers. Its aim is to get more work done in the same amount of time, ensuring that all the users get served faster.
Recently we ran in to an issue while running some SQL jobs where we were trying to process the cubes.  We got an error saying failure stating 'NT SERVICE\SQLSERVERAGENT does not have access to Analysis Services. So this is a way to automate that wit…
Familiarize people with the process of retrieving data from SQL Server using an Access pass-thru query. Microsoft Access is a very powerful client/server development tool. One of the ways that you can retrieve data from a SQL Server is by using a pa…
Via a live example, show how to extract information from SQL Server on Database, Connection and Server properties

691 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question