We have 2 sets of CSV flat files (37 columns and 15 columns). From the user perspective, these 2 type of files will treated as 1 file type. Then in the system somehow to need to cater this as seperate events.
So what we have done:
1. Connetion MAnager (flat file), we created 37 columns initially to cater this.
2. Some with Flat File Source as well OLE DB Destination with 37 columns.
What we found that if the data was loaded with 37 columns then this thing is ok BUT if we loaded with 15 columns data then the database fill in with almost 1/3 of this data due the data try to fill in column 16-37. So for instance we knew that from the original data that it has 12000 records but now in the datasbe almost 4000 records and we check the column 16-37 and there is a data in there.
The question are:
1. How to force only upload what ever in the original file is.
2. If there is no way to do this, how to detech the number of columns before proceeding this step.
I am appreciated your comment.