Solved

MS SQL SSIS xml source task

Posted on 2014-07-23
9
300 Views
Last Modified: 2016-02-11
I am importing xml multiple times a day from a vendor. However when SSIS created the ID's for nested XML data it is not unique.   So importing the first time and I get 3-4 records it looks fine. However subsequent imports all use the same ID's so it isn't unique, how do I go about changing this as I cant find anything about it.
0
Comment
Question by:CaptainGiblets
  • 4
  • 2
9 Comments
 
LVL 4

Expert Comment

by:Nazermohideeen
ID: 40214749
From what I understand, I think you might be able to add a Derived column and add a GUID unique ID, which will be unique all the time.
0
 
LVL 6

Author Comment

by:CaptainGiblets
ID: 40214758
But that would only work for the rows that go into each table. The way SSIS does it it links all the data together using more than 1 primary key and links them all together. If i added a derived column with a GUID it wouldnt work as some columns that are auto created have the same value 4-5 times to link them to certain records etc.
0
 
LVL 4

Expert Comment

by:Nazermohideeen
ID: 40214812
Need more information about your SSIS. May be a screenshot or something.
0
What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

 
LVL 6

Author Comment

by:CaptainGiblets
ID: 40214832
Will post back tomorrow when in office but it creates 8 tables which all use some sort of unique id and sometimes a non unique. For example I have a questionanswerid which then l
0
 
LVL 6

Author Comment

by:CaptainGiblets
ID: 40219007
Sorry was side tracked.

here is an example of the XML file
C--New-folder-campaneo-Download.xml
0
 
LVL 6

Author Comment

by:CaptainGiblets
ID: 40226684
Still need help with this if anyone has any ideas.
0
 
LVL 37

Accepted Solution

by:
ValentinoV earned 500 total points
ID: 40238329
I have the impression you're currently adding the data from the files to the same tables that already contain data from the previous files, without any staging phase.  Is that right?

To avoid your issue you should work with staging tables.  For each file import they need to get truncated.  That way you import an XML file into empty destination tables and you won't face that duplicate ID issue. Once the data is in the staging tables you can use a second Data Flow transform to extract it from there and move it into the final tables, which might be modeled a bit different, depending on your requirements.
0

Featured Post

VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
powershell code to list items in dropdown 6 57
get the results from a fuction is ssis 5 19
export sql results to csv 6 34
XSLT Display Label Name 1 22
For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
I have a large data set and a SSIS package. How can I load this file in multi threading?
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.
Via a live example, show how to shrink a transaction log file down to a reasonable size.

786 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question