Solved

how to import xml from a website, parse it and save the results into mssql 2k8 r2

Posted on 2013-01-25
8
352 Views
Last Modified: 2013-01-27
OK...

I have an external XML web site (http://rss.timegenie.com/forex2.xml) that I need to import into MSSQL Server 2008 R2.

1. is this even possible?

2. how would this be done?

Thanks,
John
0
Comment
Question by:jschultz0614
8 Comments
 

Author Comment

by:jschultz0614
Comment Utility
I tried the example found at http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_26631018.html with no luck, it only returns an empty rowset.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
Here are some good examples
http://stackoverflow.com/questions/3628846/insert-xml-into-sql-server-2008-database

Get the insert working outside of your site, them get a simple insert working, and only then put it all together. Too many things can be a problem when doing it all at once.
0
 
LVL 51

Accepted Solution

by:
Mark Wills earned 500 total points
Comment Utility
Well, can do it all within a single SQL Script, but, need to use xp_cmdshell

You can turn it on at the start and turn it off at the end if needed. I have included turning on the option which allows xp_cmdshell.

Anyway, check out below, I have tried it and it works...

You just need to change the physical file path and server names (etc)

-- To allow changes to XP_CMDSHELL advanced options to be available.
EXEC sp_configure 'show advanced options', 1
GO
-- To update the currently configured value for advanced options.
RECONFIGURE
GO
-- To enable the feature.
EXEC sp_configure 'xp_cmdshell', 1
GO
-- To update the currently configured value for this feature.
RECONFIGURE
GO
-- if needed, turn off the advanced options again


-- Now, lets create a script within the database then we can export that and use that script to grab the page and then load the XML

if object_id('js_mydownload_script','U') is not null drop table js_mydownload_script
create table js_mydownload_script (line varchar(255))

insert js_mydownload_script values ('var fso = new ActiveXObject(''Scripting.FileSystemObject'');')
insert js_mydownload_script values ('var xmlhttp=new ActiveXObject("microsoft.xmlhttp");')
insert js_mydownload_script values ('xmlhttp.open("GET", "http://rss.timegenie.com/forex2.xml", false);')
insert js_mydownload_script values ('xmlhttp.send();')
insert js_mydownload_script values ('var data=xmlhttp.responsetext;')
insert js_mydownload_script values ('var file = fso.CreateTextFile("c:\\ee\\forex2.xml",true);')  -------- change file path
insert js_mydownload_script values ('file.writeLine(data);')
insert js_mydownload_script values ('file.close();')

-- right, now export that script -- remember to check and change that path.
exec xp_cmdshell 'bcp "select * from ee_test_db..js_mydownload_script" queryout "c:\ee\js_my_download_script.js" -SMY-SERVER\SQL2012 -c -T -CACP'

-- and then execute the script
exec xp_cmdshell 'cscript.exe c:\ee\js_my_download_script.js'

-- or create a generic script file externally - just a text file edited in notepad (between the /* ... */ below) - then use that passing in the params
/*
var fso = new ActiveXObject('Scripting.FileSystemObject');
var args = WScript.Arguments;
var url = args(0);
var fileName = args(1);
var xmlhttp=new ActiveXObject("microsoft.xmlhttp");
xmlhttp.open("GET", url, false);
xmlhttp.send();
var data=xmlhttp.responsetext;
var file = fso.CreateTextFile(fileName, 2);
file.writeLine(data);
file.close();
*/

-- then your execute sql becomes even easier :

exec xp_cmdshell 'cscript c:\ee\js_download_script.js http://rss.timegenie.com/forex2.xml c:\ee\forex2.xml'


-- now we have the XML file locally, we can then set about importing it...

-- first we will need a staging table
if object_id('tempdb..#my_xml_staging','U') is not null drop table #my_xml_staging
create table #my_xml_staging(id int identity, xml_col xml)

-- now bulk load the file into a staging table
INSERT INTO #my_xml_staging(Xml_Col)
SELECT * FROM OPENROWSET(BULK 'c:\ee\forex2.xml',SINGLE_BLOB) AS x

-- now lets see what we have
select R.C.value('@code','char(3)') as code,
       R.C.value('@description','varchar(100)') as description,
       R.C.value('@rate','decimal(12,4)') as rate
from #my_xml_staging X
cross apply X.xml_col.nodes('/forex/data/currency') as R(C)

-- and we can add the "insert my_real_table (code, description, rate)" as the first line above the select to insert into the real home.

-- If needed, turn off xp_cmdshell by using the sp_configure and setting to 0 instead of 1 as per above

Open in new window

0
 

Author Comment

by:jschultz0614
Comment Utility
getting the following error:
Msg 4860, Level 16, State 1, Line 51
Cannot bulk load. The file "c:\ee\forex2.xml" does not exist.

I created the ee directory. The script works only if I manually download the xml file myself.
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 

Author Comment

by:jschultz0614
Comment Utility
Disregard :), forgot to create the js script :D.

AWESOME SCRIPT btw.
0
 

Author Closing Comment

by:jschultz0614
Comment Utility
Awesome Script
0
 
LVL 51

Expert Comment

by:Mark Wills
Comment Utility
Thanks, I have been using it for a while now, and (so far) it stands the test of time...

Cheers,
Mark
0
 
LVL 1

Expert Comment

by:marrowyung
Comment Utility
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
PHP and Soap 3 27
Calculating Business Hours 19 62
SQL Maintenance Plan 3 13
BULK INSERT most recent CSV 19 17
Long way back, we had to take help from third party tools in order to encrypt and decrypt data.  Gradually Microsoft understood the need for this feature and started to implement it by building functionality into SQL Server. Finally, with SQL 2008, …
Browsing the questions asked to the Experts of this forum, you will be amazed to see how many times people are headaching about monster regular expressions (regex) to select that specific part of some HTML or XML file they want to extract. The examp…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now