Solved

Loading 300MB XML file using dbms_xmlsave.insertxml hangs or fails Oracle 9.2 AIX

Posted on 2004-08-18
5
1,788 Views
Last Modified: 2011-09-20
I'm using a procedure to load some slightly large XML files (well, only about 200MB-400MB) into Oracle tables. The XML files are VERY simple, one XML file per table, they all look something like this:

<ROWSET>
<ROW NUM="1">....    </ROW>
...
</ROWSET>

With smaller input file sizes, the tables load fine, but when we get over 100MB or so, the Oracle processes hang. While I'm not expecting to debug the server problems here (that's Metalink's job), in the mean time I'm looking for a workaround.

Here's the entire procedure I'm using to load the XML file into the table:

CREATE OR REPLACE procedure insertXML(xmlDoc IN VARCHAR2, tableName IN VARCHAR2) is       
-- declare
  l_clob           clob;
  l_bfile          bfile;
  l_ctx            dbms_xmlsave.ctxType;
  l_rows           number;
  l_warn             integer := 0;
  l_src_csid         number  := 0;
  l_src_offset         integer := 1;
  l_dest_offset         integer := 1;
  l_lang_context   integer := 0;
begin
  -- get clob from XML_LOAD_DIR
  l_bfile := BFileName('XML_LOAD_DIR', xmlDoc);
  dbms_lob.CreateTemporary(l_clob, cache => FALSE);
  dbms_lob.open(l_bfile, dbms_lob.lob_readonly);
  -- I hate DBMS_LOB !!!
  dbms_lob.loadCLOBFromFile(l_clob, l_bfile,
                   dbms_lob.lobmaxsize,
         l_dest_offset, l_src_offset,
         l_src_csid, l_lang_context,
         l_warn);
  l_ctx := dbms_xmlsave.newContext(tableName);
  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);
  dbms_xmlsave.closeContext(l_ctx);
  dbms_output.put_line('Table: ' || tableName ||
      ' Input file size ' ||
      dbms_lob.getLength(l_bfile) || ';  ' ||
      l_rows || ' rows inserted.');
  dbms_lob.close(l_bfile);
end;

When I debug, I know that the CLOB gets created correctly. It's hanging on

  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);

Again, I don't need debug on this, we have DBAs here working this issue with Metalink.

So my question is, for a workaround, I'm looking for some alternatives to load the table from the CLOB or directly from the XML file, maybe in smaller chunks, but still using the XML package. (I'm still almost a newbie at this).

Suggestions appreciated.

Thanks.
BB
0
Comment
Question by:rjbryla
  • 2
5 Comments
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 11831391
I would try to split the file into lower chunks (of course respecting the xml structure with <ROWSET> ... </ROWSET> )...
if the xml is written in the file with crlf after each </ROW>, this would be quite a simple task from the logic (of course some pages of pl/sql code)...
--> read the file, write each row into output file, after n rows , create 2nd output file etc

maybe you might want to merge the 2 procs, so when a output file has been written directly load it, but I wouldn't recommend this (because of the errorhandling & resuming mechanisms...)

CHeers
0
 

Author Comment

by:rjbryla
ID: 11834357
We thought of splitting up, but we would have to introduce two new steps or sets of steps -- splitting the file when we get it (we have no control over the source), then changing our load script to call this procedure "n" times... we want to if at all possible read the whole chunk at once.

Oracle 9.2 seems to load the CLOB fine, but just stumbles when trying to convert the XML.
0
 
LVL 143

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 500 total points
ID: 11838332
I also had problems with XML loading large files (no problem writing it), using MSXML (in VB). I had finally written my own procedure to read and write it (also because of other issues)...
You should take the time to put the procedure in place, because you won't know when the "bugfix" will be released (if at all)...
Cheers
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Latency in .net app using DB in .net 21 74
Index Details in SQL Server 2012 4 57
How to load 2 images in same column in Delphi 2 57
mysql database, schema and table creation 13 57
Entering a date in Microsoft Access can be tricky. A typo can cause month and day to be shuffled, entering the day only causes an error, as does entering, say, day 31 in June. This article shows how an inputmask supported by code can help the user a…
These days, all we hear about hacktivists took down so and so websites and retrieved thousands of user’s data. One of the techniques to get unauthorized access to database is by performing SQL injection. This article is quite lengthy which gives bas…
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

789 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question