Solved

Loading 300MB XML file using dbms_xmlsave.insertxml hangs or fails Oracle 9.2 AIX

Posted on 2004-08-18
5
1,800 Views
Last Modified: 2011-09-20
I'm using a procedure to load some slightly large XML files (well, only about 200MB-400MB) into Oracle tables. The XML files are VERY simple, one XML file per table, they all look something like this:

<ROWSET>
<ROW NUM="1">....    </ROW>
...
</ROWSET>

With smaller input file sizes, the tables load fine, but when we get over 100MB or so, the Oracle processes hang. While I'm not expecting to debug the server problems here (that's Metalink's job), in the mean time I'm looking for a workaround.

Here's the entire procedure I'm using to load the XML file into the table:

CREATE OR REPLACE procedure insertXML(xmlDoc IN VARCHAR2, tableName IN VARCHAR2) is       
-- declare
  l_clob           clob;
  l_bfile          bfile;
  l_ctx            dbms_xmlsave.ctxType;
  l_rows           number;
  l_warn             integer := 0;
  l_src_csid         number  := 0;
  l_src_offset         integer := 1;
  l_dest_offset         integer := 1;
  l_lang_context   integer := 0;
begin
  -- get clob from XML_LOAD_DIR
  l_bfile := BFileName('XML_LOAD_DIR', xmlDoc);
  dbms_lob.CreateTemporary(l_clob, cache => FALSE);
  dbms_lob.open(l_bfile, dbms_lob.lob_readonly);
  -- I hate DBMS_LOB !!!
  dbms_lob.loadCLOBFromFile(l_clob, l_bfile,
                   dbms_lob.lobmaxsize,
         l_dest_offset, l_src_offset,
         l_src_csid, l_lang_context,
         l_warn);
  l_ctx := dbms_xmlsave.newContext(tableName);
  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);
  dbms_xmlsave.closeContext(l_ctx);
  dbms_output.put_line('Table: ' || tableName ||
      ' Input file size ' ||
      dbms_lob.getLength(l_bfile) || ';  ' ||
      l_rows || ' rows inserted.');
  dbms_lob.close(l_bfile);
end;

When I debug, I know that the CLOB gets created correctly. It's hanging on

  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);

Again, I don't need debug on this, we have DBAs here working this issue with Metalink.

So my question is, for a workaround, I'm looking for some alternatives to load the table from the CLOB or directly from the XML file, maybe in smaller chunks, but still using the XML package. (I'm still almost a newbie at this).

Suggestions appreciated.

Thanks.
BB
0
Comment
Question by:rjbryla
  • 2
5 Comments
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 11831391
I would try to split the file into lower chunks (of course respecting the xml structure with <ROWSET> ... </ROWSET> )...
if the xml is written in the file with crlf after each </ROW>, this would be quite a simple task from the logic (of course some pages of pl/sql code)...
--> read the file, write each row into output file, after n rows , create 2nd output file etc

maybe you might want to merge the 2 procs, so when a output file has been written directly load it, but I wouldn't recommend this (because of the errorhandling & resuming mechanisms...)

CHeers
0
 

Author Comment

by:rjbryla
ID: 11834357
We thought of splitting up, but we would have to introduce two new steps or sets of steps -- splitting the file when we get it (we have no control over the source), then changing our load script to call this procedure "n" times... we want to if at all possible read the whole chunk at once.

Oracle 9.2 seems to load the CLOB fine, but just stumbles when trying to convert the XML.
0
 
LVL 143

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 500 total points
ID: 11838332
I also had problems with XML loading large files (no problem writing it), using MSXML (in VB). I had finally written my own procedure to read and write it (also because of other issues)...
You should take the time to put the procedure in place, because you won't know when the "bugfix" will be released (if at all)...
Cheers
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
How to join on ID, with prefix? 15 69
VB.Net SQL Query 2 Tables Different Databases 3 43
MS SQL GROUP BY 6 72
populate value based on what is selected in lov 2 27
When table data gets too large to manage or queries take too long to execute the solution is often to buy bigger hardware or assign more CPUs and memory resources to the machine to solve the problem. However, the best, cheapest and most effective so…
Azure Functions is a solution for easily running small pieces of code, or "functions," in the cloud. This article shows how to create one of these functions to write directly to Azure Table Storage.
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

713 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question