Solved

Loading 300MB XML file using dbms_xmlsave.insertxml hangs or fails Oracle 9.2 AIX

Posted on 2004-08-18
5
1,827 Views
Last Modified: 2011-09-20
I'm using a procedure to load some slightly large XML files (well, only about 200MB-400MB) into Oracle tables. The XML files are VERY simple, one XML file per table, they all look something like this:

<ROWSET>
<ROW NUM="1">....    </ROW>
...
</ROWSET>

With smaller input file sizes, the tables load fine, but when we get over 100MB or so, the Oracle processes hang. While I'm not expecting to debug the server problems here (that's Metalink's job), in the mean time I'm looking for a workaround.

Here's the entire procedure I'm using to load the XML file into the table:

CREATE OR REPLACE procedure insertXML(xmlDoc IN VARCHAR2, tableName IN VARCHAR2) is       
-- declare
  l_clob           clob;
  l_bfile          bfile;
  l_ctx            dbms_xmlsave.ctxType;
  l_rows           number;
  l_warn             integer := 0;
  l_src_csid         number  := 0;
  l_src_offset         integer := 1;
  l_dest_offset         integer := 1;
  l_lang_context   integer := 0;
begin
  -- get clob from XML_LOAD_DIR
  l_bfile := BFileName('XML_LOAD_DIR', xmlDoc);
  dbms_lob.CreateTemporary(l_clob, cache => FALSE);
  dbms_lob.open(l_bfile, dbms_lob.lob_readonly);
  -- I hate DBMS_LOB !!!
  dbms_lob.loadCLOBFromFile(l_clob, l_bfile,
                   dbms_lob.lobmaxsize,
         l_dest_offset, l_src_offset,
         l_src_csid, l_lang_context,
         l_warn);
  l_ctx := dbms_xmlsave.newContext(tableName);
  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);
  dbms_xmlsave.closeContext(l_ctx);
  dbms_output.put_line('Table: ' || tableName ||
      ' Input file size ' ||
      dbms_lob.getLength(l_bfile) || ';  ' ||
      l_rows || ' rows inserted.');
  dbms_lob.close(l_bfile);
end;

When I debug, I know that the CLOB gets created correctly. It's hanging on

  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);

Again, I don't need debug on this, we have DBAs here working this issue with Metalink.

So my question is, for a workaround, I'm looking for some alternatives to load the table from the CLOB or directly from the XML file, maybe in smaller chunks, but still using the XML package. (I'm still almost a newbie at this).

Suggestions appreciated.

Thanks.
BB
0
Comment
Question by:rjbryla
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
5 Comments
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 11831391
I would try to split the file into lower chunks (of course respecting the xml structure with <ROWSET> ... </ROWSET> )...
if the xml is written in the file with crlf after each </ROW>, this would be quite a simple task from the logic (of course some pages of pl/sql code)...
--> read the file, write each row into output file, after n rows , create 2nd output file etc

maybe you might want to merge the 2 procs, so when a output file has been written directly load it, but I wouldn't recommend this (because of the errorhandling & resuming mechanisms...)

CHeers
0
 

Author Comment

by:rjbryla
ID: 11834357
We thought of splitting up, but we would have to introduce two new steps or sets of steps -- splitting the file when we get it (we have no control over the source), then changing our load script to call this procedure "n" times... we want to if at all possible read the whole chunk at once.

Oracle 9.2 seems to load the CLOB fine, but just stumbles when trying to convert the XML.
0
 
LVL 143

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 500 total points
ID: 11838332
I also had problems with XML loading large files (no problem writing it), using MSXML (in VB). I had finally written my own procedure to read and write it (also because of other issues)...
You should take the time to put the procedure in place, because you won't know when the "bugfix" will be released (if at all)...
Cheers
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This post looks at MongoDB and MySQL, and covers high-level MongoDB strengths, weaknesses, features, and uses from the perspective of an SQL user.
Your data is at risk. Probably more today that at any other time in history. There are simply more people with more access to the Web with bad intentions.
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…
This is a high-level webinar that covers the history of enterprise open source database use. It addresses both the advantages companies see in using open source database technologies, as well as the fears and reservations they might have. In this…

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question