Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Loading 300MB XML file using dbms_xmlsave.insertxml hangs or fails Oracle 9.2 AIX

Posted on 2004-08-18
5
Medium Priority
?
1,846 Views
Last Modified: 2011-09-20
I'm using a procedure to load some slightly large XML files (well, only about 200MB-400MB) into Oracle tables. The XML files are VERY simple, one XML file per table, they all look something like this:

<ROWSET>
<ROW NUM="1">....    </ROW>
...
</ROWSET>

With smaller input file sizes, the tables load fine, but when we get over 100MB or so, the Oracle processes hang. While I'm not expecting to debug the server problems here (that's Metalink's job), in the mean time I'm looking for a workaround.

Here's the entire procedure I'm using to load the XML file into the table:

CREATE OR REPLACE procedure insertXML(xmlDoc IN VARCHAR2, tableName IN VARCHAR2) is       
-- declare
  l_clob           clob;
  l_bfile          bfile;
  l_ctx            dbms_xmlsave.ctxType;
  l_rows           number;
  l_warn             integer := 0;
  l_src_csid         number  := 0;
  l_src_offset         integer := 1;
  l_dest_offset         integer := 1;
  l_lang_context   integer := 0;
begin
  -- get clob from XML_LOAD_DIR
  l_bfile := BFileName('XML_LOAD_DIR', xmlDoc);
  dbms_lob.CreateTemporary(l_clob, cache => FALSE);
  dbms_lob.open(l_bfile, dbms_lob.lob_readonly);
  -- I hate DBMS_LOB !!!
  dbms_lob.loadCLOBFromFile(l_clob, l_bfile,
                   dbms_lob.lobmaxsize,
         l_dest_offset, l_src_offset,
         l_src_csid, l_lang_context,
         l_warn);
  l_ctx := dbms_xmlsave.newContext(tableName);
  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);
  dbms_xmlsave.closeContext(l_ctx);
  dbms_output.put_line('Table: ' || tableName ||
      ' Input file size ' ||
      dbms_lob.getLength(l_bfile) || ';  ' ||
      l_rows || ' rows inserted.');
  dbms_lob.close(l_bfile);
end;

When I debug, I know that the CLOB gets created correctly. It's hanging on

  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);

Again, I don't need debug on this, we have DBAs here working this issue with Metalink.

So my question is, for a workaround, I'm looking for some alternatives to load the table from the CLOB or directly from the XML file, maybe in smaller chunks, but still using the XML package. (I'm still almost a newbie at this).

Suggestions appreciated.

Thanks.
BB
0
Comment
Question by:rjbryla
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
5 Comments
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 11831391
I would try to split the file into lower chunks (of course respecting the xml structure with <ROWSET> ... </ROWSET> )...
if the xml is written in the file with crlf after each </ROW>, this would be quite a simple task from the logic (of course some pages of pl/sql code)...
--> read the file, write each row into output file, after n rows , create 2nd output file etc

maybe you might want to merge the 2 procs, so when a output file has been written directly load it, but I wouldn't recommend this (because of the errorhandling & resuming mechanisms...)

CHeers
0
 

Author Comment

by:rjbryla
ID: 11834357
We thought of splitting up, but we would have to introduce two new steps or sets of steps -- splitting the file when we get it (we have no control over the source), then changing our load script to call this procedure "n" times... we want to if at all possible read the whole chunk at once.

Oracle 9.2 seems to load the CLOB fine, but just stumbles when trying to convert the XML.
0
 
LVL 143

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 2000 total points
ID: 11838332
I also had problems with XML loading large files (no problem writing it), using MSXML (in VB). I had finally written my own procedure to read and write it (also because of other issues)...
You should take the time to put the procedure in place, because you won't know when the "bugfix" will be released (if at all)...
Cheers
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Azure Functions is a solution for easily running small pieces of code, or "functions," in the cloud. This article shows how to create one of these functions to write directly to Azure Table Storage.
Recently I was talking with Tim Sharp, one of my colleagues from our Technical Account Manager team about MongoDB’s scalability. While doing some quick training with some of the Percona team, Tim brought something to my attention...
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question