Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1862
  • Last Modified:

Loading 300MB XML file using dbms_xmlsave.insertxml hangs or fails Oracle 9.2 AIX

I'm using a procedure to load some slightly large XML files (well, only about 200MB-400MB) into Oracle tables. The XML files are VERY simple, one XML file per table, they all look something like this:

<ROWSET>
<ROW NUM="1">....    </ROW>
...
</ROWSET>

With smaller input file sizes, the tables load fine, but when we get over 100MB or so, the Oracle processes hang. While I'm not expecting to debug the server problems here (that's Metalink's job), in the mean time I'm looking for a workaround.

Here's the entire procedure I'm using to load the XML file into the table:

CREATE OR REPLACE procedure insertXML(xmlDoc IN VARCHAR2, tableName IN VARCHAR2) is       
-- declare
  l_clob           clob;
  l_bfile          bfile;
  l_ctx            dbms_xmlsave.ctxType;
  l_rows           number;
  l_warn             integer := 0;
  l_src_csid         number  := 0;
  l_src_offset         integer := 1;
  l_dest_offset         integer := 1;
  l_lang_context   integer := 0;
begin
  -- get clob from XML_LOAD_DIR
  l_bfile := BFileName('XML_LOAD_DIR', xmlDoc);
  dbms_lob.CreateTemporary(l_clob, cache => FALSE);
  dbms_lob.open(l_bfile, dbms_lob.lob_readonly);
  -- I hate DBMS_LOB !!!
  dbms_lob.loadCLOBFromFile(l_clob, l_bfile,
                   dbms_lob.lobmaxsize,
         l_dest_offset, l_src_offset,
         l_src_csid, l_lang_context,
         l_warn);
  l_ctx := dbms_xmlsave.newContext(tableName);
  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);
  dbms_xmlsave.closeContext(l_ctx);
  dbms_output.put_line('Table: ' || tableName ||
      ' Input file size ' ||
      dbms_lob.getLength(l_bfile) || ';  ' ||
      l_rows || ' rows inserted.');
  dbms_lob.close(l_bfile);
end;

When I debug, I know that the CLOB gets created correctly. It's hanging on

  l_rows := dbms_xmlsave.insertxml(l_ctx,l_clob);

Again, I don't need debug on this, we have DBAs here working this issue with Metalink.

So my question is, for a workaround, I'm looking for some alternatives to load the table from the CLOB or directly from the XML file, maybe in smaller chunks, but still using the XML package. (I'm still almost a newbie at this).

Suggestions appreciated.

Thanks.
BB
0
rjbryla
Asked:
rjbryla
  • 2
1 Solution
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
I would try to split the file into lower chunks (of course respecting the xml structure with <ROWSET> ... </ROWSET> )...
if the xml is written in the file with crlf after each </ROW>, this would be quite a simple task from the logic (of course some pages of pl/sql code)...
--> read the file, write each row into output file, after n rows , create 2nd output file etc

maybe you might want to merge the 2 procs, so when a output file has been written directly load it, but I wouldn't recommend this (because of the errorhandling & resuming mechanisms...)

CHeers
0
 
rjbrylaAuthor Commented:
We thought of splitting up, but we would have to introduce two new steps or sets of steps -- splitting the file when we get it (we have no control over the source), then changing our load script to call this procedure "n" times... we want to if at all possible read the whole chunk at once.

Oracle 9.2 seems to load the CLOB fine, but just stumbles when trying to convert the XML.
0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
I also had problems with XML loading large files (no problem writing it), using MSXML (in VB). I had finally written my own procedure to read and write it (also because of other issues)...
You should take the time to put the procedure in place, because you won't know when the "bugfix" will be released (if at all)...
Cheers
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now