Solved

Optimize union insert (runaway trans log) - table size, nolock and 'nolog'?

Posted on 2006-11-16
6
532 Views
Last Modified: 2008-02-01
Hi all,

Stable SQL2000 setup on W2000. 7g db on drive (100meg trans log - Simple log mode) with another 10g free.  Working with 3 similar tables, trying to union insert them to one table. Existing 3 tables no more than 2-3g, 15mil rows.  Using stored proc like this:

INSERT INTO [CLAIMS_A_2002_04]
( CLAIM_TYPE, CL_YEAR, FPTYPE, DT_recipnum, DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3x)
SELECT 'A' AS 'CLAIM_TYPE', left([DOS1],4) as 'CL_YEAR', 'FP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT12, CPT3, CPT4x
   FROM [acg_temp].[dbo].[FTP A Claims FP 2002_04] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2002 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3x
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2002] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2003 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2003] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2004 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2004] with (nolock)
GO
SET QUOTED_IDENTIFIER OFF
GO
SET ANSI_NULLS ON
GO

Procedure seems to start off okay but never finishes: last run blew up db to 10g with trans log of 7g.  Usually ends with messge 'truncate log' or 'can't write to log' etc.  Is there anything I can add/rewrite to get these tables into one table??   Figured the 'nolock' would do it, but now out of ideas.  Is there any nolog switch etc?  I can keep everybody out of the db while q is running, and have full backup so not concerned about rollback etc.  Why is it writing so much to the logs??  really would help if this would work!

thanks!!

wes
0
Comment
Question by:new_wes
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 

Author Comment

by:new_wes
ID: 17958351
PS - Row size on empty target table is at most 200char (below), so expecting 15m rows * 200 char = 3g of data??  What am i missing that the db/log would grow by almost 3x that much and still not complete?

CREATE TABLE [CLAIMS_A_2002_04] (
      [CLAIMSAKEY] [numeric](18, 0) IDENTITY (1, 1) NOT NULL ,
      [CLAIM_TYPE] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CL_YEAR] [bigint] NULL ,
      [FPTYPE] [char] (3) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DT_recipnum] [char] (12) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DOS1] [char] (11) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DOS2] [char] (11) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PDC] [char] (6) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [SDC] [char] (6) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [TOTPAID] [char] (17) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PROVTYPE] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PTPPS] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT1] [char] (10) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT2] [char] (5) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT3x] [char] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) ON [PRIMARY]
GO
0
 
LVL 29

Accepted Solution

by:
Nightman earned 500 total points
ID: 17958398
In simple recovery model everything is written to the log - it is just flushed as soon as the transaction has been committed.

I would recommend that you perform the transaction in small batches (a few hundred thousand records at a time) and in between run BACKUP LOG MyDB WITH TRUNCATE_ONLY to dump the contents of the committed transactions from the logs. Also, increase the size of the log file before hand - every time it grows automatically (database as well) it will slow down performance. When you are done, run DBCC SHRINKFILE ('filename') for the database and the log file (use the logical filename, not the phsycal).

And get more hard drive space.
0
 

Author Comment

by:new_wes
ID: 17958676
ha! no kidding about the drivespace.  thanks!!  trying now.  any ideas why the log file seems to grow X times larger than the transactions it is logging??
0
The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

 
LVL 29

Expert Comment

by:Nightman
ID: 17958714
Probably the index changes as well as the data. Don't know what your indexing strategy is, but the indexes are also changing, not just the data,
0
 

Author Comment

by:new_wes
ID: 17959743
hmm - no indexing yet - just imported thsee from text . . .

but success with the queries!!  total runtime less than 1hr.  thanks very much, makes moving forward much easier.
0
 
LVL 29

Expert Comment

by:Nightman
ID: 17959771
Glad I could help.
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction In my previous article (http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/SSIS/A_9150-Loading-XML-Using-SSIS.html) I showed you how the XML Source component can be used to load XML files into a SQL Server database, us…
The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…

751 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question