Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Optimize union insert (runaway trans log) - table size, nolock and 'nolog'?

Posted on 2006-11-16
6
Medium Priority
?
536 Views
Last Modified: 2008-02-01
Hi all,

Stable SQL2000 setup on W2000. 7g db on drive (100meg trans log - Simple log mode) with another 10g free.  Working with 3 similar tables, trying to union insert them to one table. Existing 3 tables no more than 2-3g, 15mil rows.  Using stored proc like this:

INSERT INTO [CLAIMS_A_2002_04]
( CLAIM_TYPE, CL_YEAR, FPTYPE, DT_recipnum, DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3x)
SELECT 'A' AS 'CLAIM_TYPE', left([DOS1],4) as 'CL_YEAR', 'FP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT12, CPT3, CPT4x
   FROM [acg_temp].[dbo].[FTP A Claims FP 2002_04] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2002 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3x
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2002] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2003 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2003] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2004 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2004] with (nolock)
GO
SET QUOTED_IDENTIFIER OFF
GO
SET ANSI_NULLS ON
GO

Procedure seems to start off okay but never finishes: last run blew up db to 10g with trans log of 7g.  Usually ends with messge 'truncate log' or 'can't write to log' etc.  Is there anything I can add/rewrite to get these tables into one table??   Figured the 'nolock' would do it, but now out of ideas.  Is there any nolog switch etc?  I can keep everybody out of the db while q is running, and have full backup so not concerned about rollback etc.  Why is it writing so much to the logs??  really would help if this would work!

thanks!!

wes
0
Comment
Question by:new_wes
  • 3
  • 3
6 Comments
 

Author Comment

by:new_wes
ID: 17958351
PS - Row size on empty target table is at most 200char (below), so expecting 15m rows * 200 char = 3g of data??  What am i missing that the db/log would grow by almost 3x that much and still not complete?

CREATE TABLE [CLAIMS_A_2002_04] (
      [CLAIMSAKEY] [numeric](18, 0) IDENTITY (1, 1) NOT NULL ,
      [CLAIM_TYPE] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CL_YEAR] [bigint] NULL ,
      [FPTYPE] [char] (3) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DT_recipnum] [char] (12) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DOS1] [char] (11) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DOS2] [char] (11) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PDC] [char] (6) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [SDC] [char] (6) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [TOTPAID] [char] (17) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PROVTYPE] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PTPPS] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT1] [char] (10) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT2] [char] (5) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT3x] [char] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) ON [PRIMARY]
GO
0
 
LVL 29

Accepted Solution

by:
Nightman earned 2000 total points
ID: 17958398
In simple recovery model everything is written to the log - it is just flushed as soon as the transaction has been committed.

I would recommend that you perform the transaction in small batches (a few hundred thousand records at a time) and in between run BACKUP LOG MyDB WITH TRUNCATE_ONLY to dump the contents of the committed transactions from the logs. Also, increase the size of the log file before hand - every time it grows automatically (database as well) it will slow down performance. When you are done, run DBCC SHRINKFILE ('filename') for the database and the log file (use the logical filename, not the phsycal).

And get more hard drive space.
0
 

Author Comment

by:new_wes
ID: 17958676
ha! no kidding about the drivespace.  thanks!!  trying now.  any ideas why the log file seems to grow X times larger than the transactions it is logging??
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 29

Expert Comment

by:Nightman
ID: 17958714
Probably the index changes as well as the data. Don't know what your indexing strategy is, but the indexes are also changing, not just the data,
0
 

Author Comment

by:new_wes
ID: 17959743
hmm - no indexing yet - just imported thsee from text . . .

but success with the queries!!  total runtime less than 1hr.  thanks very much, makes moving forward much easier.
0
 
LVL 29

Expert Comment

by:Nightman
ID: 17959771
Glad I could help.
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
A Stored Procedure in Microsoft SQL Server is a powerful feature that it can be used to execute the Data Manipulation Language (DML) or Data Definition Language (DDL). Depending on business requirements, a single Stored Procedure can return differe…
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.

824 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question