Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Optimize union insert (runaway trans log) - table size, nolock and 'nolog'?

Posted on 2006-11-16
6
Medium Priority
?
534 Views
Last Modified: 2008-02-01
Hi all,

Stable SQL2000 setup on W2000. 7g db on drive (100meg trans log - Simple log mode) with another 10g free.  Working with 3 similar tables, trying to union insert them to one table. Existing 3 tables no more than 2-3g, 15mil rows.  Using stored proc like this:

INSERT INTO [CLAIMS_A_2002_04]
( CLAIM_TYPE, CL_YEAR, FPTYPE, DT_recipnum, DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3x)
SELECT 'A' AS 'CLAIM_TYPE', left([DOS1],4) as 'CL_YEAR', 'FP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT12, CPT3, CPT4x
   FROM [acg_temp].[dbo].[FTP A Claims FP 2002_04] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2002 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3x
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2002] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2003 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2003] with (nolock)
UNION ALL
SELECT 'A' AS 'CLAIM_TYPE', 2004 as 'CL_YEAR', 'NFP' as 'FPTYPE', [DT_recipnum],  DOS1, DOS2, PDC, SDC, TOTPAID, PROVTYPE, PTPPS, CPT1, CPT2, CPT3
   FROM [acg_temp].[dbo].[FTP A Claims nonFP 2004] with (nolock)
GO
SET QUOTED_IDENTIFIER OFF
GO
SET ANSI_NULLS ON
GO

Procedure seems to start off okay but never finishes: last run blew up db to 10g with trans log of 7g.  Usually ends with messge 'truncate log' or 'can't write to log' etc.  Is there anything I can add/rewrite to get these tables into one table??   Figured the 'nolock' would do it, but now out of ideas.  Is there any nolog switch etc?  I can keep everybody out of the db while q is running, and have full backup so not concerned about rollback etc.  Why is it writing so much to the logs??  really would help if this would work!

thanks!!

wes
0
Comment
Question by:new_wes
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 

Author Comment

by:new_wes
ID: 17958351
PS - Row size on empty target table is at most 200char (below), so expecting 15m rows * 200 char = 3g of data??  What am i missing that the db/log would grow by almost 3x that much and still not complete?

CREATE TABLE [CLAIMS_A_2002_04] (
      [CLAIMSAKEY] [numeric](18, 0) IDENTITY (1, 1) NOT NULL ,
      [CLAIM_TYPE] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CL_YEAR] [bigint] NULL ,
      [FPTYPE] [char] (3) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DT_recipnum] [char] (12) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DOS1] [char] (11) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [DOS2] [char] (11) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PDC] [char] (6) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [SDC] [char] (6) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [TOTPAID] [char] (17) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PROVTYPE] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [PTPPS] [char] (2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT1] [char] (10) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT2] [char] (5) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
      [CPT3x] [char] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) ON [PRIMARY]
GO
0
 
LVL 29

Accepted Solution

by:
Nightman earned 2000 total points
ID: 17958398
In simple recovery model everything is written to the log - it is just flushed as soon as the transaction has been committed.

I would recommend that you perform the transaction in small batches (a few hundred thousand records at a time) and in between run BACKUP LOG MyDB WITH TRUNCATE_ONLY to dump the contents of the committed transactions from the logs. Also, increase the size of the log file before hand - every time it grows automatically (database as well) it will slow down performance. When you are done, run DBCC SHRINKFILE ('filename') for the database and the log file (use the logical filename, not the phsycal).

And get more hard drive space.
0
 

Author Comment

by:new_wes
ID: 17958676
ha! no kidding about the drivespace.  thanks!!  trying now.  any ideas why the log file seems to grow X times larger than the transactions it is logging??
0
Survive A High-Traffic Event with Percona

Your application or website rely on your database to deliver information about products and services to your customers. You can’t afford to have your database lose performance, lose availability or become unresponsive – even for just a few minutes.

 
LVL 29

Expert Comment

by:Nightman
ID: 17958714
Probably the index changes as well as the data. Don't know what your indexing strategy is, but the indexes are also changing, not just the data,
0
 

Author Comment

by:new_wes
ID: 17959743
hmm - no indexing yet - just imported thsee from text . . .

but success with the queries!!  total runtime less than 1hr.  thanks very much, makes moving forward much easier.
0
 
LVL 29

Expert Comment

by:Nightman
ID: 17959771
Glad I could help.
0

Featured Post

Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question