• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 53
  • Last Modified:

Improve speed of cursor

I have an SSIS ETL process in which i use the multiple flat file connection to create 2 main tables:
  • DE_MASTER
  • DE_MASTER_ARCHIVE

I then have a cursor to union these 2 tables in SELECT Statement.  In the SELECT statement i am selecting most (almost all) of the columns.  This cursor populates other normalized tables, so the source tables are essentially read only after the ETL process is done.

My question is:

How can i improve the performance of this SELECT?  The union of the 2 tables produces about 3m rows and takes about 15 mins
There are no indexes on each of the source tables in the union.
0
gdinunzio
Asked:
gdinunzio
  • 5
  • 4
  • 3
  • +1
2 Solutions
 
Russ SuterCommented:
Adding indexes where they would be useful could improve performance but your best bet by far is to retool the process to use a set-based approach instead of a cursor. With VERY few exceptions, cursor-based actions can be remade into set-based ones and the performance gains are significant to say the least.

Without you providing the code there's not much specific advice I can give you on how you might convert this into a set-based approach.
0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
i fully agree with above: avoid cursors at all costs.
insert into  ... from ... join ....
and updates with joins usually does the job.
https://www.experts-exchange.com/articles/1517/UPDATES-with-JOIN-for-everybody.html

and if really you need a row by row handling
https://www.experts-exchange.com/articles/13640/processing-cursor-vs-temp-table-syntax.html
1
 
Jim HornMicrosoft SQL Server Developer, Architect, and AuthorCommented:
Would help if you could copy-paste the T-SQL into this question.
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
Russ SuterCommented:
and if really you need a row by row handling
https://www.experts-exchange.com/articles/13640/processing-cursor-vs-temp-table-syntax.html
I like your thinking here. Even temp tables are far more efficient than cursors. Put simply, ask most database experts and they'll tell you that cursors are pretty much the worst thing you can do in a database. Even the oft vilified trigger does not receive so much disdain.
0
 
gdinunzioAuthor Commented:
Apologies but i had to spend a minute obfuscating the sensitive data names...

Here is the basic union statement:

SELECT             
            m.N_ID, RTRIM(m.N_Name), TRY_PARSE(m.N_DOB AS DATE) AS DoB, m.N_SEX, m.N_INT,
            TRY_PARSE(m.N_DATE AS DATE),             
            TRY_PARSE(m.N_TIME AS TIME),
            TRY_PARSE(m.N_DATE_PR AS DATE),
            TRY_PARSE(m.N_TIME_PR AS TIME), m.N_INT_PR,
                        
            NULLIF(m.C_FIRA, ''), m.C_PREL, m.C_PMIS, m.C_OPEN,      

            m.A_ID,      TRY_PARSE(m.A_DT AS DATE) AS ADt, TRY_PARSE(SUBSTRING (m.A_TM, 1, 2) + ':' + SUBSTRING (m.A_TM, 3, 2) AS TIME) AS ATm      
      FROM DE_MASTER AS m      
      UNION
      SELECT             
            ma.N_ID, RTRIM(ma.N_Name), TRY_PARSE(ma.N_DOB AS DATE) AS DoB, ma.N_SEX, ma.N_INT,
            TRY_PARSE(ma.N_DATE AS DATE),             
            TRY_PARSE(ma.N_TIME AS TIME),
            TRY_PARSE(ma.N_DATE_PR AS DATE),
            TRY_PARSE(ma.N_TIME_PR AS TIME), ma.N_INT_PR,
                        
            NULLIF(ma.C_FIRA, ''), ma.C_PREL, ma.C_PMIS, ma.C_OPEN,      

            ma.A_ID,      TRY_PARSE(ma.A_DT AS DATE) AS ADt, TRY_PARSE(SUBSTRING (ma.A_TM, 1, 2) + ':' + SUBSTRING (ma.A_TM, 3, 2) AS TIME) AS ATm      
      FROM DE_MASTER_ARCHIVE AS ma      

The cursor presently creates name records in table N then needs the "N" PK from that insert to create records in table c then needs the "C" PK to create records in table "A"
0
 
Russ SuterCommented:
I don't see a cursor here. Is the code you provided extracted from the inside of a cursor block?

That aside I do see a performance issue. Just about every column you're selecting is being run through a function. That makes indexing next to impossible. Certainly the built-in SQL optimizer won't have a clue what to do with this. You really should try to avoid using functions like this. I'm not sure why you're doing a TRY_PARSE() on a field called N_TIME. Shouldn't that field already be the correct data type?
0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
try also union all instead of union
0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
try also union all instead of union
0
 
gdinunzioAuthor Commented:
Unfortunately no...the source data was all text...even the dates were text.
0
 
Russ SuterCommented:
UNION ALL won't give you any performance advantage. All that will do is show duplicates between union statements if there are any.

This is starting to look like what you really need to do is parse the text outside of SQL (perhaps using a .NET project) then transfer the data into your database. While SQL Server is capable of data manipulation that isn't what it's good at. Let the .NET framework do the heavy lifting and type coercion and then SQL can do its job more efficiently.
0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
union does an implicit distinct... hence i do see a performance impact...
0
 
gdinunzioAuthor Commented:
Thanks guys...excellent info!
0
 
Russ SuterCommented:
union does an implicit distinct... hence i do see a performance impact...
Ah, yes you're right. I hadn't thought of it that way for some reason. It could improve performance but in this case any improvement would be completely overshadowed by the rest of the slowdown.

Also, I tend to use good indexes when I do this stuff so I've never noticed any practical benefit even though it's theoretically there. Still, if you have no practical reason not to use UNION ALL then why not squeeze every nanosecond of performance out of it?
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 5
  • 4
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now