We help IT Professionals succeed at work.
Get Started

SSIS custom data source component - buffer overflow handling

492 Views
Last Modified: 2013-09-17
Developing a SQL Server Integration Services Custom Data Source Data Flow Component.

SSIS has a maximum buffer size and row limit to "chunk" work based on buffer size.

This requires a source component to recognize when the buffer is full, and then suspend until another buffer is available and resume loading from the source to the new buffer.

What I don't see covered anywhere is:
 - how to effectively deal with a full buffer
    > calculate the size, count rows and ensure closing the buffer before overflow (except that I can't see how to get the max size or max rows from the parent data flow task)
    > wait for a buffer overflow exception and rewind to the last row/bytes written to resume with a new buffer.
 - how to store the "bytes/rows to skip" information for the component to pick up the next time it's invoked by SSIS.
Comment
Watch Question
This problem has been solved!
Unlock 1 Answer and 3 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE