NJThomson
asked on
SSIS custom data source component - buffer overflow handling
Developing a SQL Server Integration Services Custom Data Source Data Flow Component.
SSIS has a maximum buffer size and row limit to "chunk" work based on buffer size.
This requires a source component to recognize when the buffer is full, and then suspend until another buffer is available and resume loading from the source to the new buffer.
What I don't see covered anywhere is:
- how to effectively deal with a full buffer
> calculate the size, count rows and ensure closing the buffer before overflow (except that I can't see how to get the max size or max rows from the parent data flow task)
> wait for a buffer overflow exception and rewind to the last row/bytes written to resume with a new buffer.
- how to store the "bytes/rows to skip" information for the component to pick up the next time it's invoked by SSIS.
SSIS has a maximum buffer size and row limit to "chunk" work based on buffer size.
This requires a source component to recognize when the buffer is full, and then suspend until another buffer is available and resume loading from the source to the new buffer.
What I don't see covered anywhere is:
- how to effectively deal with a full buffer
> calculate the size, count rows and ensure closing the buffer before overflow (except that I can't see how to get the max size or max rows from the parent data flow task)
> wait for a buffer overflow exception and rewind to the last row/bytes written to resume with a new buffer.
- how to store the "bytes/rows to skip" information for the component to pick up the next time it's invoked by SSIS.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
'cause no one else provided an answer
HTH,
Kent