MS SQL Server: Modify data while looping through a datareader

Hi fellow experts,

I assume that this is a standard problem, but I haven't found an ideal answer to it.

Simplified, I have a table Persons with columns PersonID and Field1. Right now this table (in real life) contains 50.000 records and is about 200 MB large (with more fields, of course). in I create two connections, do a SELECT * FROM MyTable WHERE Field1='A' and loop through the connected datareader. In the loop, I do some computations and execute a "UPDATE MyTable SET Field1='B' WHERE PersonID=@PersonID" on the second connection, where @PersonID is fetched from the datareader.

In code, this looks like that:

cmd1.Connection = conn1
cmd2.Connection = conn2

cmd1.CommandText = "SELECT PersonID, Staat, ...somefields... FROM Personen where Staat IS NULL"
dr = cmd1.ExecuteReader()
Do While dr.Read
  [.. do some calculations ..]
  cmd2.CommandText = "UPDATE Personen SET ...somefields..., Staat=@Staat WHERE PersonID=@PersonID"
  cmd2.Parameters.Add("@PersonID", Data.SqlDbType.Int).Value = dr("PersonID")
  cmd2.Parameters.Add("@Staat", Data.SqlDbType.Int).Value = 1

Open in new window

The problem is, that the UPDATE command times out and never executs properly. I'm not surprised, because I'm modifying the data while it's being read by the datareader. Surely some kind of locking taking place here.

What would be the best way to loop through data, work on it, and write the modified records back?

Thank you for your support,

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Brian CroweDatabase AdministratorCommented:
Read the records into a datatable, make your adjujstments then commit them.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
StaudteAuthor Commented:
Will this read the entire 200MB into the RAM of the server? (Sorry, not familiar with datatables, although I have already seen the code I need to handle them.)
Brian CroweDatabase AdministratorCommented:
Yes it would pull them all into memory.  Personally I would find a way to do all this on the server via a stored procedure but I don't know enough about [.. do some calculations ..] to determine if that's possible.

You could always opt to bring them through in batches if it becomes a RAM issue.  Make some alterations to your query or create a stored procedure that can pull a "page" of records at a time, make your adjustments, and update.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

StaudteAuthor Commented:
Ok, I see. This code only runs once in a lifetime, does some updates (very complex - can't easily be done in a stored proc) and will never be needed again, so using 200MB of RAM will be ok - the server should have that much. I just wanted to understand what would happen.

Thanks for the quick reply - awesome!
Brian CroweDatabase AdministratorCommented:
You might try simply extending the connection timeout value which defaults to 30 seconds.
StaudteAuthor Commented:
Why should this help? If there's a locking issue, it will never be resolved inside the datareader loop, no matter how long I set the timeout value. I have - for a test - closed the datareader inside the loop just before executing the update, and the update ran smoothly. (Of course, it wouldn't be able to do another run of the loop with the datareader closed.)
Brian CroweDatabase AdministratorCommented:
I thought you might be having a timeout issue based on the size of the dataset.  If that's not the case then just go with the original suggestion :-)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server 2008

From novice to tech pro — start learning today.