I'm a newb when it comes to threading. I have a very specific thing I would like to try out.
I want to download html from a list of url's I have. There are approximately 300 url's. I can usually get through about 40-50 url's before the downloader I am using hangs up. I want to use a timer thread to call a method after 10 seconds (I define a "hangup" as any download that is taking longer than 10 seconds: the average url takes about 1 sec to download, so this is reasonable). The timer is reset after each successful download. If a "hangup" occurs I want to stop the current thread (the one that is hung up) and start a new thread using the same url.
So something like this:
1.) start process (main)
2.) load url list from file.
3.) loop through each url, processing each one in the following manner:
a.) start thread timer.
b.) start download of url. <---(**note 1**)
c.) if download successful, stop timer and repeat loop using next url.
d.) if thread timer expires (download is unsuccesful), stop timer and repeat loop using same url. increment 'downloadAttempts' by 1.
e.) if 'downloadAttempts' > 6, stop timer and repeat loop using next url.
However, when i implement the above timer thread algorithm I get all kinds of unexpected errors. I believe they stem from the fact that when the "hang up" occurs (and I start a new thread using the timer) the OLD thread (the one that is "hung up") has not been disposed of and is using resources I need.
So a couple questions:
1.) is the general layout of my timer thread algorithm ok? Am I approaching this problem correctly?
2.) how to dispose of the thread that is hung up? can I save the id of a thread, and if it gets hung up, dispose of it from a new thread (using the timer thread for example)? (see **note 1**)
// it seems like at this point instead of using the main process thread I could launch a new one:
b.) start download of url in its own thread:
1.) start thread (save id of it) and use it to perform download
2.) if timer thread expires ("hang up" occurs) delete thread (we saved the id, so this seems possible)
3.) repeat loop using same url
Many thanks for your expert advice!