Lower level control over LWP (speed limiting, aborting)
Posted on 2009-07-16
There are two things I've been wanting to know how to do for a while now. I'm not even sure they're possible the way I want them done.
Must use the LWP (libwww perl) module.
1) I'd like to be able to throttle the speed of downloads being performed via LWP. Basically, this would be equivalent to cURL or GetRight's "speed limit" functionalities. I had thoughts on being able to -possibly- do this via putting a fairly granular select(undef,undef,undef,$timer) inside of a handler assigned to the request that would get called on each chunk. Couple problems with that:
a) I don't know if even small sleeps will be acceptable. I do know for a fact that if you have a raw socket and don't read often enough as data streams in, you suffer data dropout. I'm not sure if HTTP is actually designed in such a way that it's synchronous to the point it won't overrun the socket's buffer.
b) I'm unsure of how to figure out what timeout to put on the select() if I put it in there. It'd have to be based on how big the chunk size is, and how many chunks it would take to achieve the correct throughput over time. it'd also have to account for actual time elapsed and decide whether or not to actually sleep a little bit. It may even need to be dynamic (say, if the buffer size isn't coming back with the entire requested size). The exact algorithm eludes me.
2) I'd like to know how to abort an LWP request without terminating the entire program or otherwise doing something that would mess about with program functionality in an unwanted fashion. In the instance I'm considering, I'm doing downloads inside a Perl/Tk application. i have an abort function, but it finishes the current file and then stops before requesting another. I want to take it to the next level and have it do a proper abort in mid-download. The docs for the handlers (specifically the data handler) seemed to mention something about croaking to abort, but my reading on croak() said that it's equivalent to die(), which is not what I want. I simply want to abort the download, yet retain the instance of the running program and return control to the main Tk event loop in this case, or in other programs, simply continue onwards.
It's not like I haven't researched either of these points at all. It's that I've not found anything clear, concise, and definitive. Everything I've found on aborting is hypothetical and sketchy. My own ideas on speed throttling are hypothetical, and I've not been able to really find a reference at all to doing it within the LWP framework.
If you can provide working code examples (skeletons are fine), so much the better.
Thanks in advance!