Solved

Downloading images via HTTP only Last Modified

Posted on 2007-11-23
16
227 Views
Last Modified: 2010-04-21
Hi experts,

I need to download lots of images from a server on a daily basis via HTTP.

I need to only download the images Last Modified on a certain date.

Basically only the new ones.

Is there any way to do this please?

Thanks

ST3VO
0
Comment
Question by:ST3VO
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 4
  • 3
  • +1
16 Comments
 
LVL 16

Expert Comment

by:CodedK
ID: 20338688
Do you use a database ? MySQL ?
If so then are those images in a blob field..or do you keep the filenames?
You could query the database if this is the case and get the filenames ...then download.
0
 

Author Comment

by:ST3VO
ID: 20338738
If I could have access to that it would be perfect but all I get in a list of URL's :o/
0
 
LVL 16

Expert Comment

by:CodedK
ID: 20338749
Via HTTP you cant get the creation date of the file/picture.
And you cant rely on the filename.
I cant think a way to make this possible.
Wait for another expert input. :D
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:ST3VO
ID: 20338799
OK..Thanks!  :o)
0
 
LVL 28

Expert Comment

by:2266180
ID: 20339971
well, making it possible depends solely on the server. does the serevr provide correct last modified header? if so, then you can rely on that. otherwise, you're kind of lost.

basically, what you need to do is mimic a cache. that's all.
so you need to get familiar with the http protocol. caching part here:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html
as you will see, the modified date is not the only thing to use for caching. there is an etag header and some others if my memory helps me correctly.

you will have to go over the entire http protocol and get the necessary info out and implement that specific behaviour.

OR, if you don't need an application made for this, use an existing cache, like squid (which is a caching proxy).
so a simple way would be to install a caching proxy and get the images through that proxy. if the image was not changed, you will still get it, but from local disk, so it will be very fast and not consuming your internet connection.

anything above this, will have to follow http protocol so ... unless you have a great deal of time on your hands, I say go with squid or another local caching http proxy :)
0
 
LVL 26

Expert Comment

by:EddieShipman
ID: 20340145
Using Indy's TidHTTP control, you can use the HEAD command and get just the headers and check the LastModified.

idhttp1.Head('http://www.experts-exchange.com/expert/images/logo.jpg');
memo1.Lines.Assign(idhttp1.Response.RawHeaders);

0
 

Author Comment

by:ST3VO
ID: 20340154
My problem is that I have not access to that server. I only get a list of URL's with the links to the images from them.

Having this in mind...can I still work with your suggestions above having just link access to that server?

0
 

Author Comment

by:ST3VO
ID: 20340177
Hmmm intesting...EddieShipman!

So, i theory I could filter the headers and that the data and time of the last modified stamp of the image and if the local one is older then download it...right?

0
 

Author Comment

by:ST3VO
ID: 20340182
Is there anyway of just retrieving the Data and time or do I have to filter it?
0
 
LVL 26

Expert Comment

by:EddieShipman
ID: 20340218
Get just the last modified like this:
idhttp1.Response.RawHeaders.Values['Last-Modified']

Remember, though that that "timestamp" may have many different formats based on the server where the
images reside. You will need to figure out, from the value returned, what the actual timestamp is.

There is no "standard" way to format that info. Also, note that some servers may not return that value. It is
up to the server admin to set the headers that will be retruend. Oh, and another thing, script generated
images, such as if you return them from PHP's GDI calls, will not have that header.
0
 

Author Comment

by:ST3VO
ID: 20340321
Could you please give me a same code please as I cannot seem to get the Last Modified in the memo with:

idhttp1.Response.RawHeaders.Values['Last-Modified']

What I'm I doing wrong please?



Idhttp1.Response.RawHeaders.Values['Last-Modified'];
Idhttp1.Head('http://www.experts-exchange.com/expert/images/logo.jpg');
memo1.Lines.Assign(Idhttp1.Response.RawHeaders);

Open in new window

0
 
LVL 26

Accepted Solution

by:
EddieShipman earned 500 total points
ID: 20340350

Idhttp1.Head('http://www.experts-exchange.com/expert/images/logo.jpg');
memo1.Lines.Assign(Idhttp1.Response.RawHeaders);
ShowMEssage(Idhttp1.Response.RawHeaders.Values['Last-Modified']);

Open in new window

0
 
LVL 28

Expert Comment

by:2266180
ID: 20340366
and as I said, keep in mind that last-modified is not the only way to track if a document has changed. there is at least the etag header and maybe some more.
that's why I suggested a caching proxy. it's easier then going over the http protocol.
0
 

Author Closing Comment

by:ST3VO
ID: 31410704
Great!!! Thanks a million...I'll play around with it and see what I can come up with!!!
0
 
LVL 26

Expert Comment

by:EddieShipman
ID: 20349662
caching proxy requires downloading the file whereas the HEAD doesn't
0
 
LVL 28

Expert Comment

by:2266180
ID: 20349678
duuh.

look at the question. he WANTS to download stuff, jsut that he doesn't want to doens;d stuff more than once.
so where is the problem? the caching proxy will not download soemthing twice if it was not modified. that's the idea with the "caching".
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
In an interesting question (https://www.experts-exchange.com/questions/29008360/) here at Experts Exchange, a member asked how to split a single image into multiple images. The primary usage for this is to place many photographs on a flatbed scanner…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question