Upload hundreds of files to server

Hello I have a client who needs to upload images in batches of 500-600 at a time.  We are currently using JQuery but it can not handle this.

What type of script should I be using.
bradley525Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

RobOwner (Aidellio)Commented:
Have you thought of using drop box or ftp? Or are you looking to continue a user friendly approach?
0
bradley525Author Commented:
We need a user friendly approach.We need to integrate it into our web system that we have built. I can not use Dropbox..

I am trying to see if there is a way to upload using FTP via php and/or jQuery?
0
RobOwner (Aidellio)Commented:
As for jQuery not handling this, are you sure it's not your php?
Can your client zip the files and upload one large file?
You may want to look at an ActiveX or java applet or flash plugin that will be able to handle reliable ftp connections to your server.
As you are restricted by connection timeouts with php/jQuery.
Let me know your thoughts as there is a lot of options here and I'd want to know what's suitable for you.
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

RobOwner (Aidellio)Commented:
Sorry just seen your post, if that's the case I would look at using a plugin such as java/flash or if purely Windows then ActiveX
0
bradley525Author Commented:
I would really like to use php and jQuery. Uploading a zip file may work as well..

I would prefer to stay away from flash and/or java as I do not program either of those..
0
RobOwner (Aidellio)Commented:
Lots of options available to you. Haven't used this one but it looks promising
http://sourceforge.net/projects/zupload/
0
bradley525Author Commented:
I have full control over my server so I can update php.ini settings as well as .htaccess if needed..and any other server setting that is needed.
0
RobOwner (Aidellio)Commented:
Then it looks like it's a zip file or a restriction to how many files can be uploaded at any one time.
What exactly is the error you're getting when trying to upload that many files?
0
RobOwner (Aidellio)Commented:
It's the client literally dragging 500 files to the browser? That could quite possibly crash the browser and have nothing to do with your site at all.
Look I'm guessing here-.-would be good to know more about the issue and the process to be able to advise on this
0
Daniel HelgenbergerCommented:
Here is another way if you need these pictures in a database and want therefore using POST, like with a web form.

This is like literally dragging 500 files in the browser.

If its not already there, create simple upload form for this on your server, the field for the upload is called "file" this example:

Then use curl for this, like this simple bash script:
#!/bin/bash
for f in mydirectory; do
    curl -F file=@$f http://webserver/upload.html
    sleep 2
done

Open in new window

-F implies HTTP-POST multipart/form-data; exactly what you need for your pictures.
0
Ray PaseurCommented:
a client who needs to upload images in batches of 500-600 at a time
Unfortunately that is a problem if you insist on doing it via HTTP with a browser.  Let me try to explain why and what I've done to work around the limitations.

In the world of client/server protocols, the client is the only one who can initiate a request.  The server can only provide a response to a request.  This is, in large part, a matter of client security, to prevent rogue servers from initiating attacks on client machines.  One element of client security is the fact that servers are not allowed to read files from client machines unless the client tells the server to read the file.  This is done via an HTML file upload form (and jQuery implements a similar thing).  In the HTML environment, it is impossible for the scripts to prepopulate the information in the input controls for type="file" and while I can envision how this might be possible with javaScript, I have never seen it done and I would think it would be regarded as a security hole.  All of this leads us to the conclusion that you can't upload a "directory" of images.  You have to tell the server which files it may read, and you have to tell the server one file at a time.  You can upload several files at once, but each one must be selected manually.  Whether you're clicking through a directory structure or doing a drag-and-drop, this is laborious.  Security and convenience are always trading off.

Since I am a photographer, I upload directories of images all the time.  I use a simple, free, FTP program that takes only a few keystrokes to initiate.  This is what I would recommend for your client, too.  Give them a directory to upload the files, teach them how to do the FTP process (steps 1, 2, 3 in writing) and let it be the solution.

FWIW, I've done some work with ZIP archives and PHP, and you probably do not want to go there.  The PHP extension is flaky and release dependent, and since it's both old and buggy, it's not likely to ever get fixed.  I think the idea of having the client ZIP all the images and upload one big file sounds like a seductively easy approach, but I can guarantee that you'll be doing a lot of firefighting.

Wish there was a magic bullet, but there is not.  At some level, it might make sense for the client to just put all the images on a thumb-drive and Fedex it to you!
0
Julian HansenCommented:
I second the recommendation by tagit to use zip - as you have complete control over your server there are a number of different methods you can use to extract the archive - you don't have to use PHP - although I use the PHP zip module all the time for precisely what you are talking about now. Client zips the images - uploads the zip with a FTP client and from an admin page processes the ZIP. The script unzips the files - creates thumbnails from the images and moves the batch to the images folder - been working without a hitch for several years.

Having said that - I would look at some server process that could be invoked that could do it more efficiently - as you have control over your server this should be relatively easy to setup.

You can do it through the browser - but you can also cut down a tree with a spade - just a whole lot of extra unpleasant effort.
0
Ray PaseurCommented:
Obviously @julianH and I have had different experiences with ZIP!  Maybe some of my negative opinions of PHP ZIP have to do with creating archives on the server rather than consuming archives on the server.  

It never occurred to me that I would apply ZIP compression to an image, but in this case it could make sense.  Consider where the ZIP value comes from.  It does two things: Compression and Aggregation.  Compression is not worth much with image files, and besides, image compression has already been thought of with JPG and PNG.  But aggregation may be an advantage, since the HTTP file upload process is invoked on a per-file basis.  As such, a single ZIP archive can be used to upload many internal files in a single pass.  So your script can say to the client, "Upload a ZIP Archive" instead of "Upload a Lot of Files" and the client will not have to sit there selecting files one after another.

Of course, dealing with one large file has its own set of issues.  These man pages are particularly on point.
http://php.net/manual/en/ini.core.php#ini.upload-max-filesize
http://php.net/manual/en/ini.core.php#ini.post-max-size
http://php.net/manual/en/info.configuration.php#ini.max-input-time

Best of luck with it, ~Ray
0
Julian HansenCommented:
Obviously @julianH and I have had different experiences with ZIP!  Maybe some of my negative opinions of PHP ZIP have to do with creating archives on the server rather than consuming archives on the server.  
+1 for that - I have not had good experiences with creating archives.

Also on consuming them I have given my client specific instructions on how to create the zip - no folders in the ZIP - must be in the root folder.

As you say compression is not the reason for doing it but rather the simplicity for having to deal with one file over 100's.

In the "old days" this was in itself a problem - the line gets dropped when you are 99% of the way through the upload and educating clients to use reget (if supported) was too much like hard work - discrete file uploads seemed a more sensible way to go. However, those golden days are past and uploading several megabytes of files is no longer an issue.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
gr8gonzoConsultantCommented:
Pretty much everything said so far has been on point (I've only skimmed through the responses), so I'm just adding my experiences.

I am also a photographer and I regularly use Gallery (http://galleryproject.org/) to upload dozens of photos at a time. It does have a client that allows me to select large quantities of images that are uploaded via the browser, but even on a top-speed FIOS connection, I often have one or two images that fail and have to be re-transferred. This is usually with less than 100 images, so uploading 500+ would be that much more problematic. Granted, the images are processed as they are uploaded, but it's still not ideal to upload 500 individual files, even if it's possible via the interface (I know Flash offers methods to upload multiple files via one UI selection, for example).

I don't think ZIP files are necessary or helpful in this scenario. Chances are that a ZIP file containing 500 images is going to be too large for any connection to reliably upload, and uploading 500 images individually will be problematic over HTTP.

That said, I would further suggest an FTP(S) server approach. That way, a user can upload the files individually to an FTP server with reliable results, and chances are that you can have your script automatically set up FTP accounts for users and be able to programmatically read from the folders to look for newly-uploaded files.

It's just likely the best option for that kind of volume.
0
Dave BaldwinFixer of ProblemsCommented:
If you actually mean 300 to 500 files, I wouldn't consider anything but FTP because it's made for it.  PHP even with the FTP extensions available isn't made for doing that kind of uploads.  You have to increase the execution time way beyond normal along with available memory.  The FTP process handles files one at a time even though it may be a long list on the client.  If you lose the connection in PHP, you may lose the whole upload.  In FTP, the files that were previously uploaded are still there and you just have to upload the ones that didn't make it the first time.
0
RobOwner (Aidellio)Commented:
I agree with everyone's contents re zip files but it's certainly worth trying while you work out something else as it has minimal impact on the client short term. At this point you still have to find another solution to uploading via a web browser. Remember the file can be unzipped using the server filesystem, you don't have to use a php extension for that.
I don't know if it is a possibility for you but cloud services could be useful in this case. I know both Google drive and Microsoft's SkyDrive have desktop "folders"that you just drop whatever file(s) in them and they automatically sync with the server. All depends on what you're doing with these images
0
Bernard S.CTOCommented:
Since we are discussing abiut hundres of files and MB, a ftp-like solution si probably the most effective one.
It will need some handholding on the first occasions... but nothing crazzy.

What is needed though is to setup ftp on the server so that the ftp is foolproof, ie it is not possible to upload to any directory than the space planned.

Once this is done I would probably some common ftp programs such as filezilla or Tunnelier (although this last one can probably do too much).
Some initial tuning might be necessary, eg for resuming upload after a problem (file syncs etc).

I would think this is not a browser based project...
0
RobOwner (Aidellio)Commented:
I agree, and seems to be the general consensus that this is not achievable via a web browser.
0
bradley525Author Commented:
I've requested that this question be deleted for the following reason:

There is no correct answer I found in this post that will work correctly
0
Bernard S.CTOCommented:
I object to question deletion.
- Asker has presented a problem he tries to solve with http / browser
- General consensus among experts is that this problem is not a browser solvable problem, and have hinted with different elements at the obvious solution: FTP
- THIS IS AN ANSWER, even this does not please the asker. And from what the links the contributions provide, I would think they are very useful.

Therefore I think the asker should reconsider and select which answer(s) are the most relevant to the problem (my own contribution provides no solution)
0
RobOwner (Aidellio)Commented:
I agree with @fibo.  
We have to think about others coming across this question and wanting to know whether it is possible or not.  
Therefore it is relevant to keep the question, as it says it isn't possible with HTTP and provides many other solutions to the issue, the most prevalent being the FTP option.
0
gr8gonzoConsultantCommented:
Bizarre behavior by the author. Oh well.
0
Daniel HelgenbergerCommented:
Hello guys,
I am quite amazed (please be gentle, I am a rookie here) about this thread witch I followed the past days.
Why not allowing to close the thread? Clearly the author has a point in arguing this going the wrong direction. While a way in PHP was needed, this is all about efficiently. FTP is clearly the the best technical solution as all of you pointed out.
But - as I have often to stuff the way the client was and not the way I think is best - using FTP may not be possible at all. The author has no solution and was forced to pick one; by chance mine - witch was taken away in turn - this is why I feel inclined to answer again :)

Breadlev, in my experience you need some kind of upload client on the user's computer to do this with a web browser. This is why so many multi file uploads are done in flash.

The process also needs to be transparent in in background. Meaning the user must be able to close the browser and if interrupted, he needs to know where to pick up.

A solution may be to generate a text file with all the upload file paths and upload this. Then you generate an upload form from the text and let it submit. If interrupted the user needs to be able to use the same text file while you do the checking what is already there. Send the user a mail if everything was ok as well as in case of an error. Present a link witch shows all the files done and all the files missing and a simple submit button to continue the process.
Please keep in mind max_file_uploads in PHP:
http://www.php.net/manual/de/ini.core.php#ini.max-file-uploads

Also, it may be possible do upload a zip file. Then start a shell script on the server witch unzips and put all the files in the right place (for instance your database).

Still, there are many more ways to do it, just take a boarder approach then plain PHP. This may be the essence of all the Expert's answer here?
0
RobOwner (Aidellio)Commented:
I'm certainly not against closing the thread! :)  
It's how the thread is closed that is important.  First the question was to be deleted, which was objected on fair grounds given the ambiguity of the author's closing comment.
Then it seems that a comment was selected at random as there was no feedback and nothing that really set that comment apart from the general consensus.  
It is important to note that these questions/solutions form part of a massive knowledge base that we want to be as accurate as possible.  
If you were to come across this question you would expect to be able to find a solution of which there isn't one without the feedback from the author.
0
RobOwner (Aidellio)Commented:
The OP wanted a user friendly approach using php and/or jquery

The general consensus here is that it can't be done reliably through a web browser (OP indicated jQuery couldn't handle this and the OP didn't want to introduce any plugins such as applets, flash or activex).

The OP commented that "Uploading a zip file may work as well.." but no confirmation on this was given by the OP.

FTP was also suggested by most of the experts and seems to be what we all agree would be the best solution.

On that note I would specify the points evenly split with the following experts comments:

@Ray_Paseur on experience with images and FTP - http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_28227703.html#a39454904

@julianH more on FTP solution - http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_28227703.html#a39454950
0
bradley525Author Commented:
Thank you..There were so many different answers given here I was not able to test and try all of them. Thank you.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.