Solved

Zipping_Files_Desiign.

Posted on 2011-09-20
11
225 Views
Last Modified: 2012-06-27

I have an HTML web form (pl/sql generated ) where users can enter a search criteria, hit "find" and the system will search the ORDERS (oracle 9i) and list a webpage with all the orders that were found. The user should be able to hit "Zip & Download" button which will dump each ORDER HTML page to a directory on the server, then ZIP all the files together and download ONE zip file to the user client machine. The download can be using a direct link to the server or using mod_plsql.

I am using a java stored procedure to call the UNIX zip command described in this link.

http://www.ehow.com/how_5056873_create-zip-file-unix.html


How would you best design this in terms of implementing concurrency controls in case "john" logs in and do a search/download and "mike" logs in and does another search/download.

Would you create one /tmp/ directory to dump all the files for all users and then ZIP all those order files into a unique zip filename?
or would you use one directory per user /tmp/john/, /tmp/mike/ and clean up the directory each time they press the button.

Can I also use the same zip file name or it has to be somehow unique.

Would you also call a direct link to the ZIP file on the filesystem,

Can you advise on the best way to implement this.

0
Comment
Question by:sam15
  • 5
  • 4
  • 2
11 Comments
 
LVL 76

Accepted Solution

by:
slightwv (䄆 Netminder) earned 250 total points
Comment Utility
Personally I would create a single folder for all these.  I would also name the files with a GUID or some other similar unique name for ALL requests.  You should be able to specify a different download filename when the time comes but keep them unique on the fie system.

The main reason is a user can exit the form and the zip will continue processing.  Say john gets tired of waiting for the file and clicks the stop button and does another search?  Now he would have two files being created and only one possible to download.

One folder will also make cleanup easier.  You will probably need some process to go through the folder every once in a while to clean up abandoned files.
0
 

Author Comment

by:sam15
Comment Utility
well, after i zip the files they are no longer needed. The problem is that what about if another user hit the ZIP button for a different set, that may delete his files too.

The individual file names (inside zip file) will be order-id.html.

I am not sure if the cleanup should be for all files or just named files in the directory that the ZIP will do.
I think the design will have some issue when multiple users do it at same time even though priobablity is very low but it is still possible. I am wondering if i should lock the directory somehow until each user is done with download of the ZIp file.
0
 
LVL 19

Assisted Solution

by:xterm
xterm earned 250 total points
Comment Utility
As long as you come up with a unique name for the zip file, 100 people can be creating new ones at the same time and none will be denied, because the contents getting simultaneously accessed are only reads.  So if you create something like user-[pid of zip command].zip you'll be golden.

I agree with slightwv - unless you think there's the possibility of literally thousands of files in there, then I'd go with a single folder.  That way you can run something like "find . -mtime +1 | xargs rm" in there by cron and it will whack everything that's more than a day old.

BTW, you will want to put the file in a web-server owner (probably user apache or user nobody) writable directory so that you can present a full URL to the client to snag the file.  But remember to put an empty (0 byte) index.html file in there so that people can't browse that folder directly.  Alternatively, you can store these files outside of the web tree, but then you'd have to put a link that pops up an entirely new window to serve up the file because you can't change the content type after headers have already been sent in a conventional web page.

Let me know if you need clarification on that last part.
0
 
LVL 76

Expert Comment

by:slightwv (䄆 Netminder)
Comment Utility
>>problem is that what about if another user hit the ZIP button for a different set

I mentioned that in my post and it was repeated by xterm:  The ZIP filename is unique to the OS using something like a GUID.

>>The individual file names (inside zip file) will be order-id.html.

The files inside the zip file doesn't really matter.

>>I am not sure if the cleanup should be for all files or just named files in the directory that the ZIP will do.

What other files are you planning on storing in the 'staging' folder?  I would only store the ZIP files so removing ALL or only ZIP are the same thing.

>>I am wondering if i should lock the directory somehow until each user is done with download of the ZIp file.

No.  You are still thinking a single zip filename across the entire app.  Each click of 'download' will generate a unique filename specific to that click.

One shared folder the app has write access to.  100000 people can run at the same time and there will be 100000 unique .ZIP files created.  What is the problem?
0
 

Author Comment

by:sam15
Comment Utility
<<What other files are you planning on storing in the 'staging' folder? >
well, i have to create/dump all the individual HTML files before I can zip them in UNIX directory. These are the files I am refering to. So i would have all the indivdual files + one zip file.

I guess a unique filename ensures correctness and implements concurrency like you all said. Using one file name for everyone will not work for sure and may cause serious problems? right

I do not want to present the user with a link for the ZIP file. After he hits the "Download Button" I should generate the zip file and he should see a prompt to "save" the file on his PC so i think that is doable programmatically.

Concerning unique filename, I am more incline to use something like date/time or sequence number unless ther is a better is something more meaningful. so it will be some thing like

ORDERS-100.ZIP
ORDERS-101.ZIP

or ORDERS-092211180000 (date/time)

Any suggestions.

0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 76

Expert Comment

by:slightwv (䄆 Netminder)
Comment Utility
>>well, i have to create/dump all the individual HTML files

OH, sorry I misunderstood.  I didn't read the Java link because it probably wouldn't have made sense to me.  For some reason I thought you were creating the Zip file in memory and streaming it out to disk.  Not sure why I thought that.

Then I suggest you dynamically create a subFolder with a unique name for each request.  Then the individual file names in that folder don't matter.

For the folder name, I strongly suggest a GUID over a date/time.  What if two people execute at the same second?
0
 

Author Comment

by:sam15
Comment Utility
I think in order to use UNIZ ZIP, it will only work in files that exist in filesystem (not dynamically).

With GUID, you mean sys_guid and not normal Sequence number that keeps incrementing like
ORDERS-1.ZIP
ORDERS-2.ZIP
etc..
ORDERS-100.ZIP
0
 
LVL 76

Expert Comment

by:slightwv (䄆 Netminder)
Comment Utility
Zipping in memory depends on the app/code you are using.  It may or may not be an option for you.

>>Sequence number that keeps incrementing like

Since it appears you need a subfolder, the zip filename doesn't mean anything.

Create a unique folder per request.  Put whatever files, named whatever you want in there.  Create a zip file called whatever you want in that same folder then stream it back to the client.

It is now the folder name that is important not the file names.

0
 

Author Comment

by:sam15
Comment Utility
I am confused now.
I thought you and other commentators recommend to use one folder for all users with a unique filename per request. Why would I need to create a folder per user unless I must keep one standard filename for every download?
0
 
LVL 19

Expert Comment

by:xterm
Comment Utility
If you use a continually increasing number (like say an auto-increment field in a DB that maybe you insert every user download into), then you can just use the id from that and ORDERS-<id>.zip should work fine and never conflict with anything else.  And you can use a single folder for that too.  I think you're making this a bit more complicated than it needs to be.
0
 
LVL 76

Expert Comment

by:slightwv (䄆 Netminder)
Comment Utility
>>I thought you and other commentators recommend to use one folder for all users with a unique filename per request.

I did suggest that when I thought you were only creating one ZIP file from streaming the files needing to be zipped.  In other words, zipping in-memory.

Once I realized that assumption was in error and you needed to place several files on the disk BEFORE you zip them I had to change the suggestion.

0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Read about why website design really matters in today's demanding market.
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Via a live example, show how to restore a database from backup after a simulated disk failure using RMAN.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now