Zip up a collection of files into multiple 20mb (close enough works) parts and each individual part is self contained.

Zip or any other software
Zip up a collection of files into multiple 20mb (close enough works) parts and each individual part is self contained.
Each individual piece can be extracted separately, without needing any of the other pieces.
bitstreamincAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Paul MacDonaldDirector, Information SystemsCommented:
7-Zip (and I believe WinZip before it) will let you specify an archive size, splitting up the larger archive into smaller ones.

The self-contained bit depends on what you mean, but if you're expecting each smaller archive to be its own ZIP file, I'm not sure that's possible.  You're asking the ZIP program to "fit" your files into archives of certain sizes and I believe that's beyond what these two do.  In other words, you'd have to have all the smaller archives together in order to open the whole archive.  In order to achieve what you're looking for, I think you'd have to have a way to "pre-bundle" your data in to suitable chunks.
Bill PrewIT / Software Engineering ConsultantCommented:
With 7-ZIP it calls these "volumes" and you can do it from the GUI or command line programs.

sshot-414.png
sshot-413.png
»bp
bitstreamincAuthor Commented:
Thanks for the replies, I am aware of the split volume functionality.
I need each zip file to be self contained and not incremental pieces that rely on each other for unzipping.
It doesn't have to be zip, it can be any other piece of software and zip may not even be able to do this.
Thank you for the replies.
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

bitstreamincAuthor Commented:
Maybe it's possible to split files in increments?
For example if you have 100 files and you would like to zip them into 4 zip packages of 25.

Could you zip up the first 25, then the next 25 (another zip file) and so on, into self containing zips without manual intervention?
4 zip files, each with 25 files inside, and each one does not depend on the other for unzipping.
Joe Winograd, Fellow&MVEDeveloperCommented:
> a collection of files

How will you specify the source files — individual file names? a folder? a folder and its subfolders? multiple folders not sharing the same root folder?

Do you want a CLI, GUI, or both?

Regards, Joe

Update: Just saw your last comment. It's doable both ways — by either size, such as 20MB as you mentioned in the original question, or by count, as you mentioned in your most recent post.
bitstreamincAuthor Commented:
Specify folder (a sub folder might be needed in some instances.)


Cli or gui is fine, whatever automates the process best.

Thanks
Joe Winograd, Fellow&MVEDeveloperCommented:
What naming convention do you want for the multiple output ZIP files?

Where do you want the output ZIP files stored — source root folder? specifiable output folder? fixed location, such as AppData or Desktop?
bitstreamincAuthor Commented:
name convention = Name I provide + 001, 002, 003 etc is fine.
output path = path I provide .


Thank you
Joe Winograd, Fellow&MVEDeveloperCommented:
OK, here are my proposed specs for such a program (enhanced/generalized a bit beyond your comments):

• Allow specification of a maximum size (in MB) for each output archive file and/or a maximum number of input files to be stored in each output archive file (default is no maximum, i.e., all input files will be stored in a single archive file).

• Archive all files in a specified source folder, with an option to include subfolders (default is root folder only). Create multiple archive files, as needed, to meet the maximum size and/or count specification discussed above. Each output archive file must be independent, i.e., capable of being opened by itself without any requirement for the presence of the other archive files.

• Provide an option to specify the first component in the archive file names (default is the file name of the source folder).

• Provide an option to use a date/time stamp, with hyphens for separators (-YYYYMMDD_HHmmss-), as the second component in the archive file names (this guarantees unique names, since the time includes seconds).

• Use an N-digit number (where N is specifiable, with a default of 3), preceded by a hyphen, as the final component in the archive file names (e.g., -001, -002, -003, etc.).

• Allow specification of a destination folder where all archive files will be created.

• Use 7-Zip as the archiving tool.

• Provide choices for archive type: 7z, BZIP2, GZIP, TAR, WIM, XZ, ZIP (default is ZIP).

• Provide both a Command Line Interface (CLI) and Graphical User Interface (GUI).

Let me know if you think I missed anything, or got anything wrong. Regards, Joe
bitstreamincAuthor Commented:
Nailed it!
Bill PrewIT / Software Engineering ConsultantCommented:
Here's a starting point for you if you want to take a BAT approach, and zip based on number of files rather than a size threshold.  Adjust folders and settings near top.

@echo off
setlocal EnableDelayedExpansion

rem Define folders and options
set BaseDir=B:\EE\EE29081915\Files
set ZipDir=B:\EE\EE29081915\Zips
set ZipExe=C:\_pf\7-Zip\7z.exe
set IncludeSubDirs=N
set MaxFiles=10


rem Slightly different loops if subfolders included, process each folder
if "%IncludeSubDirs%" EQU "Y" (
    for /r "%BaseDir%" %%D in (.) do (
        call :ProcessDir "%%~dpnxD"
    )
) else (
    for /d %%D in ("%BaseDir%\*.*") do (
        call :ProcessDir "%%~D"
    )
)

exit /b

:ProcessDir [directory-path]
    rem Initialize variables
    set ZipNum=0
    set FileNum=0
    set FileList=

    rem Process all files in folder
    for %%F in ("%~1\*.*") do (
        rem Add to list
        set /a FileNum+=1
        set FileList=!FileList! "%%~F"

        rem When we hit max archive this group of filess
        if !FileNum! EQU %MaxFiles% (
            set /a ZipNum+=1
            "%ZipExe%" a "%ZipDir%\%~nx1-!ZipNum!.zip" !FileList! -tzip 1>NUL
            set FileNum=0
            set FileList=
        )
    )

    rem See if we have any files waiting to be zipped
    if !FileNum! GTR 0 (
        set /a ZipNum+=1
        "%ZipExe%" a "%ZipDir%\%~nx1-!ZipNum!.zip" !FileList! -tzip 1>NUL
    )

    exit /b

Open in new window



»bp
bitstreamincAuthor Commented:
The batch files works excellent with number for files.
Would number of bytes be possible to include in a zip instead of file?
Bill PrewIT / Software Engineering ConsultantCommented:
I considered number of bytes and then decided it was too much work for too little reward.  Here was my thinking.

There are two sizes that could be used: either the size of the files before zipping, or the size of the ZIP after compression is applied.  

The latter of these would be hard to impossible to do because you don't know the compressed size of the group of files until you ZIP them, you can't predict the size they will take in the ZIP.  So if we said make the ZIP files all 20MB or less, there's no way to know if a group of files will ZIP under that limit or not.  So it would be a massive trial and error to find combinations of files that ZIPped to the desired size.

In the other case, looking at sizes before ZIPping, that's almost as bad, unless you accept huge variances and under filled ZIP files.  Assuming the input files to be ZIPped are not all the same size, then working through them one at a time we would keep tallying a cumulative size, and when that hits the threshold, ZIP them up.  But you could have a few very small files first, all good, and then hit a 50MB file.  We check the tally, see it would put us over the threshold, and ZIP the small files, ending up with a very small ZIP file.  Or we include the last file that put us over, and this ZIP could be many times larger than the others.  Didn't feel like it would yield great results.

In theory you could take all the files at once, consider all their sizes and try to optimize the groupings to fall just under the max, but that would be a bit complex, and I wouldn't go near that in a BAT script.

Hope that helps you understand my thinking, not saying it's "right", just where I ended up and so I figured something was better than nothing and took a shot at the number of files approach.  Understand it may not meet your need.  Speaking of which, what is the motivation for this chopping up anyway?


»bp
bitstreamincAuthor Commented:
Compression isn't needed at all, so the file size tally can be done at the beginning.
I'm wondering if there is something that can be done, were it can sort by file size?

Sorting from largest to smallest and then just zip 50mb increments?

Is that something that can be done without a lot of work?
Bill PrewIT / Software Engineering ConsultantCommented:
No one file can be larger than 50MB?

And say you have 3 files of 30MB, and 3 files of 5MB.  You are okay with each of the 30MB files taking a separate ZIP, and the the three 5MB files taking another ZIP.  Whereas in theory they could have fit in a total of three ZIP files, this would use four?

So, should no compression be done during the zip command, that will process faster typically, but naturally the ZIP files could be larger.


»bp
bitstreamincAuthor Commented:
If you scan the folder in order of largest to smallest file size.
First 3 files are 30 megs and then the next 3 are 5mb.
The first 3 zips are each a 30mb file and then the next zip has 3 files of 5mb, that is fine.

That would work without compression.
Bill PrewIT / Software Engineering ConsultantCommented:
Okay, here is a version that should do what we have talked about.  Note that max file size is in bytes.

@echo off
setlocal EnableDelayedExpansion

rem Define folders and options (MaxSize is in BYTES)
set BaseDir=B:\EE\EE29081915\Files
set ZipDir=B:\EE\EE29081915\Zips
set ZipExe=C:\_pf\7-Zip\7z.exe
set IncludeSubDirs=Y
set MaxSize=52428800

rem Slightly different loops if subfolders included, process each folder
if "%IncludeSubDirs%" EQU "Y" (
    for /r "%BaseDir%" %%D in (.) do (
        call :ProcessDir "%%~dpnxD"
    )
) else (
    for /d %%D in ("%BaseDir%\*.*") do (
        call :ProcessDir "%%~D"
    )
)

exit /b

:ProcessDir [directory-path]
    rem Initialize variables
    set ZipNum=0
    set FileSize=0
    set FileList=

    rem Process all files in folder (sort largest to smallest)
    for /f "tokens=*" %%F in ('dir /b /o-sn /a-d "%~1\*.*"2^>NUL') do (

        rem Get file attributes using full path
        for %%G in ("%~1\%%~F") do (

            rem Add size to running total
            set /a FileSize+=%%~zG

            rem When we hit max size, archive this group of filess
            if !FileSize! GTR %MaxSize% (
                rem Zip these files
                set /a ZipNum+=1
                "%ZipExe%" a "%ZipDir%\%~nx1-!ZipNum!.zip" !FileList! -tzip -mx0 1>NUL

                rem Reset running total and file list for next archive
                set FileSize=%%~zG
                set FileList="%%~G"

            ) else (
                rem Still fits, add it to the list
                set FileList=!FileList! "%%~G"

            )

        )

    )

    rem See if we have any files waiting to be zipped
    if !FileSize! GTR 0 (
        set /a ZipNum+=1
        "%ZipExe%" a "%ZipDir%\%~nx1-!ZipNum!.zip" !FileList! -tzip -mx0 1>NUL
    )

    exit /b

Open in new window


»bp

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
bitstreamincAuthor Commented:
This is a winner,  thank you for all your help!
Great job on the batch files, it would of taken me days to write something half baked.
Once again, thank you for your time and great work!
Joe Winograd, Fellow&MVEDeveloperCommented:
Nailed it!
Good to know that we have the specs nailed down — and I see that Bill has come up with another one of his superb batch scripts! So I'll put the specs for the full program on the back-burner. Regards, Joe
Bill PrewIT / Software Engineering ConsultantCommented:
Welcome, glad that was useful.


»bp
bitstreamincAuthor Commented:
One issue I ran into.
It started failing on some directories.
So it would create files test01.zip, test02.zip, test04.zip
It would skip a number and the archives are missing half the files of a directory of 300 files.
So out of 300 files it would only zip 150 of them.
Could there be some issue with a directory that has over a few hundred files?
Bill PrewIT / Software Engineering ConsultantCommented:
Well, the only time it increments the zip number is right before the actual zipping, so it seems like if the number bumped up it should have zipped.  Only other thing could be if the zip is failing for some reason.  Try a test with out the 1>NUL on the end (as below) and see if it says anything interesting.  You will get a lot of output, but it will include the zip command output so look at those for possible problems.

set /a ZipNum+=1
"%ZipExe%" a "%ZipDir%\%~nx1-!ZipNum!.zip" !FileList! -tzip -mx0

Open in new window


»bp
bitstreamincAuthor Commented:
Files read from disk: 13
Archive size: 12580984 bytes (12 MiB)
Everything is Ok

I don't get individual errors on files.

Something odd going on, but it's not the script.
The files I am trying to zip up are tifs, pdf's etc and nothing odd in the filenames.
I was able to zip up a windows directory with no issues.
I have to look further into what is the issue with these files.
Bill PrewIT / Software Engineering ConsultantCommented:
Okay, keep me posted...


»bp
bitstreamincAuthor Commented:
Figured it out..
I used magic file renamer:  http://www.finebytes.com/mfr/
Used the "Cleaner" option to strip all odd character "!"#$%&'()*+,/:;<=>?@[]\^`{}|~_ "
including spaces.
Now it works like a champ!   Thanks for all your help.
Bill PrewIT / Software Engineering ConsultantCommented:
Ah yes, some special characters can upset the BAT script parser, it's not as flexible as some scripting languages...


»bp
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Software

From novice to tech pro — start learning today.