• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1725
  • Last Modified:

How do I split a huge binary file into smaller files then reassemble it?

I need to break down a 700 meg installation package which is basically an installation program that has two large msi install programs inside of it.  The installation package is being used as a container to hold the msi files which are binary files.  I need to use IBM Tivoli to push that package out to 15,000 computers at my job.  The problem is, I can not push a 700 meg install to that many devices because it will drag down the network.  

However, if I can break it down into something like 512 kb file chunks then I can get it pushed out to all those devices slowly over a few days as Tivoli will trickle it down to the PCs slowly.  The overall impact will be a lot lower with the smaller chunks because Tivoli will only allow 512 kb of data at one time per PC on the network; this is significantly better than 700 meg at one time on the network per PC.

Overall here is what I am trying to do:  Take a 700 meg .exe file and split it into many smaller 512 kb files.  Then use Tivoli to push all the 512 kb files to each PC.  Then run a VB .NET program that will read all the 512 kb files back into the oringal 700 meg file and execute it.

So, what I am looking for is how to use Visual Basic .NET to read a 700 meg binary file and output it into multiple 512 kb files (basically split the 700 meg file into many smaller files).  

Additionally, I need to know how to read all the 512 kb binary files and reassemble them back to their original 700 meg file.  
0
zonkerman
Asked:
zonkerman
  • 7
  • 5
  • 5
  • +1
1 Solution
 
weellioCommented:
winzip

action - span
they have a commandline version that should put them together,. and once together you should be able to script the executino and installation
0
 
zonkermanAuthor Commented:
Is there a free version of WinZip?  I need to have the files reassembled at each of the 15,000 computers.  If its free then I can look into it.  But if it costs then no way.  I went the winzip web site and the lowest priced product is $29.00.  $29 x 15,000 = $435,000, hehe I'll be thrown out of the computing group for suggesting that if the Winzip I need is not free.
This is why I am looking for a VB solution because whatever I can compile and run with a .net program does not cost anything to deploy per workstation.
0
 
weellioCommented:
Function SaveBinaryData(FileName, ByteArray)
  Const adTypeBinary = 1
  Const adSaveCreateOverWrite = 2
 
  'Create Stream object
  Dim BinaryStream
  Set BinaryStream = CreateObject("ADODB.Stream")
 
  'Specify stream type - we want To save binary data.
  BinaryStream.Type = adTypeBinary
 
  'Open the stream And write binary data To the object
  BinaryStream.Open
  BinaryStream.Write ByteArray
 
  'Save binary data To disk
  BinaryStream.SaveToFile FileName, adSaveCreateOverWrite
End Function
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
weellioCommented:
but seeing as i am an sms administrator what i'd do is probably recompile the msi's to where all the files are not stored within the actual msi, then create several smaller packages to deploy that copy all the files into the correct location.
0
 
clockwatcherCommented:
Imports System.IO

Module FileSplitAndReassemble

    Function ChopItUp(ByVal inputFile As String, ByVal chunkSize As Integer) As Integer

        Dim reader As New FileStream(inputFile, FileMode.Open)
        Dim fileIndex As Integer = 1
        Dim buffer(chunkSize) As Byte
        Dim writer As FileStream
        Dim bytesRead As Integer

        Do
            bytesRead = reader.Read(buffer, 0, chunkSize)
            If bytesRead > 0 Then
                writer = New FileStream(inputFile & "." & fileIndex, FileMode.Create)
                writer.Write(buffer, 0, bytesRead)
                writer.Close()
                fileIndex = fileIndex + 1
            End If
        Loop While bytesRead > 0

        reader.Close()

        ChopItUp = fileIndex - 1

    End Function

    Sub Reassemble(ByVal outputFile As String, ByVal pieces As Integer)

        Dim i As Integer
        Dim reader As FileStream
        Dim writer As New FileStream(outputFile, FileMode.CreateNew)
        Dim bytesRead As Integer
        Dim chunkSize As Integer = 4096
        Dim buffer(chunkSize) As Byte

        For i = 1 To pieces
            reader = New FileStream(outputFile & "." & i, FileMode.Open)
            Do
                bytesRead = reader.Read(buffer, 0, chunkSize)
                writer.Write(buffer, 0, bytesRead)
            Loop While bytesRead > 0
            reader.Close()
        Next

        writer.Close()

    End Sub

    Sub DoUsage()
        System.Console.WriteLine("USAGE:")
        System.Console.WriteLine("Splits/reassembles a file into/from chunks.")
        System.Console.WriteLine()
        System.Console.WriteLine("split <filename>")
        System.Console.WriteLine("   creates a set of files named filename.index and returns the number of files created")
        System.Console.WriteLine()
        System.Console.WriteLine("reassemble <filename> <numOfFiles>")
        System.Console.WriteLine("   combines files name filename.index into filename")
        System.Console.WriteLine()

    End Sub

    Sub Main(ByVal args() As String)

        Try
            If args(0) = "split" And args.Length = 2 Then
                Dim pieces As Integer = ChopItUp(args(1), 524288)
                System.Console.WriteLine("Created {0} pieces", pieces)
                Exit Sub
            End If

            If args(0) = "reassemble" And args.Length = 3 Then
                Reassemble(args(1), CInt(args(2)))
                Exit Sub
            End If

        Catch e As System.Exception
            System.Console.WriteLine("Error: {0}" & vbCrLf & vbCrLf, e.Message)
        End Try

        DoUsage()

    End Sub

End Module
0
 
amit_gCommented:
Use any of mostly free utilities to split the file. You can do a search on download.com for several options, many of them are for free.

http://www.download.com/JR-Split-File/3000-2248_4-10207367.html?tag=lst-0-1

The join back together is simple DOS command. Most split utilities would generate a batch file with the DOS command(s) in it. You don't need any special software to join.
0
 
zonkermanAuthor Commented:
Thank you everyone for your attention to this puzzle.  I know these kinds of things take time and sometimes they are not fun.  I'd like to give a special thanks to Clockwatcher for his contribution since it was the only one that addressed the request for both creating multiple files from a single large one and also for his efforts in trying to provide a solution for reassembling them back.

When I tested Clockwatchers solution, I found that the reassembly did not take place; most likely due to the offset start byte for the write being a constant value of zero.  In the end, I ended up using the .NET BinaryRead and BinaryWrite classes to solve the problem.  Thanks to Clockwatcher's efforts though, I was able to borrow from much of his structure while implementing the binary classes I stated above.

I need a little more time to test the implementation of the BinaryWrite and BinaryRead classes but so far I think I have this solved now.  Since Clockwatcher's framework did provided a running start for the changes I had to make, I'm going to proceed to accept his solution as the one that provided the most information that helped me figure out what was needed.
Once again, thank you all.
0
 
amit_gCommented:
Unless you are doing something very special and specific to your situation you should not "reinvent the wheel" :)
0
 
zonkermanAuthor Commented:
LOL,
I'm new to participating in Expert's Exchange so I'm going to be familiarizing as to how members here operate.  So, amit q, you have put forth the effort to resurrect this case by inferring that what I have done here is to perhaps "reinvent the wheel".  Please substantiate your comment by specifying exactly what you mean.  What element of this case did you find brings you to say that the effort here has gone toward reinventing the wheel?
0
 
weellioCommented:
i think he is saying that since there are already existing compiled programs out there that do this, there is nor reason for you to create your own code.

although it is always cheaper to create your own code.
0
 
amit_gCommented:
First of all, welcome to EE.

The way you described, it seems all you need is to split the file for easy distribution to several clients. This is very common and several alternatives already exist many of those are free. So it would be much better to use some of the existing solution rather than to write your own. Since those existing solutions have been used by several people for a longer period of times, you know that the solution is well tested and would more likely to work flawlessly vs. any custom solution you write. Even though it is pretty simple to split the file, when one gets into it, there are situations that one needs to take care of - obviously most of those would already been addressed into existing solution as someone would already have come across it and reported it. Another thing to note is that to reassembly the splitted files back to one, we don't need any custom solution as the o/s by itself provides a way. In fact most existing solutions generate a batch file that utilizes this method to assemble back the split files. Another benefit of using existing alternative is that some do provide CRC check to confirm the joined file. In short it is far better to use a solution that has been developed specifically for that purpose, has been tested by a wide audience on various platforms vs. developing you own - that is why the term "reinventing the wheel" :)
0
 
zonkermanAuthor Commented:
Members should not make assumptions about the needs to those requesting technical assistance.  I expected more from those acting on behalf of Experts Exchange.  I will take a breif moment here to provide some level of detail that warranted this case in the hopes that those seeking to assist others will realize it is inappropriate to make comments like "you should not reinvent the wheel".

1)  Although a commeircial product, such as WinZip, does exist that could maybe have handled this situation, it was simply out of the question considering the solution had to be deployed and implemented at over 15,000 PCs, making the cost of the solution unacceptable.

2)  Although there are several FREE utilities that could probably have been used to remedy this situation, in general, small utilities produced by unknown organizations or agency that take no accountability for their product or make no warranties are typically too risky to use as part of a major corporate solution that could affect more than 15,000 computers and potentially thousands of said company's customers.

3)  Its all about accountability.  When proposed solutions fail, corporate inquiries want to know what was done to minimize risks and who was involved.  The big question, was due diligence used to avoid potential problems.  Your chances of coming out good before a panel of management appointed investigators after saying the failure was due to your decision to use a shareware piece or some FREE unsupported software you thought would do the trick are slim.

4)  Its about support.  When such utilities are built with company resources (employees and equipment), several things emerge:  
  A) Accountability is on those who approve the construction of said utility and those who build it,
  B)  If anything fails with the utility, resources can be made available to respond seven days a week, 24 hours a day if needed (this is not what you get for a FREE utility).  
  C) When the corporate environment changes, the support organization for the utility within the company can adapt their constructed utility as often as needed to keep pace with the dynamics of the company.  In most cases you cannot get the builder of a small FREE utility to jump to your requests just because your environment is changing.

Not all software needs are built in-house at my company.  In many cases, we contract software development out to other companies.  In those cases, there is a Service Level Agreement where the company hired for services, such as application development, is repsonsible for what they deliver.  Our company is protected by a contracted agreement that they will do their part to provide maintenance and support for what has been built.  You will most likely not get Service Level Agreements by using FREE software or unsupported shareware.

The solution that was sought out in this thread is for a very important project.  Although the exact answer needed was not aquired, I did get enough information to build what was needed.  The utility that I build will be tested in a lab of over 30 computers, consisting of laptops, desktops, and various models of such types, and 5 experienced software testers.  Anything in the utility that is found to fail under the testing will be reviewed and modified by me.  That level of corrective measure you will most likely not get from a FREE utilty in a timely manner.

Hopefully, I have described enough here so that others will consider their statements carefully.  We don't all have the time explain our reasons for seeking assistance, and we should not have to.  That being said, those members trying to help others should not pass judgements on what requesting members are doing; statements like "you should not reinvent the wheel" are arrogant when made by someone not knowing the facts or signifcance of the case.



0
 
amit_gCommented:
I don't understand why you are taking it totally out of context. I have only given you an alternative and a reason for that alternative. You want to use that alternative or not is totally your decision. The "reinventing the wheel" comment was in humor and was not at all to offend you. As a matter of fact this statement is used in most professional meetings where solutions are sought where people would often say - "lets look into solution xyz because we don't want to reinvent the wheel".

Whatever you have written against using a free utility is valid but you failed to notice one point that I have been mentioning all along - splitting needs to happen only on one machine and you don't need any software or utility to join. Joining the file is built into the windows. So you don't need to worry about 15000 installations as those are not at all needed. If you are going to use lab of 30 computers and 5 experienced software tester to build a utility to split and join files, you are wasting those precious resources but again those are your resources and you can do whatever you want with those.

When you post a question in EE, you may get more than one alternative. That is the beauty of this forum. We try to provide not only "how to do this" kind of solutions but also provide "how to do this in a better way". Several times, we have taken a questioner in totally different directions only to save a lot of time and efforts because in the beginning the questioner did not even think about those possibilities.

With all that said, which solution you like and choose to implement is your call.
0
 
zonkermanAuthor Commented:
amit,
You must of missed my original note at the very top that says the following:

"Overall here is what I am trying to do:  Take a 700 meg .exe file and split it into many smaller 512 kb files.  Then use Tivoli to push all the 512 kb files to each PC.  Then run a VB .NET program that will read all the 512 kb files back into the oringal 700 meg file and execute it."

As you can see, I did not say splitting will take place at the workstation.  The plan is to reassemble at the workstation.  Here is a little more detail to help you understand:

1)  Using Wise Package Studio Pro, embed two large MSI programs into a single .exe file which will result in a file of about 700 megs
2)  Use a VB .NET tool to split the 700 meg .exe file into multipl 512 kb files
3)  Use Tiovli to collect all the 512 kb files and batch them into a single push job
4)  Push the job consisting of many 512 kb files to all target workstations (5,000 PCs at a time)
5)  Target workstations will recieve the 512 kb files slowly over a few days
6)  Use a Tivoli inventory report to identify all workstations that have recieved the required files
7)  Push a tiny utility that will cause the end workstation available sotware windows to display that a large software update is ready for them to install that could take 2 - 3 hours to run.
8)  The end user will select from the software window when they are ready to run the major update, maybe before they go home.
9)  When the user elects to run the major update, it will trigger the VB utility, which was part of the many files pushed to each workstation, to begin reassembling the 512 kb file parts back to the original .exe package that was created by Wise Package Studio Pro.  This takes place on each PC when a user runs the update from their desktop available software window.
10)  Wise Package Studio Pro created .exe package files already comes with built in CRC so I don't need to write one, we will know if there is a problem automatically based on other technologies we are using.

And there you have it.  I am not intending to split at each workstation, but I am intending to reassemble all the parts Tivoli pushed out.  This may not make sense to you if you don't know what Tivoil is but it is something like SMS.




0
 
amit_gCommented:
I think I got what your are trying to do but my approach towards achieving that is little different than yours.

>>As you can see, I did not say splitting will take place at the workstation.

Neither did I. What I said is that you need a tool/utility (custom or pre-built) to split but you don't need any tool/utility to reassemble. That is why whatever you need would run only on one machine where you prepare the package for installation. The reassembly could be a simple batch file with

copy /b HugeFile.msi.split.000+HugeFile.msi.split.001+HugeFile.msi.split.002 HugeFile.msi

Obviously your batch file (to join) would be multi-line as you will have some 1400 part file to assemble and there is a limit to how many parameters you can have in one line.

So you are left with need for a tool to split only and for that too I would use a freely available one (as I have in the past) if I were to do this and that is why I recommended that.
0
 
zonkermanAuthor Commented:
Ah ha. Okay I see what you were talking about.  So, if I modify the file splitter so that all the extensions are the same like so:
hugeFile.msi.001.part
hugeFile.msi.002.part
hugeFile.msi.003.part
hugeFile.msi.004.part

Then I can use the following command, with the .part extension as a wild card, in a .cmd file to reassemble:
copy /b *.part prod_update.exe

I'll admit that is useful and I did not know about the /b switch.  Thanks for the info.


0
 
weellioCommented:
0
 
zonkermanAuthor Commented:
For now, the following code block is working well for me to split the file.  Its structure is from clockwatcher although I rewrote a good portion of it to use BinaryRead and BinaryWrite Classes which he did not use.  As you can see, it is very little code to maintain and easy to modify for special needs:

    Function SplitFileToPieces(ByVal FileToSplit As String) As Integer
        Dim BinReader As New BinaryReader(File.Open(FileToSplit, IO.FileMode.Open, IO.FileAccess.Read))
        Dim BinWriter As BinaryWriter
        Dim fileIndex As Integer = 1
        Dim BytesInputBuff(gFileSizeForParts) As Byte
        Do
            Array.Clear(BytesInputBuff, 0, BytesInputBuff.Length)
            BytesInputBuff = BinReader.ReadBytes(gFileSizeForParts)
            If BytesInputBuff.Length > 0 Then
                BinWriter = New BinaryWriter(File.Open(FileToSplit & "." & Format$(fileIndex, "0000") & ".part", FileMode.Append, FileAccess.Write))
                BinWriter.Write(BytesInputBuff)
                BinWriter.Close()
                fileIndex = fileIndex + 1
            End If
        Loop While BytesInputBuff.Length > 0

        BinReader.Close()

        SplitFileToPieces = fileIndex - 1
    End Function
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 7
  • 5
  • 5
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now