We help IT Professionals succeed at work.

Check out this week's podcast, "Dairy Farms to Databases: Community's Hand in Technology"Listen Now

x

Problem Writing/Reading a File > 2GB - HELP :|

BladeSA
BladeSA asked
on
361 Views
Last Modified: 2010-05-03
Hi All

Though these are my last points, I have a difficult problem.

I'm using the Visual Basic Put and Get commands via Binary as a file to read data and write Data to a file (See code below).

The problem is that when I reach a file that is > 2GB's it returns an error. I'm writing a form that has to backup security recordings of up to 10GB or more and I need to be able to monitor the progress of the copy. Though below function has all that, I'm battling to use my written function because after 2GB's it returns an error! I can't figure the way around this problem... Please Help

Option Explicit
Public Const GENERIC_READ = &H80000000
Public Const FILE_SHARE_READ = &H1
Public Const OPEN_EXISTING = 3
Public Const FILE_TYPE_CHAR = &H2
Public Const FILE_TYPE_DISK = &H1
Public Const FILE_TYPE_PIPE = &H3
Public Const FILE_TYPE_UNKNOWN = &H0
Public Declare Function GetFileType Lib "kernel32" (ByVal hFile As Long) As Long
Public Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" (ByVal lpFileName As String, ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, lpSecurityAttributes As Any, ByVal dwCreationDisposition As Long, ByVal dwFlagsAndAttributes As Long, ByVal hTemplateFile As Long) As Long
Public Declare Function GetFileSizeEx Lib "kernel32" (ByVal hFile As Long, lpFileSize As Currency) As Boolean
Public Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long

Private Sub CopyFile(Source As String, Destination As String)
   
    On Error GoTo CopyErr ''If there is an error sort it out at the bottom
    '' Variable Declaration
    Dim SrcSize
    Dim ReadSource As Currency, WriteDestination As Currency
    Dim Chunk As String
    Dim GetBytes As Currency
    Dim CopiedBytes As Currency
   
    Source = TxtSrc
    Destination = TxtDest
   
    SrcSize = SizeOfFile(Source) '' Size of File function provide by Tim by my last query - Thanks Bro! :)
    '' To show copy progress
    ProgressBar1.Min = 0              '' Bar on Form
    ProgressBar1.Max = SrcSize    '' Bar on Form
   
    ReadSource = 1
    WriteDestination = 2

    Open Source For Binary As ReadSource
    Open Destination For Binary As WriteDestination
   
    GetBytes = 131072 '128kb - Reads 128K  at a time and then writes it to file

    CopiedBytes = 0

    ProgressBar1.Value = 0

    LblPercent.Caption = "0"  ''Label on form
    ''Start the copy process
    Do While CopiedBytes < SrcSize
        If GetBytes < (SrcSize - CopiedBytes) Then
            Chunk = Space(GetBytes)
            Get #ReadSource, , Chunk
        Else
            Chunk = Space(SrcSize - CopiedBytes)
            Get #ReadSource, , Chunk
        End If
        CopiedBytes = CopiedBytes + Len(Chunk)
        ProgressBar1.Value = CopiedBytes
        LblPercent.Caption = Int(CopiedBytes / SrcSize * 100)
        LblPercent.Refresh
        Form1.Caption = CopiedBytes / 1000000
        Put #WriteDestination, , Chunk
    Loop
   
    ProgressBar1.Value = 0
   
    GoTo Ex
   
CopyErr:
    MsgBox Err.Description, vbCritical, "Error"
   
Ex:
    Close #ReadSource
    Close #WriteDestination
   
    CmdCopy.Enabled = True
End Sub
Comment
Watch Question

2 gigs?
I am afraid even enlarging memory (ram) would not help: two gigs is the limit of strings, for example, and even
if I cannot swear on this, I am quite sure it is also  the limit for Put.... How where you able to record files up to 10 gigs?
In any case, if you have Nt or win2k (Requires Windows NT 4.0 or later; Win9x/ME: Not supported), give the following a try:
'in a form (Form1)
Private Sub Form_Load()
    'KPD-Team 2001
    'URL: http://www.allapi.net/
    'E-Mail: KPDTeam@Allapi.net
    Dim Ret As Long
    'set the graphics mode to persistent
    Me.AutoRedraw = True
    'print some text
    Me.Print "Click the form to abort the filecopy"
    'show the form
    Me.Show
    'start copying
    Ret = CopyFileEx("c:\verybigfile.ext", "c:\copy.ext", AddressOf CopyProgressRoutine, ByVal 0&, bCancel, COPY_FILE_RESTARTABLE)
    'show some text
    Me.Print "Filecopy completed " + IIf(Ret = 0, "(ERROR/ABORTED)", "successfully")
End Sub
Private Sub Form_Click()
    'cancel filecopy
    bCancel = 1
End Sub
'in a module
Public Const PROGRESS_CANCEL = 1
Public Const PROGRESS_CONTINUE = 0
Public Const PROGRESS_QUIET = 3
Public Const PROGRESS_STOP = 2
Public Const COPY_FILE_FAIL_IF_EXISTS = &H1
Public Const COPY_FILE_RESTARTABLE = &H2
Public Declare Function CopyFileEx Lib "kernel32.dll" Alias "CopyFileExA" (ByVal lpExistingFileName As String, ByVal lpNewFileName As String, ByVal lpProgressRoutine As Long, lpData As Any, ByRef pbCancel As Long, ByVal dwCopyFlags As Long) As Long
Public bCancel As Long
Public Function CopyProgressRoutine(ByVal TotalFileSize As Currency, ByVal TotalBytesTransferred As Currency, ByVal StreamSize As Currency, ByVal StreamBytesTransferred As Currency, ByVal dwStreamNumber As Long, ByVal dwCallbackReason As Long, ByVal hSourceFile As Long, ByVal hDestinationFile As Long, ByVal lpData As Long) As Long
    'adjust the caption
    Form1.Caption = CStr(Int((TotalBytesTransferred * 10000) / (TotalFileSize * 10000) * 100)) + "% complete..."
    'allow user input
    DoEvents
    'continue filecopy
    CopyProgressRoutine = PROGRESS_CONTINUE
End Function

Author

Commented:
Hi Cimperiali,

Thanx, I'll give it a try... I'm not at my coding machine right now, will only be able to be at it tomorrow... but right now I'm desperate to try anything. The "how" we get 10GB file is from a custom built recorder which has its own OS etc... but with the network capability on the unit, u can read the files and some of them reach 10GB.
Found something of interesting on this matter:
http://www.puremotion.com/resources/technical/2gbfilelimit/
it is about recording Avi, but intersting thing is: Fat seems to matter!
if you're using Ntfs, you should have no limits
If you're using Fat32, max should be  2-4 gigs (depending if you can use
"unsigned numbers"- and in Vb you can not, thus 2 gigs should be the limit!)

Author says:
Windows NT, 2000 and XP allow files of effectively unlimited size- as long as you have choosen the corred type of Fat.


I am not sure this can explain everithing, but it seems a reasonable explanation...Which Os are you using?
And which type of Fat do you have?
Even better: Microsoft "How to Manage Disk Capacity and Usage By Using Windows 2000"
http://support.microsoft.com/?kbid=300979#toc 

Author

Commented:
I'm programming on Windows 2000 Professional, and the source drive I'm copying from is NTFS, the destination Drive is also NTFS

Commented:
There is limitation on put and get with binary read and write.

You CANNOT bypass this.

without binary (text), you can write to file over 2GB

You may end up to break your file into serval sub files
Beg your pardon:
this may be more intersting (even if not directly related: it is hpow to enlarge paging),
but previous ones gives you nothing (sorry!)
http://support.microsoft.com/?kbid=237740
>without binary (text), you can write to file over 2GB

Thus, how?

Author

Commented:
EDDYKT,

The interesting part is if I break it up into several files, how do I put them back together after copying the data accross?

Windows itself is able to copy a file greater than 2GB, there must be a way I can do it to, with having a possibility of seeing how many I've copied while its progressing....

Commented:
Use print and you can write more than 2 GB

Commented:
>>Windows itself is able to copy a file greater than 2GB, there must be a way I can do it to, with having a possibility of seeing how many I've copied while its progressing....


They are not coping by record, they read certain byte and paste to new file

Author

Commented:
right, is there a way I can do the same as the windows way? Am I asking too much here?

Commented:
Have you tried to use FileSystemObject?

OR C++ with 64 bytes

Author

Commented:
Unfortunately not C++ literal yet :S

and I did try FilesystemObject before, however I remember problems getting the fileprogress copied bytes... that is why I went this rout, copy via records cos thus I could watch the progress of the copied file...

Ie, I have to write a log everytime, 256M is copied and do a live tally... (Clients have wierd requirements for programs sometimes :S)

Commented:
I think it may also be a limitation on 32bit (binary) since Access only supports up to 2GB and same as MSDE.

Author

Commented:
EDDYKT, is there a know way to copy source to destination in VB the same way Windows copies and get the copied progress of bytes whilst its coping?

Commented:
I don't quite know how to do it because limitiation on VB as well

VB doesn't support unsigned and this is one of reason the file is limited on 2 GB

IF you use c++, I believe you can copy at least 4 GB with 32 bit system

Author

Commented:
EDDYKT, I think I might have worked a solution around it... will get back to u shortly....

Commented:
I would like to know

8->
What about...
Using fso to copy, and while copying using filelen() or lof() on file to know how much you've copied (in a timer event...) to dipslay what users want every 256 mb.... ??
(by the way: it should be "filelen" in timer, as you will not be allowed to open the file while it is being copied...
Tested...Inside same exe, it seems fso freeze your process...May be using two exes (one to check filelen, one to perform the copy)?
Seems as if you're out of lucky...:
I tried with two exes, but seems as if Fso allocate space for file before copying inside values -ie: immediately second exe tells the whole file is copiedm while first exe is still engaged - I tested with a 128mb file.... Maybe  Api solution works better...

Author

Commented:
Right, I tried my theory. I read on a site the copying the way I coded above, if one open and closed the file continuously and seeked end of file that it would break the 2GB limit. Testing the throry it works.... but it overrights the last few bytes continuously till the file is copied, which means it doesn't really solve the problem. So I am back at square one :(

Have you guys thought of any other solutions :S

Commented:
It could possibly be you're filesystem, some filesystems (e.g. FAT) do not support files greater than 2 GB's. You should check you're filesystem and look for specifications on the internet. Every filesystem has it limitations!!!
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
I apologize... I haven't had the chance to look at the solution.... EDDYKT desserves the points... I'll release them to him now
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.