BladeSA
asked on
Problem Writing/Reading a File > 2GB - HELP :|
Hi All
Though these are my last points, I have a difficult problem.
I'm using the Visual Basic Put and Get commands via Binary as a file to read data and write Data to a file (See code below).
The problem is that when I reach a file that is > 2GB's it returns an error. I'm writing a form that has to backup security recordings of up to 10GB or more and I need to be able to monitor the progress of the copy. Though below function has all that, I'm battling to use my written function because after 2GB's it returns an error! I can't figure the way around this problem... Please Help
Option Explicit
Public Const GENERIC_READ = &H80000000
Public Const FILE_SHARE_READ = &H1
Public Const OPEN_EXISTING = 3
Public Const FILE_TYPE_CHAR = &H2
Public Const FILE_TYPE_DISK = &H1
Public Const FILE_TYPE_PIPE = &H3
Public Const FILE_TYPE_UNKNOWN = &H0
Public Declare Function GetFileType Lib "kernel32" (ByVal hFile As Long) As Long
Public Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" (ByVal lpFileName As String, ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, lpSecurityAttributes As Any, ByVal dwCreationDisposition As Long, ByVal dwFlagsAndAttributes As Long, ByVal hTemplateFile As Long) As Long
Public Declare Function GetFileSizeEx Lib "kernel32" (ByVal hFile As Long, lpFileSize As Currency) As Boolean
Public Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
Private Sub CopyFile(Source As String, Destination As String)
On Error GoTo CopyErr ''If there is an error sort it out at the bottom
'' Variable Declaration
Dim SrcSize
Dim ReadSource As Currency, WriteDestination As Currency
Dim Chunk As String
Dim GetBytes As Currency
Dim CopiedBytes As Currency
Source = TxtSrc
Destination = TxtDest
SrcSize = SizeOfFile(Source) '' Size of File function provide by Tim by my last query - Thanks Bro! :)
'' To show copy progress
ProgressBar1.Min = 0 '' Bar on Form
ProgressBar1.Max = SrcSize '' Bar on Form
ReadSource = 1
WriteDestination = 2
Open Source For Binary As ReadSource
Open Destination For Binary As WriteDestination
GetBytes = 131072 '128kb - Reads 128K at a time and then writes it to file
CopiedBytes = 0
ProgressBar1.Value = 0
LblPercent.Caption = "0" ''Label on form
''Start the copy process
Do While CopiedBytes < SrcSize
If GetBytes < (SrcSize - CopiedBytes) Then
Chunk = Space(GetBytes)
Get #ReadSource, , Chunk
Else
Chunk = Space(SrcSize - CopiedBytes)
Get #ReadSource, , Chunk
End If
CopiedBytes = CopiedBytes + Len(Chunk)
ProgressBar1.Value = CopiedBytes
LblPercent.Caption = Int(CopiedBytes / SrcSize * 100)
LblPercent.Refresh
Form1.Caption = CopiedBytes / 1000000
Put #WriteDestination, , Chunk
Loop
ProgressBar1.Value = 0
GoTo Ex
CopyErr:
MsgBox Err.Description, vbCritical, "Error"
Ex:
Close #ReadSource
Close #WriteDestination
CmdCopy.Enabled = True
End Sub
Though these are my last points, I have a difficult problem.
I'm using the Visual Basic Put and Get commands via Binary as a file to read data and write Data to a file (See code below).
The problem is that when I reach a file that is > 2GB's it returns an error. I'm writing a form that has to backup security recordings of up to 10GB or more and I need to be able to monitor the progress of the copy. Though below function has all that, I'm battling to use my written function because after 2GB's it returns an error! I can't figure the way around this problem... Please Help
Option Explicit
Public Const GENERIC_READ = &H80000000
Public Const FILE_SHARE_READ = &H1
Public Const OPEN_EXISTING = 3
Public Const FILE_TYPE_CHAR = &H2
Public Const FILE_TYPE_DISK = &H1
Public Const FILE_TYPE_PIPE = &H3
Public Const FILE_TYPE_UNKNOWN = &H0
Public Declare Function GetFileType Lib "kernel32" (ByVal hFile As Long) As Long
Public Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" (ByVal lpFileName As String, ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, lpSecurityAttributes As Any, ByVal dwCreationDisposition As Long, ByVal dwFlagsAndAttributes As Long, ByVal hTemplateFile As Long) As Long
Public Declare Function GetFileSizeEx Lib "kernel32" (ByVal hFile As Long, lpFileSize As Currency) As Boolean
Public Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
Private Sub CopyFile(Source As String, Destination As String)
On Error GoTo CopyErr ''If there is an error sort it out at the bottom
'' Variable Declaration
Dim SrcSize
Dim ReadSource As Currency, WriteDestination As Currency
Dim Chunk As String
Dim GetBytes As Currency
Dim CopiedBytes As Currency
Source = TxtSrc
Destination = TxtDest
SrcSize = SizeOfFile(Source) '' Size of File function provide by Tim by my last query - Thanks Bro! :)
'' To show copy progress
ProgressBar1.Min = 0 '' Bar on Form
ProgressBar1.Max = SrcSize '' Bar on Form
ReadSource = 1
WriteDestination = 2
Open Source For Binary As ReadSource
Open Destination For Binary As WriteDestination
GetBytes = 131072 '128kb - Reads 128K at a time and then writes it to file
CopiedBytes = 0
ProgressBar1.Value = 0
LblPercent.Caption = "0" ''Label on form
''Start the copy process
Do While CopiedBytes < SrcSize
If GetBytes < (SrcSize - CopiedBytes) Then
Chunk = Space(GetBytes)
Get #ReadSource, , Chunk
Else
Chunk = Space(SrcSize - CopiedBytes)
Get #ReadSource, , Chunk
End If
CopiedBytes = CopiedBytes + Len(Chunk)
ProgressBar1.Value = CopiedBytes
LblPercent.Caption = Int(CopiedBytes / SrcSize * 100)
LblPercent.Refresh
Form1.Caption = CopiedBytes / 1000000
Put #WriteDestination, , Chunk
Loop
ProgressBar1.Value = 0
GoTo Ex
CopyErr:
MsgBox Err.Description, vbCritical, "Error"
Ex:
Close #ReadSource
Close #WriteDestination
CmdCopy.Enabled = True
End Sub
ASKER
Hi Cimperiali,
Thanx, I'll give it a try... I'm not at my coding machine right now, will only be able to be at it tomorrow... but right now I'm desperate to try anything. The "how" we get 10GB file is from a custom built recorder which has its own OS etc... but with the network capability on the unit, u can read the files and some of them reach 10GB.
Thanx, I'll give it a try... I'm not at my coding machine right now, will only be able to be at it tomorrow... but right now I'm desperate to try anything. The "how" we get 10GB file is from a custom built recorder which has its own OS etc... but with the network capability on the unit, u can read the files and some of them reach 10GB.
Found something of interesting on this matter:
http://www.puremotion.com/resources/technical/2gbfilelimit/
it is about recording Avi, but intersting thing is: Fat seems to matter!
if you're using Ntfs, you should have no limits
If you're using Fat32, max should be 2-4 gigs (depending if you can use
"unsigned numbers"- and in Vb you can not, thus 2 gigs should be the limit!)
Author says:
Windows NT, 2000 and XP allow files of effectively unlimited size- as long as you have choosen the corred type of Fat.
I am not sure this can explain everithing, but it seems a reasonable explanation...Which Os are you using?
And which type of Fat do you have?
http://www.puremotion.com/resources/technical/2gbfilelimit/
it is about recording Avi, but intersting thing is: Fat seems to matter!
if you're using Ntfs, you should have no limits
If you're using Fat32, max should be 2-4 gigs (depending if you can use
"unsigned numbers"- and in Vb you can not, thus 2 gigs should be the limit!)
Author says:
Windows NT, 2000 and XP allow files of effectively unlimited size- as long as you have choosen the corred type of Fat.
I am not sure this can explain everithing, but it seems a reasonable explanation...Which Os are you using?
And which type of Fat do you have?
Even better: Microsoft "How to Manage Disk Capacity and Usage By Using Windows 2000"
http://support.microsoft.com/?kbid=300979#toc
http://support.microsoft.com/?kbid=300979#toc
ASKER
I'm programming on Windows 2000 Professional, and the source drive I'm copying from is NTFS, the destination Drive is also NTFS
There is limitation on put and get with binary read and write.
You CANNOT bypass this.
without binary (text), you can write to file over 2GB
You may end up to break your file into serval sub files
You CANNOT bypass this.
without binary (text), you can write to file over 2GB
You may end up to break your file into serval sub files
Beg your pardon:
this may be more intersting (even if not directly related: it is hpow to enlarge paging),
but previous ones gives you nothing (sorry!)
http://support.microsoft.com/?kbid=237740
this may be more intersting (even if not directly related: it is hpow to enlarge paging),
but previous ones gives you nothing (sorry!)
http://support.microsoft.com/?kbid=237740
>without binary (text), you can write to file over 2GB
Thus, how?
Thus, how?
ASKER
EDDYKT,
The interesting part is if I break it up into several files, how do I put them back together after copying the data accross?
Windows itself is able to copy a file greater than 2GB, there must be a way I can do it to, with having a possibility of seeing how many I've copied while its progressing....
The interesting part is if I break it up into several files, how do I put them back together after copying the data accross?
Windows itself is able to copy a file greater than 2GB, there must be a way I can do it to, with having a possibility of seeing how many I've copied while its progressing....
Use print and you can write more than 2 GB
>>Windows itself is able to copy a file greater than 2GB, there must be a way I can do it to, with having a possibility of seeing how many I've copied while its progressing....
They are not coping by record, they read certain byte and paste to new file
They are not coping by record, they read certain byte and paste to new file
ASKER
right, is there a way I can do the same as the windows way? Am I asking too much here?
Have you tried to use FileSystemObject?
OR C++ with 64 bytes
OR C++ with 64 bytes
ASKER
Unfortunately not C++ literal yet :S
and I did try FilesystemObject before, however I remember problems getting the fileprogress copied bytes... that is why I went this rout, copy via records cos thus I could watch the progress of the copied file...
Ie, I have to write a log everytime, 256M is copied and do a live tally... (Clients have wierd requirements for programs sometimes :S)
and I did try FilesystemObject before, however I remember problems getting the fileprogress copied bytes... that is why I went this rout, copy via records cos thus I could watch the progress of the copied file...
Ie, I have to write a log everytime, 256M is copied and do a live tally... (Clients have wierd requirements for programs sometimes :S)
I think it may also be a limitation on 32bit (binary) since Access only supports up to 2GB and same as MSDE.
ASKER
EDDYKT, is there a know way to copy source to destination in VB the same way Windows copies and get the copied progress of bytes whilst its coping?
I don't quite know how to do it because limitiation on VB as well
VB doesn't support unsigned and this is one of reason the file is limited on 2 GB
IF you use c++, I believe you can copy at least 4 GB with 32 bit system
VB doesn't support unsigned and this is one of reason the file is limited on 2 GB
IF you use c++, I believe you can copy at least 4 GB with 32 bit system
ASKER
EDDYKT, I think I might have worked a solution around it... will get back to u shortly....
I would like to know
8->
8->
What about...
Using fso to copy, and while copying using filelen() or lof() on file to know how much you've copied (in a timer event...) to dipslay what users want every 256 mb.... ??
Using fso to copy, and while copying using filelen() or lof() on file to know how much you've copied (in a timer event...) to dipslay what users want every 256 mb.... ??
(by the way: it should be "filelen" in timer, as you will not be allowed to open the file while it is being copied...
Tested...Inside same exe, it seems fso freeze your process...May be using two exes (one to check filelen, one to perform the copy)?
Seems as if you're out of lucky...:
I tried with two exes, but seems as if Fso allocate space for file before copying inside values -ie: immediately second exe tells the whole file is copiedm while first exe is still engaged - I tested with a 128mb file.... Maybe Api solution works better...
I tried with two exes, but seems as if Fso allocate space for file before copying inside values -ie: immediately second exe tells the whole file is copiedm while first exe is still engaged - I tested with a 128mb file.... Maybe Api solution works better...
ASKER
Right, I tried my theory. I read on a site the copying the way I coded above, if one open and closed the file continuously and seeked end of file that it would break the 2GB limit. Testing the throry it works.... but it overrights the last few bytes continuously till the file is copied, which means it doesn't really solve the problem. So I am back at square one :(
Have you guys thought of any other solutions :S
Have you guys thought of any other solutions :S
It could possibly be you're filesystem, some filesystems (e.g. FAT) do not support files greater than 2 GB's. You should check you're filesystem and look for specifications on the internet. Every filesystem has it limitations!!!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I apologize... I haven't had the chance to look at the solution.... EDDYKT desserves the points... I'll release them to him now
I am afraid even enlarging memory (ram) would not help: two gigs is the limit of strings, for example, and even
if I cannot swear on this, I am quite sure it is also the limit for Put.... How where you able to record files up to 10 gigs?
In any case, if you have Nt or win2k (Requires Windows NT 4.0 or later; Win9x/ME: Not supported), give the following a try:
'in a form (Form1)
Private Sub Form_Load()
'KPD-Team 2001
'URL: http://www.allapi.net/
'E-Mail: KPDTeam@Allapi.net
Dim Ret As Long
'set the graphics mode to persistent
Me.AutoRedraw = True
'print some text
Me.Print "Click the form to abort the filecopy"
'show the form
Me.Show
'start copying
Ret = CopyFileEx("c:\verybigfile
'show some text
Me.Print "Filecopy completed " + IIf(Ret = 0, "(ERROR/ABORTED)", "successfully")
End Sub
Private Sub Form_Click()
'cancel filecopy
bCancel = 1
End Sub
'in a module
Public Const PROGRESS_CANCEL = 1
Public Const PROGRESS_CONTINUE = 0
Public Const PROGRESS_QUIET = 3
Public Const PROGRESS_STOP = 2
Public Const COPY_FILE_FAIL_IF_EXISTS = &H1
Public Const COPY_FILE_RESTARTABLE = &H2
Public Declare Function CopyFileEx Lib "kernel32.dll" Alias "CopyFileExA" (ByVal lpExistingFileName As String, ByVal lpNewFileName As String, ByVal lpProgressRoutine As Long, lpData As Any, ByRef pbCancel As Long, ByVal dwCopyFlags As Long) As Long
Public bCancel As Long
Public Function CopyProgressRoutine(ByVal TotalFileSize As Currency, ByVal TotalBytesTransferred As Currency, ByVal StreamSize As Currency, ByVal StreamBytesTransferred As Currency, ByVal dwStreamNumber As Long, ByVal dwCallbackReason As Long, ByVal hSourceFile As Long, ByVal hDestinationFile As Long, ByVal lpData As Long) As Long
'adjust the caption
Form1.Caption = CStr(Int((TotalBytesTransf
'allow user input
DoEvents
'continue filecopy
CopyProgressRoutine = PROGRESS_CONTINUE
End Function