[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

Need fastest method to read a binary file

Posted on 2002-04-11
8
Medium Priority
?
441 Views
Last Modified: 2012-06-21
Hello Experts !

I have to do a cyclic redundancy check 32-bit (CRC32) for given files to ensure (not strong algorithm but sufficient and faster than SHA,MD5,Blowfish) it's authenticity.
System:

PII,MMX
450 MHz
256MB RAM
Win2000

This are the benchmark values for the current algorithm:

FileName        1MB.TXT     10MB.TXT
Size(Bytes)     1.048.576   10.485.760
Duration (s)    3,19        32,02
MB/s            0,314       0,312

Code:

'------------------------------------------------------'FORM
'------------------------------------------------------

Private Sub cmdCompute_Click()
    Dim FSO As FileSystemObject
    Dim Start As Single
    Dim Duration As Double
    Dim lCrc32Value As Long
    Dim FileStr As String
   
   
    Me.MousePointer = vbHourglass
    Start = Timer
       
    Set FSO = New FileSystemObject
    lCrc32Value = InitCrc32()
    FileStr = FSO.GetFile(cmbPfade.Text).OpenAsTextStream.ReadAll
    lCrc32Value = AddCrc32(FileStr, lCrc32Value)
    txtCRC32.Text = Hex$(GetCrc32(lCrc32Value))
       
       
    Duration = CDbl(Timer) - CDbl(Start)
    If Duration > 0 Then
        txtMBProSec.Text = Format(CDbl(txtDateiGröße.Text) / (Duration * CDbl(1048576)), "0.000")
    Else
        txtMBProSec.Text = "Duration too short!"
    End If
    txtDuration.Text = Format(Duration, "0.00")
    Me.MousePointer = vbDefault
End Sub
'------------------------------------------------------'MODULE
'------------------------------------------------------
Private Crc32Table(255) As Long

Public Function InitCrc32(Optional ByVal Seed As Long = &HEDB88320, Optional ByVal Precondition As Long = &HFFFFFFFF) As Long
    Dim iBytes As Integer 'counter
    Dim iBits As Integer 'counter
    Dim lCrc32 As Long 'value
    Dim lTmpCrc32 As Long 'value

    For iBytes = 0 To 255
        lCrc32 = iBytes
        'Now iterate through each bit in counter byte
        For iBits = 0 To 7
            'Right shift unsigned long 1 bit
            lTmpCrc32 = lCrc32 And &HFFFFFFFE
            lTmpCrc32 = lTmpCrc32 \ &H2
            lTmpCrc32 = lTmpCrc32 And &H7FFFFFFF
            If (lCrc32 And &H1) <> 0 Then 'Now check if temporary is less than zero and then mix Crc32 checksum with Seed value
                lCrc32 = lTmpCrc32 Xor Seed
            Else
                lCrc32 = lTmpCrc32
            End If
        Next
        Crc32Table(iBytes) = lCrc32 'Put Crc32 checksum value in the holding array
    Next
    'After this is done, set function value to the precondition value
    InitCrc32 = Precondition
End Function

Public Function AddCrc32(Item As String, ByVal Crc32 As Long) As Long
    'ByVal Item As String   <= string that is to be checksum-computed
    Dim bCharValue As Byte
    Dim iCounter As Long 'V.K. Aufgrund großer Dateien geändert zu long
    Dim lIndex As Long
    Dim lAccValue As Long
    Dim lTableValue As Long
   
    For iCounter = 1 To Len(Item)
        bCharValue = Asc(Mid$(Item, iCounter, 1)) 'Get ASCII value for the current character
        'Right shift an Unsigned Long 8 bits
        lAccValue = Crc32 And &HFFFFFF00
        lAccValue = lAccValue \ &H100
        lAccValue = lAccValue And &HFFFFFF
        'Now select the right adding value from the holding table
        lIndex = Crc32 And &HFF
        lIndex = lIndex Xor bCharValue
        lTableValue = Crc32Table(lIndex)
        'Then mix new Crc32 value with previous accumulated Crc32 value
        Crc32 = lAccValue Xor lTableValue
    Next
    AddCrc32 = Crc32 'Set function value the the new Crc32 checksum
End Function

Public Function GetCrc32(ByVal Crc32 As Long) As Long
    GetCrc32 = Crc32 Xor &HFFFFFFFF 'Set function to the current Crc32 value
End Function

Does any expert know a way to increase noticeable the time performance by changing the code ?


Regards

V.K.
0
Comment
Question by:VK
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 2
8 Comments
 
LVL 6

Author Comment

by:VK
ID: 6933436
PS:

Add this to the declaration Section of the module.

Option Explicit
Option Compare Text

Private Crc32Table(255) As Long

The files hve an average size of 300k, but files up to 10MB are possible.

V.K.
0
 
LVL 18

Expert Comment

by:deighton
ID: 6933460
I don't know what the fastest way to get data from a file in VB is, I read binary files like this..

    Dim x As String
    Dim iFile As Long
   
    Dim times
   
    iFile = FreeFile
   
    times = Timer
   
    Open "c:\download\msaoe.exe" For Binary As iFile
    x = String(LOF(iFile), 0)
    Get iFile, , x
   
    Close iFile
   
   
    MsgBox Timer - times



Have you considered c?
0
 
LVL 6

Author Comment

by:VK
ID: 6933513
Hello deighton !

I've now tested your suggested code:

1. It's about 16% faster (noticeable ?).
2. I'm getting a different values for CRC32 !
   a. FileStr = FSO.GetFile(cmbPfade.Text).OpenAsTextStream.ReadAll
      CRC32 = "40E22B13"
   b. Get #DateiNr, , FileStr
      CRC32 = "B225DEA7"

V.K.

0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 6

Author Comment

by:VK
ID: 6933544
I've made some changes:

Replaced

Set FSO = New FileSystemObject
FileStr = FSO.GetFile(cmbPfade.Text).OpenAsTextStream.ReadAll


to

Dim DateiNr As Integer
DateiNr = FreeFile
Open cmbPfade.Text For Binary As DateiNr
FileStr = String(LOF(DateiNr), 0)
Get DateiNr, , FileStr
Close DateiNr

And

bCharValue = Asc(Mid$(Item, iCounter, 1)) 'Get ASCII value for the current character

to

bCharValue = AscB(Mid$(Item, iCounter, 1)) 'Get ASCII value for the current character

Performace:

0,430 MB/s

CRC32 ="44134EDB"

I will inrease points to 100 if somebody can explain me the different CRC32-values.

V.K.
0
 
LVL 6

Author Comment

by:VK
ID: 6933601
My next approach:

Private Sub cmdCompute_Click()
    Dim Start As Single
    Dim Duration As Double
    Dim lCrc32Value As Long
    Dim ByteArray() As Byte
    Dim FileNr As Integer
   
    Start = Timer
    lCrc32Value = InitCrc32()
                       
    FileNr = FreeFile
    Open cmbPfade.Text For Binary As FileNr
    ReDim ByteArray(LOF(FileNr) - 1)
    Get FileNr, , ByteArray
    Close FileNr
       
    lCrc32Value = AddCrc32(ByteArray, lCrc32Value)
    txtCRC32.Text = Hex$(GetCrc32(lCrc32Value))
       
    Duration = CDbl(Timer) - CDbl(Start)
    If Duration > 0 Then
        txtMBProSec.Text = Format(CDbl(txtDateiGröße.Text) / (Duration * CDbl(1048576)), "0.000")
    Else
        txtMBProSec.Text = "Duration too short!"
    End If
    txtDuration.Text = Format(Duration, "0.00")
End Sub

Public Function AddCrc32(Item() As Byte, ByVal Crc32 As Long) As Long
    Dim bCharValue As Byte
    Dim iCounter As Long
    Dim lIndex As Long
    Dim lAccValue As Long
    Dim lTableValue As Long
   
    For iCounter = 0 To UBound(Item)
        bCharValue = Item(iCounter)
        lAccValue = Crc32 And &HFFFFFF00
        lAccValue = lAccValue \ &H100
        lAccValue = lAccValue And &HFFFFFF
        lIndex = Crc32 And &HFF
        lIndex = lIndex Xor bCharValue
        lTableValue = Crc32Table(lIndex)
        Crc32 = lAccValue Xor lTableValue
    Next
    AddCrc32 = Crc32
End Function


Performace:

0,725 MB/s

CRC32 ="B225DEA7"


V.K.
0
 
LVL 6

Author Comment

by:VK
ID: 6933649
PS:

All performance Data is for the execution in the vb-ide.
Compiled to an exe:

10,751 MB/s !! -> thats OK

But what sould be the correct CRC32 ?
Is it only important to have different CRC's for different files and vica versa ?

Example:

When i create a Text file (63Byte) with the following content (of course without quotes):

'http://www.experts-exchange.com/jsp/qManageQuestion.jsp?qid=20287363'

--> CRC32 = "FC69DB59"

50 Pts for anyone who can explain me the correct CRC32.

V.K.
0
 
LVL 18

Accepted Solution

by:
deighton earned 200 total points
ID: 6933692
yes I know why it is, I was wrong, wrong, wrong


each place of the string soaks up two bytes, the following might work, reading it into a byte array. - but converting to a string might slow you down.

Of course you could work with the byte array, the crc code might be faster.

sorry to send you down a bit of a blind alley.

Private Sub Command1_Click()

   Dim bytes() As Byte
   Dim iFile As Long
   
   Dim times
   
   Dim x As String
   
   iFile = FreeFile
   
   times = Timer
   
   Open "c:\download\msaoe.exe" For Binary As iFile
   ReDim bytes(0 To LOF(iFile) - 1)
   
   
   Get iFile, , bytes
   
   Close iFile
   
   x = StrConv(bytes, vbUnicode)
   

End Sub



0
 
LVL 6

Author Comment

by:VK
ID: 6933811
Thank you for your assistance in realizing the difference between ASCII and Unicode.

Because it isn't relevant what CRC32 (but unique to a file) i'm getting i don't have to convert the string.

Regards

V.K.

 
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction In a recent article (http://www.experts-exchange.com/A_7811-A-Better-Concatenate-Function.html) for the Excel community, I showed an improved version of the Excel Concatenate() function.  While writing that article I realized that no o…
You can of course define an array to hold data that is of a particular type like an array of Strings to hold customer names or an array of Doubles to hold customer sales, but what do you do if you want to coordinate that data? This article describes…
Get people started with the process of using Access VBA to control Outlook using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Microsoft Outlook. Using automation, an Access applic…
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…
Suggested Courses

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question