Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

VB.NET - Retrieve File Info From Multiple Files Via Multiple Threads?

Posted on 2010-11-19
8
Medium Priority
?
530 Views
Last Modified: 2012-05-10
Hi,

I'm attempting to write a windows service that will use the filesystemwatcher to monitor for file creations, changes, renames and deletions. When it detects any of those, it will then pass information such as Date / Time, owner, full path, file name and an MD5 checksum to a SQL database. Whenever a file is created, the MD5 checksum of the new file is then used to check all previous records in the database to search for any duplicates, if a duplicate is found, an e-mail is then sent to the appropiate people.

All of the above works fine, however, if I do a large file copy (17,000 files), the service sees two records, a creation of a new file and then a change once the copy of that particular file is complete. I'd expect to see 34,000 records in the database, however, i'm only seeing somewhere in the region of 150 records.

I'm just wondering whether the files are being copied quicker than the service can run the SQL query, in which case, how would I go about making the service run the same code in multiple threads whilst ensuring that all threads don't jump on the same file?


Thanks in advance.
0
Comment
Question by:Graham2510
  • 4
  • 4
8 Comments
 

Author Comment

by:Graham2510
ID: 34173142
Below is the code, I thought this might help...

Imports System.IO
Imports System.Security.AccessControl
Imports System.Security.Principal
Imports System.Management
Imports System.Security.Cryptography
Imports System.Text
Imports System.Data.SqlClient
Imports System.Net.Mail


Public Class Form1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        Dim RenameWatcher As FileSystemWatcher = New FileSystemWatcher()
        RenameWatcher.Path = "D:\99. Testing\DupTest"
        RenameWatcher.NotifyFilter = (NotifyFilters.FileName)
        RenameWatcher.EnableRaisingEvents = True
        RenameWatcher.IncludeSubdirectories = True
        AddHandler RenameWatcher.Renamed, AddressOf FileRenamed

        Dim CreateWatcher As FileSystemWatcher = New FileSystemWatcher()
        CreateWatcher.Path = "D:\99. Testing\DupTest"
        CreateWatcher.NotifyFilter = (NotifyFilters.FileName)
        CreateWatcher.EnableRaisingEvents = True
        CreateWatcher.IncludeSubdirectories = True
        AddHandler CreateWatcher.Created, AddressOf FileCreated

        Dim ChangeWatcher As FileSystemWatcher = New FileSystemWatcher()
        ChangeWatcher.Path = "D:\99. Testing\DupTest"
        ChangeWatcher.NotifyFilter = (NotifyFilters.Size)
        ChangeWatcher.EnableRaisingEvents = True
        ChangeWatcher.IncludeSubdirectories = True
        AddHandler ChangeWatcher.Changed, AddressOf FileChanged

        Dim DeletedWatcher As FileSystemWatcher = New FileSystemWatcher()
        DeletedWatcher.Path = "D:\99. Testing\DupTest"
        DeletedWatcher.NotifyFilter = (NotifyFilters.LastAccess Or NotifyFilters.LastWrite Or NotifyFilters.FileName Or NotifyFilters.DirectoryName)
        DeletedWatcher.EnableRaisingEvents = True
        DeletedWatcher.IncludeSubdirectories = True
        AddHandler DeletedWatcher.Deleted, AddressOf FileDeleted

    End Sub



    ''' ''''''''''''''''''''''''''''''''''''''''''' RENAMED ''''''''''''''''''''''''''''''''''''''''''''
    Private Sub FileRenamed(ByVal Sender As Object, ByVal MonitoredFile As RenamedEventArgs)
        Dim RenamedFileFullPath As String = MonitoredFile.FullPath
        Dim FileName As String = MonitoredFile.Name
        Dim OldFullPath As String = MonitoredFile.OldFullPath
        Dim MD5Hash = RenamedgetFileMd5(RenamedFileFullPath)
        Dim Owner = GetFileOwner(RenamedFileFullPath)
        Dim Date_Time As String = Date.Now
        Dim SqlStr As String
        Dim ConnString As String

        ConnString = "Data Source=SERVER\DB; UID=DOMAIN\USER; Password=MYPASS; Integrated Security=True"
        SqlStr = "INSERT into [Master].[dbo].[FileInfo]([Date_Time], [Owner], [Full_Path], [Old_Path], [File_Name], [MD5_Checksum], [Action]) VALUES('" & Date_Time & "', '" & Owner & "', '" & RenamedFileFullPath & "', '" & OldFullPath & "', '" & FileName & "', '" & MD5Hash & "', 'RENAMED')"

        Dim SqlConn As New SqlConnection()
        Dim SqlCmd As New SqlCommand()

        SqlConn.ConnectionString = ConnString
        SqlConn.Open()
        SqlCmd.Connection = SqlConn
        SqlCmd.CommandText = SqlStr
        SqlCmd.ExecuteNonQuery()
        SqlConn.Close()
    End Sub


    ''' ''''''''''''''''''''''''''''''''''''''''''' CHANGED ''''''''''''''''''''''''''''''''''''''''''''
    Private Sub FileChanged(ByVal Sender As Object, ByVal MonitoredFile As FileSystemEventArgs)
        Dim ChangedFileFullPath As String = MonitoredFile.FullPath
        Dim FileName As String = MonitoredFile.Name
        Dim MD5Hash = RenamedgetFileMd5(ChangedFileFullPath)
        Dim Owner = GetFileOwner(ChangedFileFullPath)
        Dim Date_Time As String = Date.Now


        Dim SqlStr As String
        Dim ConnString As String

        ConnString = "Data Source=SERVER\DB; UID=DOMAIN\USER; Password=MYPASS; Integrated Security=True"
        SqlStr = "INSERT into [Master].[dbo].[FileInfo]([Date_Time], [Owner], [Full_Path], [Old_Path], [File_Name], [MD5_Checksum], [Action]) VALUES('" & Date_Time & "', '" & Owner & "', '" & ChangedFileFullPath & "', 'N/A', '" & FileName & "', '" & MD5Hash & "', 'CHANGED')"

        Dim SqlConn As New SqlConnection()
        Dim SqlCmd As New SqlCommand()

        SqlConn.ConnectionString = ConnString
        SqlConn.Open()
        SqlCmd.Connection = SqlConn
        SqlCmd.CommandText = SqlStr
        SqlCmd.ExecuteNonQuery()
        SqlConn.Close()
    End Sub


    ''' ''''''''''''''''''''''''''''''''''''''''''' DELETED ''''''''''''''''''''''''''''''''''''''''''''
    Private Sub FileDeleted(ByVal Sender As Object, ByVal MonitoredFile As FileSystemEventArgs)
        Dim DeletedFileFullPath As String = MonitoredFile.FullPath
        Dim FileName As String = MonitoredFile.Name
        Dim Date_Time As String = Date.Now
        Dim SqlStr As String
        Dim ConnString As String

        ConnString = "Data Source=SERVER\DB; UID=DOMAIN\USER; Password=MYPASS; Integrated Security=True"
        SqlStr = "INSERT into [Master].[dbo].[FileInfo]([Date_Time], [Owner], [Full_Path], [Old_Path], [File_Name], [MD5_Checksum], [Action]) VALUES('" & Date_Time & "', 'N/A', '" & DeletedFileFullPath & "', 'N/A', '" & FileName & "', 'N/A', 'DELETED')"

        Dim SqlConn As New SqlConnection()
        Dim SqlCmd As New SqlCommand()

        SqlConn.ConnectionString = ConnString
        SqlConn.Open()
        SqlCmd.Connection = SqlConn
        SqlCmd.CommandText = SqlStr
        SqlCmd.ExecuteNonQuery()
        SqlConn.Close()
    End Sub


    ''' ''''''''''''''''''''''''''''''''''''''''''' CREATED ''''''''''''''''''''''''''''''''''''''''''''
    Private Sub FileCreated(ByVal Sender As Object, ByVal MonitoredFile As FileSystemEventArgs)
        Dim CreatedFileFullPath As String = MonitoredFile.FullPath
        Dim FileName As String = MonitoredFile.Name
        Dim MD5Hash = RenamedgetFileMd5(CreatedFileFullPath)
        Dim Owner = GetFileOwner(CreatedFileFullPath)
        Dim Date_Time As String = Date.Now

        Dim SqlStr2 As String
        Dim ConnString2 As String

        ConnString2 = "Data Source=SERVER\DB; UID=DOMAIN\USER; Password=MYPASS; Integrated Security=True"
        SqlStr2 = "SELECT TOP 1 * FROM Master.dbo.FileInfo WHERE (MD5_Checksum = '" & MD5Hash & "') ORDER BY Date_Time DESC"

        Dim SqlConn2 As New SqlConnection()
        Dim SQLCmd2 As New SqlCommand()
        Dim SQLdr As SqlDataReader

        SqlConn2.ConnectionString = ConnString2
        SqlConn2.Open()
        SQLCmd2.Connection = SqlConn2
        SQLCmd2.CommandText = SqlStr2
        SQLdr = SQLCmd2.ExecuteReader

        Dim CheckHash As String
        Dim CheckFileName As String
        Dim CheckFullPath As String
        Dim CheckOwner As String



        While SQLdr.Read()
            CheckHash = (SQLdr("MD5_Checksum"))
            CheckFileName = (SQLdr("File_Name"))
            CheckFullPath = (SQLdr("Full_Path"))
            CheckOwner = (SQLdr("Owner"))

            If Not CheckHash = "" Then
                Dim smtpServer As New SmtpClient()
                Dim mail As New MailMessage()
                smtpServer.Credentials = New Net.NetworkCredential("MYUSER", "MYPASSWORD")
                smtpServer.Port = 25
                smtpServer.Host = "MYMAILSERVER"
                smtpServer.EnableSsl = False
                mail = New MailMessage()
                mail.From = New MailAddress("FromAddress@Domain.com")
                mail.To.Add("ToAddress@Domain.com")
                mail.Subject = "Duplicate File Created"
                mail.Body = "A user has just created a duplicate file, the details are as follows..." & vbCrLf & vbCrLf & "New File Path: " & vbTab & CreatedFileFullPath & " " & vbCrLf & "Created By: " & vbTab & Owner & vbCrLf & vbCrLf & vbCrLf & "ORIGINAL FILE DETAILS" & vbCrLf & vbCrLf & "Full Path: " & vbTab & CheckFullPath & vbCrLf & "Owner: " & vbTab & CheckOwner
                smtpServer.Send(mail)
            End If
        End While

        SQLdr.Close()
        SqlConn2.Close()

        Dim SqlStr As String
        Dim ConnString As String
        ConnString = "Data Source=SERVER\DB; UID=DOMAIN\USER; Password=MYPASS; Integrated Security=True"
        SqlStr = "INSERT into [Master].[dbo].[FileInfo]([Date_Time], [Owner], [Full_Path], [Old_Path], [File_Name], [MD5_Checksum], [Action]) VALUES('" & Date_Time & "', '" & Owner & "', '" & CreatedFileFullPath & "', 'N/A', '" & FileName & "', '" & MD5Hash & "', 'CREATED')"
        Dim SqlConn As New SqlConnection()
        Dim SqlCmd As New SqlCommand()
        SqlConn.ConnectionString = ConnString
        SqlConn.Open()
        SqlCmd.Connection = SqlConn
        SqlCmd.CommandText = SqlStr
        SqlCmd.ExecuteNonQuery()
        SqlConn.Close()
    End Sub







    Public Function GetFileOwner(ByRef strFile As String) As String

        Try
            Dim wmiFileSecurity As New ManagementObject("Win32_LogicalFileSecuritySetting.path='" & strFile & "'")
            Dim propSecurity As ManagementBaseObject
            Dim descriptorSecurity As ManagementBaseObject
            Dim objectOwner As ManagementBaseObject

            propSecurity = wmiFileSecurity.InvokeMethod("GetSecurityDescriptor", Nothing, Nothing)
            descriptorSecurity = CType(propSecurity.Properties("Descriptor").Value(), ManagementBaseObject)
            objectOwner = CType(descriptorSecurity.Properties("Owner").Value, ManagementBaseObject)
            Return (objectOwner.Properties("Domain").Value.ToString & "\" & _
               objectOwner.Properties("Name").Value.ToString)
        Catch ex As Exception
            MsgBox(ex.ToString)
        End Try

    End Function




    Private Function RenamedgetFileMd5(ByVal filePath As String) As String
        System.Threading.Thread.Sleep(1000)
        Dim File() As Byte = System.IO.File.ReadAllBytes(filePath)
        Dim Md5 As New MD5CryptoServiceProvider()
        Dim byteHash() As Byte = Md5.ComputeHash(File)
        Return Convert.ToBase64String(byteHash)
    End Function


End Class

Open in new window

0
 
LVL 83

Accepted Solution

by:
CodeCruiser earned 2000 total points
ID: 34173655
The FileSystemWatcher is not a fool proof way of monitoring files as it has many known issues.

Anyway, you can move all the code to a function and then in the file created event, just create a new thread and pass it the path of the file and it would then handle the rest. But this could get ugly as you dont want to create 10000 threads in your app so I would suggest you use a Queue and a single thread which keeps running and processing the files from the queue.

Examples

http://vb.net-informations.com/collections/vb.net_Queue.htm

http://visualbasic.about.com/od/usingvbnet/a/NewCalc01.htm
0
 

Author Comment

by:Graham2510
ID: 34174117
Hi CodeCruiser,

Thanks for the suggestion. I've read the two links that you supplied but unfortunately, as my programming skills are at a novice level, i'm not 100% sure on how to implement a queue into my app. Would you be able to offer some assistance?


Thanks.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 83

Expert Comment

by:CodeCruiser
ID: 34174467
You can declare a queue at the class level. Then in the FileCreated event, just add the file name to the queue, for example

    Private Sub FileCreated(ByVal Sender As Object, ByVal MonitoredFile As FileSystemEventArgs)
     QueueName.Enqueue(MonitoredFile.Name)
    End Sub

Then in the thread method, you can use

FileToProcess = QueueName.DeQueue()
0
 

Author Comment

by:Graham2510
ID: 34174645
Hi,

Sorry if this question seems dumb, it's Friday afternoon and my brain doesn't work too well at the best of times!

What do you mean by thread method?
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 34174688
Leave it till Monday and read about MultiThreading on monday morning.
0
 

Author Comment

by:Graham2510
ID: 34267634
Hi9 CodeCruiser,

I've finally managed to get back onto this project and with a clear head I worked out the queueing. Everything runs smoothly now.

Thanks.
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 34267645
Glad to help :-)
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A while ago, I was working on a Windows Forms application and I needed a special label control with reflection (glass) effect to show some titles in a stylish way. I've always enjoyed working with graphics, but it's never too clever to re-invent …
Creating an analog clock UserControl seems fairly straight forward.  It is, after all, essentially just a circle with several lines in it!  Two common approaches for rendering an analog clock typically involve either manually calculating points with…
Want to learn how to record your desktop screen without having to use an outside camera. Click on this video and learn how to use the cool google extension called "Screencastify"! Step 1: Open a new google tab Step 2: Go to the left hand upper corn…
Is your data getting by on basic protection measures? In today’s climate of debilitating malware and ransomware—like WannaCry—that may not be enough. You need to establish more than basics, like a recovery plan that protects both data and endpoints.…
Suggested Courses

773 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question