Advertisement

05.05.2008 at 06:36AM PDT, ID: 23376338
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

Handling large number of file in Visual Basic 2008

Tags: Microsoft, Visual Basic, 2008, Visual Basic, 2008
I have a small problem where I am unsure how to angle the solution.

I am writing a large website that handles a lot of image uploaded by FTP from webcameras. So my simple task is to monitor a large number of folders (in the range of 500-750) continously and always keep the folders empty except from the newest image.

So I need to iterate the folders continously and delete all files in the folder at any gives time except for the newest.

I am not asking for a complete code here but just an attach plan. Shoud I make a watcher-thread per folder og just have one thread loop through all folder 24/7. Should I make a 10sec pause between each iteration? Is the one way of iteration / deletion that's better than another

And I want to do with a server aplication written in Visual Basic 2008...

Any ideas?
Start your free trial to view this solution
Question Stats
Zone: Programming
Question Asked By: logicmedia
Solution Provided By: SStory
Participating Experts: 1
Solution Grade: B
Views: 0
Translate:
Loading Advertisement...
05.05.2008 at 08:15AM PDT, ID: 21500607

Rank: Master

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
05.05.2008 at 12:24PM PDT, ID: 21502324

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
05.05.2008 at 02:29PM PDT, ID: 21503183

Rank: Master

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
Loading Advertisement...
Microsoft
  • Internet Protocols
  • Applications
  • Development
  • OS
  • Hardware
  • Windows Security
Apple
  • Operating Systems
  • Hardware
  • Programming
  • Networking
  • Software
Internet
  • Search Engines
  • File Sharing
  • WebTrends / Stats
  • Spy / Ad Blockers
  • Web Browsers
  • New Net Users
  • Web Development
  • Chat / IM
  • Anti Spam
  • Web Servers
  • Anti-Virus
  • Email Clients
Gamers
  • Tips
  • Online / MMORPG
  • Puzzle
  • Emulators
  • Action / Adventure
  • Role Playing
  • Consoles
  • Game Programming
  • Strategy
  • Sports
  • Misc
  • Computer Games
Digital Living
  • Hardware
  • Automotive
  • New Net Users
  • New Users
  • Software
  • Digital Music
  • Gaming World
  • Home Security
  • Apple
  • Networking Hardware
Virus & Spyware
  • Vulnerabilities
  • IDS
  • Encryption
  • Anti-Virus
  • Operating Systems Security
  • Software Firewalls
  • WebApplications
  • Cell Phones
  • Operating Systems
  • Internet
  • Hardware Firewalls
Hardware
  • Displays / Monitors
  • Handhelds / PDAs
  • Components
  • Peripherals
  • Laptops/Notebooks
  • Servers
  • Misc
  • Apple
  • Embedded Hardware
  • Networking Hardware
  • Storage
  • Desktops
  • New Users
Software
  • System Utilities
  • Industry Specific
  • Network Management
  • Photos / Graphics
  • Page Layout
  • VMware
  • Misc
  • Web Development
  • OS
  • CYGWIN
  • Voice Recognition
  • Virtualization
  • Message Queue
  • Quality Assurance
  • Security
  • Firewalls
  • MultiMedia Applications
  • Development
  • Database
  • Office / Productivity
  • Business Management
  • OS/2 Apps
  • Server Software
  • Internet / Email
ITPro
  • OS
  • Storage
  • Encryption
  • Operating Systems Security
  • Apple Hardware
  • Laptops & Notebooks
  • Servers
  • Networking Hardware
  • Peripherals
  • Devices
  • Displays / Monitors
  • WebTrends / Stats
  • Search Engines
  • Firewalls
  • Web Computing
  • WebApplications
  • IDS
  • Vulnerabilities
  • Email Clients
  • File Sharing
  • Spy / Ad Blockers
  • Web Browsers
  • Web Servers
  • Networking
  • Anti-Virus
  • Consulting
  • Chat / IM
  • Anti Spam
Developer
  • Web Servers
  • Web Browsers
  • Game Programming
  • Dev Tools
  • Industry Specific
  • Office / Productivity
  • Database
  • CYGWIN
  • Web Development
  • Search Engines
  • File Sharing
  • WebTrends / Stats
  • Programming
  • Content Management
  • Application Servers
  • Protocols
Storage
  • Removable Backup Media
  • Storage Technology
  • Servers
  • Grid
  • Remote Access
  • Backup / Restore
  • Misc
  • Hard Drives
OS
  • Miscellaneous
  • Security
  • Development
  • Linux
  • VMware
  • MainFrame OS
  • Unix
  • Apple
  • OS / 2
  • AS / 400
  • BeOS
  • Microsoft
  • VMS / OpenVMS
Database
  • Oracle
  • Miscellaneous
  • MySQL
  • Software
  • Sybase
  • Contact Management
  • PostgreSQL
  • Data Manipulation
  • Clarion
  • InterSystems Cache
  • Siebel
  • MUMPS
  • OLAP
  • SQLBase
  • SAS
  • GIS & GPS
  • 4GL
  • Berkeley DB
  • DB2
  • Informix
  • Interbase / Firebird
  • FoxPro
  • Reporting
  • LDAP
  • Filemaker Pro
  • MS SQL Server
  • dBase
  • MS Access
Security
  • Misc
  • Web Browsers
  • Software Firewalls
  • Operating Systems Security
  • File Sharing
  • Spy / Ad Blockers
  • Vulnerabilities
  • WebApplications
  • IDS
  • Anti-Virus
  • Encryption
  • Anti Spam
  • Email Clients
  • VPN
  • Chat / IM
Programming
  • Editors IDEs
  • Installation
  • Handhelds / PDAs
  • Multimedia Programming
  • System / Kernel
  • Automation
  • Algorithms
  • Game
  • Signal Processing
  • Project Management
  • Open Source
  • Database
  • Misc
  • Languages
  • Processor Platforms
  • Theory
Web Development
  • Scripting
  • Blogs
  • Web Servers
  • Software
  • Search Engines
  • Web Graphics
  • Web Services
  • Images
  • Internet Marketing
  • Images and Photos
  • Components
  • Document Imaging
  • Web Languages/Standards
  • Illustration
  • WebApplications
  • Fonts
  • WebTrends / Stats
  • Authoring
  • Digital Camera Software
  • Miscellaneous
Networking
  • Protocols
  • Apple Networking
  • Network Management
  • Message Queue
  • Application Servers
  • Content Management
  • File Servers
  • Email Servers
  • Misc
  • Java Editors & IDEs
  • Wireless
  • Networking Hardware
  • Backup / Restore
  • System Utilities
  • ISPs & Hosting
  • Web Servers
  • Storage Technology
  • Removable Backup Media
  • Servers
  • Web Computing
  • Broadband
  • Grid
  • OS / 2
  • Novell Netware
  • Unix Networking
  • Windows Networking
  • Security
  • Telecommunications
  • Operating Systems
  • Linux Networking
Other
  • Lounge
  • Business Travel
  • Community Support
  • New Net Users
  • Philosophy / Religion
  • Math / Science
  • Miscellaneous
  • URLs
  • Expert Lounge
  • Politics
  • Puzzles / Riddles
  • Automotive
Community Support
  • Suggestions
  • New to EE
  • New Topics
  • CleanUp
  • Announcements
  • General
  • Feedback
  • Input
  • EE Bugs
 
05.05.2008 at 08:15AM PDT, ID: 21500607

Rank: Master

750 threads would NOT make the thing better. It would probably bog it down quite a bit.

Dotnet has the FileSystemWatcher.

I don't know how well it would work with 750 of them. I would think it would bog down things.

In my mind a balance would be better.  Consider a thread pool of 10-15 worker max. One thread should control the rest by assigning jobs to them.  That way 10-15 threads are looking at different places.
Using 15, that means that (750/15)=50 iterations would have all 750 done...more or less.  You will need to play with the number as too many threads are counter productive. Find the optimum number and let it run.

Of course if you are real ambitious and have multiple computers sitting around doing nothing, you could write some remoting objects that sit on various computers and have your main computer tell each of them what to do.  10 computers, each running 10 threads, means 100threads processing.  In this case, you might just have each of the 10 computers looking at 75 folders using 10 threads. This would make it super quick.  It just depends on what resources you have and how long you have to process it.

If it just needs to get done in a decent amount of time, you may just use one computer with 10 threads or so.  Maybe each of the thread, either deletes the file itself, or adds it to a queue where another thread watches the queue and deletes everything in it.  I'm not sure which way would be quicker...I do know that the thread finding it would be free to keep looking as soon as it made a string entry in a queue...which would take less time than file I/O.
Accepted Solution
 
05.05.2008 at 12:24PM PDT, ID: 21502324
Each came will place a picture in its folder every 2 second so things need to be running smoothly in order to keep the file count low. I like the idea of the queue and then have one or more ?? threads doing the actual file delete.

Would it make a difference if it's only one thread deleting - or should there be multiple. What about the watching - should that be distributed? Should I put a sleep in the thread every now or then in order to give the system a little rest or should I rely on Windows not to let each thread load the system too much?

- thomas
 
05.05.2008 at 02:29PM PDT, ID: 21503183

Rank: Master

The backgroundworker component could be used if you do threads and not filesystemwatchers

See this:
http://www.experts-exchange.com/Programming/Languages/.NET/Visual_Basic.NET/Q_23371828.html

Well, one thread deleting means it would check the queue  at a set interval to see if the count was >0
If so, it would start deleting until the <Queuename>.count=0
This would obviously take as long as it takes....  again it depends on how quickly you want it to happen. If it must happen extremely quickly, then the distributed method might be the best.  Each of 10 computers has 75 folders that it always checks and handles the deleting.  This could be done by remoting, or just by one program that has a .Ini or .XML file that tell which folders to watch for.  Then just have the program run on each of the 10. This would work almost as good and maybe less complicated.  Only you'd have to somehow keep it running on 10 computers... maybe a Windows Service on all 10 computer would solve that problem.

I would just do it on one computer with 10 or so threads and see if it is fast enough and if not start dividing the load.  
 
 
20080236-EE-VQP-29 / EE_QW_2_20070628