We help IT Professionals succeed at work.

How do I handle a file on network share using a web service

High Priority
165 Views
Last Modified: 2019-08-04
How do I handle this scenario:

A web service will receive data and write to a file on a network share.
Another application is running as a service that checks for files, reads the data, processes, saves the data to a database then deletes the file.
This application must run in multiple data centers and on multiple servers in each data center.  How do I ensure that each file is processed
only once.
Comment
Watch Question

Test your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016
Commented:
Assuming that there is only one copy of the file, on a single central share, then I would use the database to control this.  Basically you set up a database table that will be the "queue" of files waiting to be processed.  As a file is received by the web service, after the file is stored in the share location, a row is added to the queue table indicating the file is ready to be processed.  Then you have one or more of the application services running that can service that queue.  Part of the trick is to make sure that one and only one of those picks up the file for processing.  You can find different approaches to this with a little searching on the interwebs, but one approach is for the server applications to look for pending entries, place a lock on that row when they intend to process it, update the row to "being processed", process the file, and then update the row to "processed".  The delicate part is making sure that only one server gets lock access to the queue entry, and the database locking is key here.  As long as the code is written properly and checks that the data hasn't changed since it read it last, and it makes sure it can get a lock before processing etc, then you should be able to avoid collisions.  Hope this helps some, at least at a conceptual level.


»bp
TechIsaacIT Contracting

Author

Commented:
Thanks.  Totally appreciated.
TechIsaacIT Contracting

Author

Commented:
I'll give it a try.