Z_Beeblebrox
asked on
Controlling access to a file
Hi,
I have a relatively complex situation here. What I want to do is to have a file on the hard drive. This file can be read and written to by an arbitrary number of programs on the computer. What I would like to do is maintain a system of read and write locks on the file. The way I envision it is that any program that requests read access to the file will be given access as long as no one currently has a write lock or has a pending write lock. When someone requests a write lock, no more read locks will be granted and once all existing read locks have been released the write lock will be granted. If possible I would like to suspend the execution of requestors until their lock request is granted. How can I do this?
Note that I have taken a quick look at the mutex which works well for single entry as well as the waitforsingleobject which suspend execution until the state changes. Both of these may come in handy.
Also note that I do not want to have a server managing this. It all must be done by the requesting clients, preferably with the OS doing as much of the work as possible.
I realize this is a hard question and I am willing to up the points if necessary.
Zaphod.
I have a relatively complex situation here. What I want to do is to have a file on the hard drive. This file can be read and written to by an arbitrary number of programs on the computer. What I would like to do is maintain a system of read and write locks on the file. The way I envision it is that any program that requests read access to the file will be given access as long as no one currently has a write lock or has a pending write lock. When someone requests a write lock, no more read locks will be granted and once all existing read locks have been released the write lock will be granted. If possible I would like to suspend the execution of requestors until their lock request is granted. How can I do this?
Note that I have taken a quick look at the mutex which works well for single entry as well as the waitforsingleobject which suspend execution until the state changes. Both of these may come in handy.
Also note that I do not want to have a server managing this. It all must be done by the requesting clients, preferably with the OS doing as much of the work as possible.
I realize this is a hard question and I am willing to up the points if necessary.
Zaphod.
ASKER
Hi ramy,
That would work fairly well except for a few problems:
- If two programs try to edit the file at the same time, one of them will overwrite the other's changes
- If the new file is copied over the original while someone is reading it, what will happen?
- Programs may read an out of date version of the file (this might not be a bad thing)
Zaphod.
That would work fairly well except for a few problems:
- If two programs try to edit the file at the same time, one of them will overwrite the other's changes
- If the new file is copied over the original while someone is reading it, what will happen?
- Programs may read an out of date version of the file (this might not be a bad thing)
Zaphod.
Write another file in the same directory, and maintain entries that describe what access is being granted, etc. at any given time.
Each client could check the entries, and act accordingly.
Each client could check the entries, and act accordingly.
ASKER
Hi,
Is there any way to queue up requests in windows, so each program that wants to access that file makes a request and then suspends until it is granted?
Zaphod.
Is there any way to queue up requests in windows, so each program that wants to access that file makes a request and then suspends until it is granted?
Zaphod.
This isn't very elegant but it should work.
First create a Read/Write Lock file for the apps to register their requests ( A DB table would be much easier to allow for easy checks for who's turn)
When an app makes a request enable a timer set to what ever time you want and a Public variable that records it's place in line.
Chks the file/db for a write request and if there is deny a read.
Enter the apps request in the list and assign it the next number
On the Timer event check if the Number assigned is next in the list and then allow that app to have their request
When there turn comes up allow the app to do the read or the edit and when the app is finished delete the apps entry in the Read/Write list
First create a Read/Write Lock file for the apps to register their requests ( A DB table would be much easier to allow for easy checks for who's turn)
When an app makes a request enable a timer set to what ever time you want and a Public variable that records it's place in line.
Chks the file/db for a write request and if there is deny a read.
Enter the apps request in the list and assign it the next number
On the Timer event check if the Number assigned is next in the list and then allow that app to have their request
When there turn comes up allow the app to do the read or the edit and when the app is finished delete the apps entry in the Read/Write list
If you want to proxide such service to any program (not only to program written by you) you need to write system driver for HDD. But I think it is not good idea to do such service.
Read please MSDN article about file opening (system function CreateFile) http://msdn.microsoft.com/library/psdk/winbase/filesio_7wmd.htm
Program logic may be different. If developer of the program created program correctly, your service is needless. In some cases you will improve work of programs but in other you can corrupt its work. I think that good idea is to provide such service only to those programs which know about it.
E.g. you can create system service program and other applications can tell your service that they want to use your arbitrage operations for certain file(s). I think it is more correct than your first variant.
Read please MSDN article about file opening (system function CreateFile) http://msdn.microsoft.com/library/psdk/winbase/filesio_7wmd.htm
Program logic may be different. If developer of the program created program correctly, your service is needless. In some cases you will improve work of programs but in other you can corrupt its work. I think that good idea is to provide such service only to those programs which know about it.
E.g. you can create system service program and other applications can tell your service that they want to use your arbitrage operations for certain file(s). I think it is more correct than your first variant.
Hi!
Basicly it is this same in here now, the Access driver has solved this like this:
http://support.microsoft.com/support/kb/articles/Q186/3/04.ASP
Now this is for database, not for regular files, and to make this work on every file need's similar prog like a virus protector is. These subclass system messages very wide, and VB is not very best choice to make such wide message hook.
Also if the "prosessor" is only in client macine, it can't detect these messages in the servers OS, so this is impossible.
Working Message hook dll's for VB are in Spy Work's pack.
So if this file is Access database this LDB reader will help other vice not much.
Matti
Basicly it is this same in here now, the Access driver has solved this like this:
http://support.microsoft.com/support/kb/articles/Q186/3/04.ASP
Now this is for database, not for regular files, and to make this work on every file need's similar prog like a virus protector is. These subclass system messages very wide, and VB is not very best choice to make such wide message hook.
Also if the "prosessor" is only in client macine, it can't detect these messages in the servers OS, so this is impossible.
Working Message hook dll's for VB are in Spy Work's pack.
So if this file is Access database this LDB reader will help other vice not much.
Matti
Hi zaphod,
In my point of view some of these few problems are not problems at all and some you would have faced them too if all clients were editing the same file at the same time.
For example the second problem is not a problem at all because no body will be reading the original file at any time, every client needs to edit will have his own temp copy that will edit it so the original file won't be open at any time.
The third problem i don't know why the programs will read out of date file, it is always updated once the client complete editing, think what if all clients where editing the same file? the file won't be updated too while a client is editing it will be updated after the client finishes completely updating and the same here.
The first problem i guess could be solved too but let us know first the type of file you are talking about.
Ramy...
In my point of view some of these few problems are not problems at all and some you would have faced them too if all clients were editing the same file at the same time.
For example the second problem is not a problem at all because no body will be reading the original file at any time, every client needs to edit will have his own temp copy that will edit it so the original file won't be open at any time.
The third problem i don't know why the programs will read out of date file, it is always updated once the client complete editing, think what if all clients where editing the same file? the file won't be updated too while a client is editing it will be updated after the client finishes completely updating and the same here.
The first problem i guess could be solved too but let us know first the type of file you are talking about.
Ramy...
ASKER
Hi,
Basically what I am doing is making my own DB engine. As a result, I need to allow concurrent read access to the file, but only sequential write access during which no one can read. Right now I have come up with a messy but functional solution to the problem but I wonder if anyone can see any problems with it or any improvements.
I have created a second file which will be used to control access to the file. This file will store 5 variables:
a boolean to say if the file is locked (ie being edited or processes waiting to edit it)
a long containing the thread ID of the thread editing the file if any
three string contains comma delimited lists of the thread IDs of the threads reading the file, waiting to read the file and waiting to write the file.
I will also have a mutex which will control access to the control file. Any time a thread wants to read the file, they first must get the mutex for the control file. Then, they check to see if the file is locked. If it isn't, they add their thread ID to the list of threads reading the file then release the mutex and being reading the file. If it is, then they add their thread ID to the list of pending threads and suspend themself.
If a thread wants to edit the file, then they first get the mutex for the control file, then check if there are any threads reading the file. If there are none, then it sets its thread ID to be the one editing the file and sets the boolean to true and begins editing the file. If there are then it sets the boolean to true and appends itself to the list of threads pending to write then suspends itself.
Any time a process completes its operation (either read or write), it is responsible for resuming the next thread(s) by examining the various variables.
The only problem with this that I see is that it will fail catastrophically if one of the threads fail. The only way I see around this would be instead of suspending threads, have them go to sleep for a certain period of time then resume and verify that the threads with access to the file are still operating, but I am not sure how to do this.
So, any comments?
Zaphod.
Basically what I am doing is making my own DB engine. As a result, I need to allow concurrent read access to the file, but only sequential write access during which no one can read. Right now I have come up with a messy but functional solution to the problem but I wonder if anyone can see any problems with it or any improvements.
I have created a second file which will be used to control access to the file. This file will store 5 variables:
a boolean to say if the file is locked (ie being edited or processes waiting to edit it)
a long containing the thread ID of the thread editing the file if any
three string contains comma delimited lists of the thread IDs of the threads reading the file, waiting to read the file and waiting to write the file.
I will also have a mutex which will control access to the control file. Any time a thread wants to read the file, they first must get the mutex for the control file. Then, they check to see if the file is locked. If it isn't, they add their thread ID to the list of threads reading the file then release the mutex and being reading the file. If it is, then they add their thread ID to the list of pending threads and suspend themself.
If a thread wants to edit the file, then they first get the mutex for the control file, then check if there are any threads reading the file. If there are none, then it sets its thread ID to be the one editing the file and sets the boolean to true and begins editing the file. If there are then it sets the boolean to true and appends itself to the list of threads pending to write then suspends itself.
Any time a process completes its operation (either read or write), it is responsible for resuming the next thread(s) by examining the various variables.
The only problem with this that I see is that it will fail catastrophically if one of the threads fail. The only way I see around this would be instead of suspending threads, have them go to sleep for a certain period of time then resume and verify that the threads with access to the file are still operating, but I am not sure how to do this.
So, any comments?
Zaphod.
I think you'd better create some logical layers for your application. One layer only read/write to file at certain position, other calculates position in a file using record index, combine records in a buffer to produce other operations, third - change data in a buffer according SQL clause, forth - parse clause text etc.
But, as far as I understand, you want to create simple database application. Main idea is that you work with one object to read/write file. So you don't need to open/close file in each thread. Thread allocates buffer for object to work and calls its method to read/write data from file position N. All arbitrage you will do inside object. Using object you can create cache of file in a memory or use different methods to access file. If you test enouph your object code and remove bugs - you will know that inside your object methods programs won't fall. You can lock only those parts of file which you are writing. You can create some oprimizations using this object etc.
If you are interested in this variant we can discuss some points more detaily.
Is it your school project?
Good luck!
But, as far as I understand, you want to create simple database application. Main idea is that you work with one object to read/write file. So you don't need to open/close file in each thread. Thread allocates buffer for object to work and calls its method to read/write data from file position N. All arbitrage you will do inside object. Using object you can create cache of file in a memory or use different methods to access file. If you test enouph your object code and remove bugs - you will know that inside your object methods programs won't fall. You can lock only those parts of file which you are writing. You can create some oprimizations using this object etc.
If you are interested in this variant we can discuss some points more detaily.
Is it your school project?
Good luck!
ASKER
This isn't a school project, just something I want to do for "fun" and experience :) I am also working on a few other apps which require very basic database backends and I don't like using Access.
The problem I am facing is managing the access that different programs have to the database, to ensure that they don't conflict. The method I proposed above should work, but I discovered that I cannot seem to suspend the current process. Looking closer at the MSDN docs seems to indicate that the suspendthread api call only works on windows CE, which is odd. Does anyone know how to suspend the current process and then resume it on 95/98/ME/NT/2000?
Zaphod.
The problem I am facing is managing the access that different programs have to the database, to ensure that they don't conflict. The method I proposed above should work, but I discovered that I cannot seem to suspend the current process. Looking closer at the MSDN docs seems to indicate that the suspendthread api call only works on windows CE, which is odd. Does anyone know how to suspend the current process and then resume it on 95/98/ME/NT/2000?
Zaphod.
Process and thread are different things (as it is written in MSDN). You cannot stop other process but can stop/continue one of your process threads. You only can send messages to other applications. But I don't know system messages like 'Suspend' or 'continue'. I think that there is no such messages or they are undocumented.
As a matter of fact, database should solve all conflicts itself, e.g. Access (Jet) does it. As I wrote before, it is not good idea to manage other applications.
As a matter of fact, database should solve all conflicts itself, e.g. Access (Jet) does it. As I wrote before, it is not good idea to manage other applications.
ASKER
Hi,
I do not want to suspend other processes, my clients (which I am writing) should suspend themselves. They can then be resumed by another process. Is this possible?
Alternatively, is there a way of determining the current usage of a semaphore? All I want to do is to suspend a process until the semaphore is no longer in use.
Zaphod.
I do not want to suspend other processes, my clients (which I am writing) should suspend themselves. They can then be resumed by another process. Is this possible?
Alternatively, is there a way of determining the current usage of a semaphore? All I want to do is to suspend a process until the semaphore is no longer in use.
Zaphod.
ok. Let's talk about your task.
Do you customer want that you create manager not for all tasks but for some specific tasks which will be written by you or somebody other but they will be a part of some system? I mean situation when you create technology which you (or somebody other) will use for some customer(s).
1. If yes - it is more simple than you told before. I have wrote you about service which can be used to arbitrage programs to access to a certain system resource.
2. If no - you can suspend your own processes but (AFAIK) you cannot resume other process. Manager can only set/clear flags. Processes can see whether flags are set and continue to work or no.
For first variant there are a lot of methods to implement it. Please answer what of variants do you have.
Do you customer want that you create manager not for all tasks but for some specific tasks which will be written by you or somebody other but they will be a part of some system? I mean situation when you create technology which you (or somebody other) will use for some customer(s).
1. If yes - it is more simple than you told before. I have wrote you about service which can be used to arbitrage programs to access to a certain system resource.
2. If no - you can suspend your own processes but (AFAIK) you cannot resume other process. Manager can only set/clear flags. Processes can see whether flags are set and continue to work or no.
For first variant there are a lot of methods to implement it. Please answer what of variants do you have.
ASKER
Hi,
I am creating this system for my own use. What I want to created is a distributed system for controlling access to the file, so no central server. I am willing to use APIs but I don't want to have to run a service. I am creating a DLL which programs can reference which will manage all of the DB functions. Therefore I can trust the clients implicitly since I have a layer written by me between the user code and the file.
The programs that use the DLL may or may not be written by me.
The whole goal behind this exercise is to create a very easy to use DB, with no installation required, no services to run, etc. All that is needed is to register a DLL. I am somewhat disappointed that it is proving to be so difficult to do.
BTW, if I can suspend my own process, but cannot resume other processes, how can my process ever get resumed?
Zaphod.
I am creating this system for my own use. What I want to created is a distributed system for controlling access to the file, so no central server. I am willing to use APIs but I don't want to have to run a service. I am creating a DLL which programs can reference which will manage all of the DB functions. Therefore I can trust the clients implicitly since I have a layer written by me between the user code and the file.
The programs that use the DLL may or may not be written by me.
The whole goal behind this exercise is to create a very easy to use DB, with no installation required, no services to run, etc. All that is needed is to register a DLL. I am somewhat disappointed that it is proving to be so difficult to do.
BTW, if I can suspend my own process, but cannot resume other processes, how can my process ever get resumed?
Zaphod.
I see that I told uncorrectly phrase.
You cannot suspend process. You can suspend one of threads of your process. You cannot suspend main thread of your process.
You can create named event, mutex or semaphore to allow or restrict access to resource for different processes. Then you will use system function WaitForSingleObject (or WaitForMultiplyObjects) to wait until resource will be released. (Don't forget to handle other events, e.g. DoEvents)
If you create COM objects you also can do it using COM events but it is slightly more difficult than using named syncronization objects.
MSDN:
CreateEvent http://msdn.microsoft.com/library/psdk/winbase/synchro_8ub8.htm
Syncronization objects http://msdn.microsoft.com/library/psdk/winbase/synchro_5gqb.htm
Wait functions
http://msdn.microsoft.com/library/psdk/winbase/synchro_5vqr.htm
Best regards.
You cannot suspend process. You can suspend one of threads of your process. You cannot suspend main thread of your process.
You can create named event, mutex or semaphore to allow or restrict access to resource for different processes. Then you will use system function WaitForSingleObject (or WaitForMultiplyObjects) to wait until resource will be released. (Don't forget to handle other events, e.g. DoEvents)
If you create COM objects you also can do it using COM events but it is slightly more difficult than using named syncronization objects.
MSDN:
CreateEvent http://msdn.microsoft.com/library/psdk/winbase/synchro_8ub8.htm
Syncronization objects http://msdn.microsoft.com/library/psdk/winbase/synchro_5gqb.htm
Wait functions
http://msdn.microsoft.com/library/psdk/winbase/synchro_5vqr.htm
Best regards.
ASKER
Hi,
I have done some research into all of these. I can do everything with them except differentiate between exclusive access and non-exclusive access. What I mean by this is that if two programs are currently reading the file, and then a third wants to write to it, that program needs to wait for the first two to finish. So what should he wait on?
Zaphod.
I have done some research into all of these. I can do everything with them except differentiate between exclusive access and non-exclusive access. What I mean by this is that if two programs are currently reading the file, and then a third wants to write to it, that program needs to wait for the first two to finish. So what should he wait on?
Zaphod.
You can use flag FILE_SHARE_READ in function CreateFile to restrict processes which wants to write to file.
Other variant (I think it is better for your case) is to use functions LockFile, LockFileEx and UnlockFile to lock part of file.
LockFile http://msdn.microsoft.com/library/psdk/winbase/filesio_63xh.htm
LockFileEx http://msdn.microsoft.com/library/psdk/winbase/filesio_39h4.htm
UnlockFile http://msdn.microsoft.com/library/psdk/winbase/filesio_1jl1.htm
Best regards.
Other variant (I think it is better for your case) is to use functions LockFile, LockFileEx and UnlockFile to lock part of file.
LockFile http://msdn.microsoft.com/library/psdk/winbase/filesio_63xh.htm
LockFileEx http://msdn.microsoft.com/library/psdk/winbase/filesio_39h4.htm
UnlockFile http://msdn.microsoft.com/library/psdk/winbase/filesio_1jl1.htm
Best regards.
ASKER
I looked into those as well, but how do I wait until the lock is released. AFAIK those functions will just returns errors if the file is already locked, or in use and so the process must keep retrying. I would like to queue up requests rather than spin locking.
Zaphod.
Zaphod.
To queue requests you have to create one object which will handle all your requests. Variants which I know how to do this: create service or COM server. COM server can be self-registered, as for service, you need to write some information to registry.
Your processes will send request messages to service (e.g. 'read x bytes from file y start from position z', 'write ...'). This queries can be placed to queue(s) (e.g. one file - one queue) and processed by service. To connect with clients you can use named pipes or COM interfaces. So you can send result of reading to clients from server or data to write from client to server.
Your processes will send request messages to service (e.g. 'read x bytes from file y start from position z', 'write ...'). This queries can be placed to queue(s) (e.g. one file - one queue) and processed by service. To connect with clients you can use named pipes or COM interfaces. So you can send result of reading to clients from server or data to write from client to server.
ASKER
But this requires running a service. Just to make sure there is no confusion, this will all be on one computer. I do not intend to support networked access to this file. As a result, things like mutexes and such work well for queuing and controlling access, I just don't know how to switch between concurrent access and exclusive access. Maybe there just is no way to do this using only windows without spinlocking, in which case I guess I will have to spinlock.
Zaphod.
Zaphod.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Using a mutex will suspend the process, so busy waiting is not required. Mutexes also support FIFO queues of processes waiting to gain ownership. This is the sort of set up I want. But it is looking like I will have to go for a compromise. Here is what I am currently thinking.
Each process that goes to read the file creates a unique mutex and places its name in the control file. When it is done reading it releases the mutex and removes its name from the control file. When a process comes to write, it tries to get each of the existing mutexes sequentially.
Zaphod.
Each process that goes to read the file creates a unique mutex and places its name in the control file. When it is done reading it releases the mutex and removes its name from the control file. When a process comes to write, it tries to get each of the existing mutexes sequentially.
Zaphod.
ASKER
The benefit of the method I just described is that it will handle process that do not cleanly exit, since those mutexes will be freed by windows.
ASKER
Thanks for all your help. I tried to implement the method I described but it turned out to be a nightmare. As a result, for now I will only allow exclusive access to the file using a mutex. If it becomes a performance issue, I will revisit this.
Zaphod.
Zaphod.
It is a hard situation really but there's i simple idea i don't know if it will work with you or not.
When a client needs to write to the file you may take temp copy from the file with any other name (don't make this name fixed or you will face the same problem again) and then open it to the client for editing and when he's done copy it again to the original file.
hope this will work with you
Good Luck:))