Visual FoxPro Application hangs when accessed from multiple workstations

Hoping to find some nudging in the right direction with this issue.  We cannot say exactly when this started, neither can the end users, but slowly certain functions of this custom FoxPro application have started to take 10-15 seconds to respond, when the response used to be instantaneous.   The program will also hang periodically for each user, once a day or more.  
All clients are Windows 7 Pro, Server is SBS2008, all machines connect at gigabit speed.  

We have been working for a few weeks on the issue and tried a variety of fixes that resolved these types of issues for others.   Some of the changes we have made are:

1.  Disabling SMB2, enabling SMB1, disabling OpLocks on server/client
2.  Installed various hotfixes = see attached
3.   Reindexed the foxpro tables
4.  Changed various settings on the NIC, interrupt moderation disabled, Jumbo Packet changed value from 1514 to 9014,  Large Send Offload/TCP Connection Offload/TCP-UDP Checksum offload all disabled
5.  Moved shared Foxpro data to a workstation and tested accessing - issue went away so we contacted Microsoft Suppor but haven't tested with multiple users accessing the data at once this way (currently in progress of testing this scenario)
6.  Contacted Microsoft Critical Business Support - they did similar work to the above notes but also updated all TCP/IP dll's to most current version, took network logs on both server and client during the Application hang ups which their senior analysts looked at (weren't able to come up with anything specific, only that there was much more network traffic showing when FoxPro data was shared on the server vs when we did test it shared from a workstation)   Microsoft is at the end of helping us due to FoxPro not being supported any longer.  They stated this would be a "Best Effort" type situation.
7.  A local cabling company did thorough test on network cabling - a few small issues were discovered and resolved but nothing major


I'm hoping others that have run into this with aging FoxPro applications can shed some light on what may be going on here.
hotfixes.jpg
thecomputerplaceAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

thecomputerplaceAuthor Commented:
Just tested sharing the FoxPro data from a Windows 7 machine and seeing the same results - program doesn't lag when only open from a single workstation but when accessed by multiple workstations bringing up a work order takes 10-15 seconds (10-15 times as long).
pcelbaCommented:
It seems you did everything you could do and we all are sure the problem is in Microsoft software... Microsoft is doing everything to cease shared files access. The most profitable way is client-server today. (Cloud server preferably.)

The end of FoxPro support should not mean end of Win32 API based applications support and you should not simply accept that explanation from Microsoft.

Visual FoxPro just opens some server files for shared access and calls appropriate Windows functions (via Win32 API) for this purpose. You may obtain the list of these functions in e.g. Process Monitor when you set the filter to "vfp9.exe" process. Then you may ask Microsoft whether these functions are still supported or not. (I don't have exact function names available but four of them are the most important ones: CreateFile, ReadFile, LockFile, and WriteFile).

Running Process Monitor on several workstations could show which Win32 API function causes the processing delay.

Next step should be to write a small C application which calls exactly same Win32 API functions as your VFP application and then to ask Microsoft for the help because I suppose the C language is still supported and it should provide exactly same delays as VFP program...

Even the MS Access database placed on the server and accessed from several workstations should provide exactly same (bad) results. This could also be of Microsoft interest...

OK, back to the real (FoxPro) world... To have as much data locally as possible is the solution. Server data should be updated when necessary only. This means app changes... Are you ready for them?

Another (simpler but more expensive) solution is to access the app via Remote Desktops. The app resides on a Terminal Server and all app data are stored locally on this server. Users can access the app via Remote Desktops. The most expensive parts of this solution are client licenses and also the server hardware when the number of concurrent users exceeds certain level. But the app runs without any code change obviously.

VFP app can support hundred and more concurrent users when running as multithreaded COM server. This, of course, requires major app rewrite but it also costs much less than SQL Server solution.

The last solution is to downgrade OS for FoxPro app to Windows XP/Server 2003.

BTW, did you switch SMB2 off on both the server and all workstations?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Olaf DoschkeSoftware DeveloperCommented:
The last behaviour you report points to SMB/OpLocks, but you addressed that as point 1. Are you sure Oplocks don't occur? There are several instructions and tools you can use, but what did you do? Did you reboot the server? To see whether the registry changes or tools you used worked maybe it would be good to monitor, whether oplocks occur. Googling "oplock monitoring" I find a solution at netapp.com.

The situation might get better using a normal instead of SBS server OS, you could test a 2012R2 server within the 30 day trial to see if migrating there would help. If so, it'd be a hard decision to accept higher CAL costs. As alternative option you might migrate the VFP data to a Linux Server and create a Samba file share. If the server only shares the data and no windows process runs serverside, that'd also work.

There absolutely were no recent changes to the software or database? Only the data growth? And that also has been normal growth? Performance degrades and that can degrade worse than linear, but jumping from split seconds to a factor 10-15 wouldn't be normal.

A jump of network traffic can point to bad joins. A missing where clause or join condition can cause a scale factor of queries doing a cross join. Due to SET FILTER on a result set that might not become apparent, but i would be seen on the network traffic, which you did report as jumping higher.

Bye, Olaf.
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

thecomputerplaceAuthor Commented:
Sorry for the delay in reply - thank you both for you input on this.   While I did disable SMB2 on both server and client when testing I did not disable it on all, only the ones I was testing with, which may be part of my problem.  I did disable oplocks as well but did not use any monitoring tools to verify it was turned off, just changed the registry setting, rebooted server/client and tested the app.  

I'm not a programmer, so changing the code is out of the question for me.

There are no changes to the software or database that we are aware of.  Its not something that is updated or maintained by the vendor (a one man gig), he is only called on in times of need.

I'll do more testing and report back - thanks
thecomputerplaceAuthor Commented:
I've tested this more - just using a small network switch with a couple clients connected to the server - same issues.  I wasn't able to verify whether Oplocks are occurring or not but will continue to test for that.    

We have tested Terminal Services and this appears to work fine - if no other resolution comes forth this may be our go to.

pcelba - you mentioned  "did you switch SMB2 off on both the server and all workstations?"   If i had only done this on the ones testing with is that an issue?

Microsoft continues to push the case away saying its only a best-effort case as FoxPro isn't supported - i've pushed them with arguments aided by your points (appreciated) and asked for a move from the SBS team to one more well versed with this type of issue.   Awaiting the supervisor of the tech i was working with to get back to me.
pcelbaCommented:
SMB2 must be turned off on the server and all workstations used for the testing. All other workstations which do not open DBF files are irrelevant.
thecomputerplaceAuthor Commented:
That's how I tested - I'll have to look at OpLocks more to verify its actually off - I'll test and get back - thanks
thecomputerplaceAuthor Commented:
Nothing I tried helped the speed of this VFP App over the network  - Microsoft continued to troubleshoot and gave us access to a Senior technician.  We tried various fixes, updating system files, patches, disabling various NIC settings, etc.  Took network and process traces while running the VFP Application when it hangs vs when it runs fine to compare the two, nothing conclusive from Microsoft Tech Support.  Tried another network card in case the driver/a nic setting was causing issue, didn't resolve it.   I tested recently with sharing the data from a Windows XP machine in a test environment (not XP Mode, tested that initially but that didn't work) and the issue seems to have gone away.
 
Planning to test in the production environment to see if any bugs occur,  will update when i have tested.
Wei BostwickCommented:
my family wanted form name a few weeks ago and was informed about a document management site that has 6,000,000 forms . If others need Acord 127 also , here's a link http://goo.gl/Ms4ebj
thecomputerplaceAuthor Commented:
Hosting the VFP App on another machine running Windows XP natively (not via Windows 7's XP mode) resolved the issues instantly and completely although we did not put this fix into productioin.  Some developers are looking at rewriting the app and a server replacement may be in the works.   Thanks to all who contributed.  I learned much through these trials.  I appreciate the effort.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
FoxPro

From novice to tech pro — start learning today.