Data Buffer Size

You should never rely on the operating system to provide efficiency gains like this -- it is simply too hard to track or manage, and the last thing you want to do is start hacking your users' registry data to "fix" it.  

It is very simple to write your own buffer manager.  Allocate a buffer of, say, 64K, along with two variables (current read position and current end position).  Create a function to read from the buffer a given number of bytes.  The logic is then simple:
If BytesToRead < EndPos-CurrPos then
    Copy BytesToRead bytes from the buffer to the String
    Move forward CurrPos by BytesToRead
else
    Copy EndPos-CurrPos Bytes to the String
    Reduce BytesToRead by EndPos-CurrPos
    Read another 64K Block
    Set CurrPos to 0
    Set EndPos to 64K (or number of bytes read)
    Copy BytesToRead bytes from the buffer to the end of the String
    Move forward CurrPos by BytesToRead
endif

This is a great idea for reading as well as writing.  I have observed a LARGE number of applications that read or write a few bytes at a time, and the network overhead is simply horrible.  It's not until you show the developer the Wireshark trace that they actually realize what their code is doing.  I commend you for finding it on your own!

Bill:
Thanks for the response.   I've worked with you before on the Btrieve/PSQL forum.

I understand what you're saying about not relying on the OS..

The original program works fine for most all of my customers with the exception of one.

I have re-written my code to read entire file (47K) in as one chunk and on my local system wireshark shows one read but 32 - 1460 byte packets moving to workstation from server.  After some extensive testing we will include in our next product release.  But, for now, rather than recompiling 2 dozen production programs that use same code I'm looking for something to change on my customers system.

Can you point me to the registry entry that controls buffering across network?  I'd like to show it to my customers IT Dept as a possible interim solution.

Regards,

Lee

OK, you caught me -- it was a cop-out.  ;-)

My development work goes back to the early 80's, and I started using a network analyzer for application optimization in the early 90's.  I did a lot of work in the NetWare days and found that you could change the "Shared" setting on the file, or even set it to ReadOnly, in order to get the buffered network I/O to work for you.  Then, the clients started changing (the Win95 client was particularly nasty and changed every few months), and there were some client-side settings that started appearing, too.  As we moved to Windows servers, I simply lost the desire to try to track this any more and started to code buffering into all of my own applications that needed better performance.

In short, I've lost track of the settings that control this.  I've seen machines that buffer, and those that don't, and I simply have not been able to figure out the differences.  If you have two machines side-by-side, then you certainly have some options to debug this further.  You'll want to compare server-side OS, workstation side OS, use of any AV applications, type of share, file flags settings, number of users in at the same time, SMB/SMB2 protocol in use, and more.  Perhaps with a good comparison, you'll know more.  Of course, taking a machine (like your own laptop) that buffers in one environment and doesn't in another would exclude the workstation from the mix and help focus the search.

This might help, though:
    http://blogs.msdn.com/b/openspecification/archive/2009/04/10/smb-maximum-transmit-buffer-size-and-performance-tuning.aspx

Bill:
You've been most helpful.

I'm going to have my customer try using a different workstation and see if results are different.  I can't get to his site in Boston - I'm in Florida.

It's a new installation and I can sent a modified version of one production program that uses Big Buffer for reading files.  If that works then I'll go ahead and recompile everything and send him new system in 2-3 weeks.

By showing them speed improvement with modified code I'm sure I can get them to settle down and suffer with what they have for a short time.

On another note- if you care to address here -, sometime ago we discussed PSQL9 WGE program load speed in peer-peer network XP Workstation and Win7 WS hosting Data.  Win Firewall on Win7 ( W3dbsmgr.exe is listed as firewall exception) was causing startup speed problem for XP Workstation.  Removing Firewall solved problem but not a good solution. .  Any thoughts on this?  Have you experienced this with Goldstar customers?

Lee

PSQLv9 was never designed for use with Win7.  The FIRST version that is certified with Win7 is v10.30, so upgrading is an obvious solution.  

The newer installers provide the ability to install the database engine as a service, which ensures that the engine is started and ready to go when the OS boots up, as opposed to needing some to log into the server.  The old hack for PSQLv9 works, too, but not as well as the native install.  The newer installers also register the services with the Win7 firewall, as they are supposed to.

As with the app buffering, a network trace will show you exactly what the client is waiting for, and may yield a hint as to how to fix it.  If you can grab a sample & post it, then we can see what is going on....

Bill:
We run PSQL9 WGE as service on Win7.  Do we need to register the service with the Win7 firewall? If so, how's that done.

Attached WireShark dump illustrates buffer problem - Line 45 thru 6363  - approx 45 seconds to read 47K

Lee

forgot to click Attach -= rename extension to .pcap
BSO-Shark

Hey -- take a look at Packet 45 in the trace data, and open up the SMB decode in Wireshark.  Look at the various flags, file attributes, and other values.  Try comparing this to a system which *is* buffering the data, and see if there are notable differences.  I bet we find something in there.

Thanks Bill...
No PSQL component on server - it's just a remote drive (140 miles from workstation) - Just the one Workstation with W3dbsmgr,exe right now.

We use Btrieve primatives and once files are open it seems to work ok.

Thanks again for all your help.
I had better close this thread before we get too far from original subject.

I'll probably be asking for more at sometime in future.

Regards,

Lee