Solved

Data Buffer Size

Posted on 2012-04-04
11
441 Views
Last Modified: 2012-04-05
Am using VB to read a file opened for Random with record length of 16.
On my office network I am using WireShark to look at data traffic and see that Windows is sending me 4096 bytes from server and allowing me to access 16 bytes at a time -  no problem.

On a customers site - same program - there is no buffering occurring - each read of 16 bytes is being requested from server.

My question is ... where is the setup that allows buffering in 4K blocks for small data requests?

I's really sloooow!.

Any help is appreciated.
0
Comment
Question by:trims30
  • 6
  • 5
11 Comments
 
LVL 28

Expert Comment

by:Bill Bach
ID: 37810719
You should never rely on the operating system to provide efficiency gains like this -- it is simply too hard to track or manage, and the last thing you want to do is start hacking your users' registry data to "fix" it.  

It is very simple to write your own buffer manager.  Allocate a buffer of, say, 64K, along with two variables (current read position and current end position).  Create a function to read from the buffer a given number of bytes.  The logic is then simple:
If BytesToRead < EndPos-CurrPos then
    Copy BytesToRead bytes from the buffer to the String
    Move forward CurrPos by BytesToRead
else
    Copy EndPos-CurrPos Bytes to the String
    Reduce BytesToRead by EndPos-CurrPos
    Read another 64K Block
    Set CurrPos to 0
    Set EndPos to 64K (or number of bytes read)
    Copy BytesToRead bytes from the buffer to the end of the String
    Move forward CurrPos by BytesToRead
endif

This is a great idea for reading as well as writing.  I have observed a LARGE number of applications that read or write a few bytes at a time, and the network overhead is simply horrible.  It's not until you show the developer the Wireshark trace that they actually realize what their code is doing.  I commend you for finding it on your own!
0
 

Author Comment

by:trims30
ID: 37811061
Bill:
Thanks for the response.   I've worked with you before on the Btrieve/PSQL forum.

I understand what you're saying about not relying on the OS..

The original program works fine for most all of my customers with the exception of one.

I have re-written my code to read entire file (47K) in as one chunk and on my local system wireshark shows one read but 32 - 1460 byte packets moving to workstation from server.  After some extensive testing we will include in our next product release.  But, for now, rather than recompiling 2 dozen production programs that use same code I'm looking for something to change on my customers system.

Can you point me to the registry entry that controls buffering across network?  I'd like to show it to my customers IT Dept as a possible interim solution.

Regards,

Lee
0
 
LVL 28

Expert Comment

by:Bill Bach
ID: 37811575
OK, you caught me -- it was a cop-out.  ;-)

My development work goes back to the early 80's, and I started using a network analyzer for application optimization in the early 90's.  I did a lot of work in the NetWare days and found that you could change the "Shared" setting on the file, or even set it to ReadOnly, in order to get the buffered network I/O to work for you.  Then, the clients started changing (the Win95 client was particularly nasty and changed every few months), and there were some client-side settings that started appearing, too.  As we moved to Windows servers, I simply lost the desire to try to track this any more and started to code buffering into all of my own applications that needed better performance.

In short, I've lost track of the settings that control this.  I've seen machines that buffer, and those that don't, and I simply have not been able to figure out the differences.  If you have two machines side-by-side, then you certainly have some options to debug this further.  You'll want to compare server-side OS, workstation side OS, use of any AV applications, type of share, file flags settings, number of users in at the same time, SMB/SMB2 protocol in use, and more.  Perhaps with a good comparison, you'll know more.  Of course, taking a machine (like your own laptop) that buffers in one environment and doesn't in another would exclude the workstation from the mix and help focus the search.
0
 
LVL 28

Expert Comment

by:Bill Bach
ID: 37811599
0
 

Author Comment

by:trims30
ID: 37811759
Bill:
You've been most helpful.

I'm going to have my customer try using a different workstation and see if results are different.  I can't get to his site in Boston - I'm in Florida.

It's a new installation and I can sent a modified version of one production program that uses Big Buffer for reading files.  If that works then I'll go ahead and recompile everything and send him new system in 2-3 weeks.

By showing them speed improvement with modified code I'm sure I can get them to settle down and suffer with what they have for a short time.

On another note- if you care to address here -, sometime ago we discussed PSQL9 WGE program load speed in peer-peer network XP Workstation and Win7 WS hosting Data.  Win Firewall on Win7 ( W3dbsmgr.exe is listed as firewall exception) was causing startup speed problem for XP Workstation.  Removing Firewall solved problem but not a good solution. .  Any thoughts on this?  Have you experienced this with Goldstar customers?

Lee
0
Network it in WD Red

There's an industry-leading WD Red drive for every compatible NAS system to help fulfill your data storage needs. With drives up to 8TB, WD Red offers a wide array of solutions for customers looking to build the biggest, best-performing NAS storage solution.  

 
LVL 28

Expert Comment

by:Bill Bach
ID: 37811849
PSQLv9 was never designed for use with Win7.  The FIRST version that is certified with Win7 is v10.30, so upgrading is an obvious solution.  

The newer installers provide the ability to install the database engine as a service, which ensures that the engine is started and ready to go when the OS boots up, as opposed to needing some to log into the server.  The old hack for PSQLv9 works, too, but not as well as the native install.  The newer installers also register the services with the Win7 firewall, as they are supposed to.

As with the app buffering, a network trace will show you exactly what the client is waiting for, and may yield a hint as to how to fix it.  If you can grab a sample & post it, then we can see what is going on....
0
 

Author Comment

by:trims30
ID: 37812115
Bill:
We run PSQL9 WGE as service on Win7.  Do we need to register the service with the Win7 firewall? If so, how's that done.

Attached WireShark dump illustrates buffer problem - Line 45 thru 6363  - approx 45 seconds to read 47K

Lee
0
 

Author Comment

by:trims30
ID: 37812134
forgot to click Attach -= rename extension to .pcap
BSO-Shark
0
 
LVL 28

Accepted Solution

by:
Bill Bach earned 500 total points
ID: 37812257
I see what you mean about the slow startup.  The reading of the non-Pervasive files is taking up almost a full minute! Ugh!

As for the engine startup, here's what is happening.  First, the database client attempts to contact the remote server at 10.10.0.27 via the Btrieve port.  It tries this several times, and gets rejected each time.  You can see this at packets 6890, 6907, etc. This tells us that something is blocking access to the database engine, or that there is no database engine on the server.  

Then, after wasting another 6.2 seconds waiting for the server to respond, the LOCAL Workgroup Engine activates and starts to access the files itself.  This is fine if the engine is SUPPOSED to be on the client, instead of the server.  However, this now makes the workstation the database server, and hampers performance even further.

So, the question is now: Is there a database engine on the server that should be used?  If so, then you need to address the firewall issue.  try opening port 3351 manually & see if that helps.  If there is NO database engine on the server, then you can try to reconfigure the environment, but it definitely will work VERY slowly across this WAN link.
0
 
LVL 28

Expert Comment

by:Bill Bach
ID: 37812323
Hey -- take a look at Packet 45 in the trace data, and open up the SMB decode in Wireshark.  Look at the various flags, file attributes, and other values.  Try comparing this to a system which *is* buffering the data, and see if there are notable differences.  I bet we find something in there.
0
 

Author Comment

by:trims30
ID: 37812330
Thanks Bill...
No PSQL component on server - it's just a remote drive (140 miles from workstation) - Just the one Workstation with W3dbsmgr,exe right now.

We use Btrieve primatives and once files are open it seems to work ok.

Thanks again for all your help.
I had better close this thread before we get too far from original subject.

I'll probably be asking for more at sometime in future.

Regards,

Lee
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Suggested Solutions

A quick step-by-step overview of installing and configuring Carbonite Server Backup.
I'm a big fan of Windows' offline folder caching and have used it on my laptops for over a decade.  One thing I don't like about it, however, is how difficult Microsoft has made it for the cache to be moved out of the Windows folder.  Here's how to …
This Micro Tutorial will teach you the basics of configuring your computer to improve its speed. It will also teach you how to disable programs that are running in the background simultaneously. This will be demonstrated using Windows 7 operating…
This Micro Tutorial will give you a introduction in two parts how to utilize Windows Live Movie Maker to its maximum capability. This will be demonstrated using Windows Live Movie Maker on Windows 7 operating system.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now