Why is it taking so much SMB traffic to transfer a few MB of data over a 100M Ethernet WAN?

We are experiencing unbelievably long delays opening database files (Pervasive SQL) over a Wide Area Network.   We can open application (.exe) files and get upward of 30M/sec data transfers.  Our ping latency is <2ms.  But it takes over 4 minutes to open a 5-50M image file.  Wireshark shows us that most of the traffic is SMB:  Read AndX Request; Read AndX Response.  Once this sequence is complete (SMB Close Request), the actual data transfers in 1 or 2 seconds.

Another interesting Wireshark observation:  The same application, running on a local 100M segment, shows similar symptoms, BUT, the SMB AndX Request/Response traffic lasts approximately 30 seconds instead of 240.

We have adjusted the MSS size on our routers to accomodate the carrier's 155M Ring routing.  Our optimum MSS size is 500  (ip tcp adjust-mss 500) on all interfaces.  This adjustment fixed our tcp retransmission problems over the wan.

Our application is patient treatment - this issue effects patient care.

Here is a condensed view of the beginning and end of the WAN SMB Traffic:

132      26.495153      TCP      dbreporter > simbaexpress [FIN, ACK] Seq=1277 Ack=4732 Win=64771 Len=0
133      26.495995      SMB      Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \DB\IMAGES\48\00004d88\0000be65.w01
134      26.497291      TCP      simbaexpress > dbreporter [ACK] Seq=4732 Ack=1278 Win=65535 Len=0
135      26.497308      TCP      simbaexpress > dbreporter [FIN, ACK] Seq=4732 Ack=1278 Win=65535 Len=0
136      26.497323      TCP      dbreporter > simbaexpress [ACK] Seq=1278 Ack=4733 Win=64771 Len=0
137      26.498291      SMB      Trans2 Response, QUERY_PATH_INFO, Error: STATUS_OBJECT_NAME_NOT_FOUND
138      26.498536      SMB      Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \DB\IMAGES\48\00004d88\0000be65.d01
139      26.501115      SMB      Trans2 Response, QUERY_PATH_INFO
140      26.501418      SMB      Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path:
141      26.503835      SMB      Trans2 Response, QUERY_PATH_INFO
142      26.503973      SMB      Trans2 Request, QUERY_FS_INFO, Query FS Attribute Info
143      26.506256      SMB      Trans2 Response, QUERY_FS_INFO
144      26.506459      SMB      NT Create AndX Request, Path: \DB\IMAGES\48\00004d88\0000be65.d01
145      26.522334      SMB      NT Create AndX Response, FID: 0x405d
146      26.522593      SMB      Trans2 Request, QUERY_FILE_INFO, FID: 0x405d, Query File Internal Info
147      26.524931      SMB      Trans2 Response, FID: 0x405d, QUERY_FILE_INFO
148      26.525126      SMB      Trans2 Request, QUERY_FILE_INFO, FID: 0x405d, Query File Standard Info
149      26.527410      SMB      Trans2 Response, FID: 0x405d, QUERY_FILE_INFO
150      26.527561      SMB      Read AndX Request, FID: 0x405d, 256 bytes at offset 0
151      26.530026      SMB      Read AndX Response, FID: 0x405d, 256 bytes

Read AndX Request and Read AndX Response sequence continues for about 4 minutes, adding 256 bytes to the offset with each exchange.

169054      253.840054      SMB      Read AndX Request, FID: 0x405d, 256 bytes at offset 21598048
169055      253.842274      SMB      Read AndX Response, FID: 0x405d, 0 bytes
169056      253.842483      SMB      Close Request, FID: 0x405d
169057      253.844716      SMB      Close Response, FID: 0x405d
169058      253.848252      TCP      iclpv-dm > simbaexpress [SYN] Seq=0 Win=65535 Len=0 MSS=1460
169059      253.850374      TCP      simbaexpress > iclpv-dm [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Bill BachPresident and Btrieve GuruCommented:
Have you asked the developer if the application is specifically written to read 256 bytes at a time?

You could probably use Process Monitor to watch the workstation, and see if you see repeated 256-byte reads.
If this is the case, then the developer needs to update the app to read larger (e.g. 64K) chunks.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jim P.Commented:
Another thing to consider is that a LAN is done as MegaBytes per second. A WAN is Megabits per second. Divide by 8 to get the speed.
--> Our optimum MSS size is 500  

This is part of your problem.  On your LAN you MSS is going to be 1460, this means over the ATM link you are going to need to do 3 transfer requests for every 1 on your LAN.

Of course the fact that you are actually only doing 256 bytes per transfer (as BillBach pointed out) means that you are actually doing 6 requests for every 1 on the LAN.

I'm not sure you can actully get up to the 64K chunks though, IIRC SMB requests the DF bit to be on and so the largest chunck it can send is equal to the smallest MTU along the path.  If your ATM's MTU only allows for a 500 byte MSS, then the most data SMB will read in a chunk is 500.

You need to try and get the ATM link to do at least a MSS of 1460, equal to that of your LAN.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Jimpen, wrong LAN speeds are bits also.  It is 100 mega bits on a LAN not bytes.  ATM running at 155 Mbps is faster than a 100 Mbps LAN.
Jim P.Commented:

Thanks for the correction. Its been a while since I've had to delve deeply into LAN/WAN speeds and in-depth network aspects.

Once you get them setup correctly it fades. ;-(
nociSoftware EngineerCommented:
You introduce a small delay between every packet waiting for ACK ==> less packets = less waiting (besides the extra overhead lots of small work causes).
Big transfers help in throughput.  BTW  SMB can send block of up to 64 KB at one (if needed, yes they will get fragmented but upfront.

As a general side node:
If you want to access databases remotely then please look into tools like mysql, mssql, oracle...
There you send the query over the wire and get only the answer back. (not all intermediate search data).
With access (without the mssql - light), dbase (II & III), pervasive etc. you can download the file multiple time looking for values in all the index buckets and then go over the file again to access data.

This kind of databases was allways meant to be used locally and not over a network. That it can be done doesn't imply that it is the best solution.
aspwvAuthor Commented:
You pointed me in the right direction.  I wasn't sure which layer the problem was created on.  When I dug into it at the application layer, I discovered that it was caused by a digital camera being set at too high of a resolution:  They typically used image fles less than 1M for ID photos; recently, they started using the highest resolution on the camera - > 20M images.  The resulting delay was acceptable on the LAN, totally unacceptable over the WAN.

Thanks for your help!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis

From novice to tech pro — start learning today.