Why is connection speed between these sites so SLOW!!!

So are you saying that each site gets those speeds when doing speedtests online, something like speedtest.net, but when you transfer data between the two locations, you only be 250kbps?

What transfer protocol are you using to measure the 250Kbps?

let me just clarify some concerns and questions raised:

1) i know it's 250kbps several ways - for instance, just looking at the folder the file is downloading to, and hitting the 'refresh' button every few seconds and noting the change in the size of the file (it's a 40GB file).  Also i've used several network traffic monitors, both built into windows, and 3rd party

2) yes speedtests show those speeds, but i've downloaded large files, such as 4.5GB DVD images from microsoft, that are downloading at 2.0MB/sec.

3) I'm aware of the upstream in one site / downstream in the other, but as you stated, when both sides have a guaranteed 16x2, and in speedtests and real-life transfers/uploads they are hitting 2 MegaBYTES per second, it doesn't make any sense why between the two connections, just a mile or two apart, my speed would be so limited.

4) DURING the transfers (sometimes at 3 in the morning - hardly peak times), i've remoted into both computers (one in each site), and run speed-tests, and they're all blazing fast.

This sounds like a problem with something in the OS itself. Just out of curiosity, what operating systems are you running on each side?

So even while you're seeing the very slow transfer, you can run a concurrent speed test and it shows normal speeds?

I still think you'll need to examine the actual traffic during the transfer to look for reason why it's so slow. Things that might explain such a slow speed could be packet loss, excessive retransmits, or fragmentation. To be honest, I don't think that's going to be the case though. Other traffic would be affected the same way. The more I think about it, the more I think Comcast needs to be asked directly to look into this.

Is there a chance this traffic could look anything like file sharing traffic or something like that? Stuff Comcast might be motivated to throttle? Shouldn't happen, but...

Or, is this going through a VPN tunnel?

You have said that you can see that the file transfer is only happening at 250kbps, but you haven't mentioned what type of transfer it is.  Do you have a VPN setup between the two, and doing a windows file sharing style transfer?  Is it an ftp upload between the two sites?  I suspect that the problem is not with your connection, but with how the files are being transfered.

>Both sites are within 2 miles of each other
That does not mean anything, especially with Comcast. They are prone to sub-optimal routing

>16x2, but I just did a speedtest in both sites, and they're both at 22Mbps down and around 7 or 8Mbps up.

That is a good test, this at least shows that both circuits are at least capable of that!

>I would have thought being that they have a much higher up-stream (and down), and plus the fact they're on the same network, only 2 miles (less) apart, that the transfers would be much quicker

Just because they are on the same network and very close by (Physically, does not mean nothing, maybe on a network that was locally on the same subnet), but I am sure you are going  through a few routers to get to each other. You have to provide some details in order for us to provide you with adequate information.

Can you please provide a traceroute to and from the offices, I would like to see the latency and path both offices are taking to get to each other.

Also, you have to take into consideration that you are not guaranteed anything with a cable connection, the network is shared by many customers; so depending on what the available bandwidth is compared to the aggregate bandwidth at the CMTS, the upstream link might be saturated at the time of your downloads/uploads between sites.

Billy

both are windows 2003 servers, and one is hosting an FTP site, so the traffic is FTP.  the only boxes they're going through on my end are an ASA at the one site, and a PIX at the other.  No VPN.

From one site to the other:
  1   126 ms     1 ms   126 ms  192.168.1.2
  2     *        *        *     Request timed out.
  3   283 ms   378 ms   432 ms  75-148-130-210-Houston.hfc.comcastbusiness.net [
75.148.130.210]

i'm not even sure where to start with looking into packet loss, etc.

if comcast is throttling FTP traffic, i'm going to be mad =)

A useful test would be to run the traffic through a VPN tunnel then to shield it from Comcast's eyes, to see if there's a difference. You could also try running it on a non-standard port, though that my very well not make a difference either way.

yeah 432ms is high latency for 2 miles that is what your issue is more than likely, can you provide a traceroute from the other side?

You can also see that at hop 1 you are at 126ms, too high for the first hop.


Also, from each site can you please provide bandwidth test from:

http://129.119.200.191:7123/

After each test, click on statistics and copy and paste the information please.

Thanks
Billy

As for packet loss, as simple test would be a continuous ping, using ping -t ipaddress.  Let it run for a few minutes to get a good idea.  If you have access to a linux machine, you can change how long it waits before sending out another packet, and get an even better idea.  I am not sure the command line arguments exactly, but look for flood pinging.  There are also alternate ping utilities that you can download for Windows that will flood ping.  The best way to watch for packet loss would be through a wireshark analysis of the actual transfer.

As another test to determine what is causing the problems, can you try to ftp to the recieving site from a different source?  That will help determine if it is something in the mix of the two locations, or if it is simply the receiving end.  Do you do transfers both directions? If so, are they slow both ways?

To answer some questions, i performed the following tests:

1) speed test from both sites, each with no bandwidth being used for other applications.
Site 1: 30Mbit down / 6Mbit up
Site 2: 18Mbit down / 6Mbit up

2) Tracert from one site to the other

Site 1 to 2:
  1     *        *        *     Request timed out.
  2    11 ms    12 ms    11 ms  c-98-197-129-214.hsd1.tx.comcast.net [98.197.129
.214]

Site 2 to 1:  
1    <1 ms    <1 ms    <1 ms  192.168.1.2
  2     *        *        *     Request timed out.
  3   143 ms    14 ms    17 ms  75-148-130-210-Houston.hfc.comcastbusiness.net [
75.148.130.210]

3) FTP transfer over Comcast network
From site 1 to site 2: ~200KBps
From site 2 to site 1: ~880KBps (THIS is more like it!! - but only one direction... even though both sites have the same upload speed of 6Mbit)

Windows File Transfer over a VPN connection
From site 1 to site 2: starts out at 900KB/sec; after a few seconds jumps to EXACTLY ~250KBps
From site 2 to site 1: starts out at 900KB/sec; after a few seconds jumps to EXACTLY ~250KBps

TEST TO THIRD PARTY SERVER (on 20Mbit synchronous fiber connection)
From site1 to 3rd party server: ~200KBps (???)
From 3rd party server to site1: ~750KBps
From site2 to 3rd party server: ~220KBps
From 3rd party server to site2: ~575KBps

ADDITIONAL SPEED TESTS:
Speakeasy.net/Speedtest/
-Site1 - 20Mbit down / 7.5Mbit up
-Site2 - 18.63Mbit down / 6.46Mbit up

Bandwidthplace.com
-Site1 - 18Mbit down / 6.5Mbit up
-Site2 - 18.5Mbit down / 5.5Mbit up


Also, rfc1180 - that website - the application on it isn't loading.  Just shows up as a red 'x'.  Tried on several computers and several browsers, with the same result.

There's a difference between Kbps and KBps, and i'm aware of that.  KBps is kiloBITS per second.  kbps is kiloBYTES per second.  As in, 250kbps = 2MBps.  Capital B indicating bits, not bytes.

No, i'm not seeing the speed i should be seeing.  There is no way i should be seeing 200kbps (for the record, that's 200 kilo BYTES per second).  if that's my sustained speed, then that means my connection is 1.6Mbit (or 1.6MBps, whichever you prefer).

Correct - so the site to site transfers will top out at 250kbyte/s (exactly the 2MB upload limit). Obviously 200kbyte/s is somewhat below that, but that might happen some of the time due to sub-optimal conditions such as other traffic or possible congestion.

>KBps is kiloBITS per second.  kbps is kiloBYTES per second.  As in, 250kbps = 2MBps.  Capital B indicating bits, not bytes.

I'm afraid I will have to disagree; kb = kilo bits and KB = kilo bytes

http://www.speedguide.net/read_articles.php?id=115

250kbps (kilo bits per second) =  31.25KBps (kilo bytes per second)
There are 8bits to a byte, therefore you will have to divide 250000/8 = 31250

Billy

You mentioned you are running Windows Server 2003 on both ends. I'm betting anything that it has something to do with TcpWindowSize. Try these three websites.

http://www.speedguide.net/read_articles.php?id=157

http://www.petri.co.il/increase_internet_connection_speed_in_windows_xp.htm

http://www.petri.co.il/increase_broadband_connection_speed_in_windows_xp_2003.htm

rfc1180 is correct on this one.  If you have been saying that your file is increasing in size by 1 megabyte every four seconds(or about 250 kilobytes every second), this is the expect file transfer of a 2 megabit upload.  Anton74's mention of PowerBoost is why the speed does not maintain the initial high rate you see, and it was a good catch on his part for noticing the disrepency here.  On your connection, the 45 gigabyte file should take the following time to transfer:

45 GB * 1024 MB/GB = 46,080 MB
46,080 MB * 1024 KB/MB = 47,185,920 KB
47,185,920 KB * 1024 KB/MB = 48,318,382,080 Bytes
48,318,382,080 Bytes * 8 bit/Byte = 386,547,056,640 bits
386,547,056,640 bits / 1000 bit/Kb = 386,547,056.64 Kb
386,547,056.64 Kb / 1000 Kb/Mb = 386,547.05664 Mb
386,547.05664 Mb / 2Mbps = 193,273.52832 s
193,273.52832 s / 3600s/hour = 53.6870912 hours

Network speeds(ethernet, dsl, cable) are always measured in powers of 10 bits per second, not 2^8 bytes per second.

Note that the 45GB was a *download* from Microsoft; it would have been subject to the 16Mbit/s download rate limit, not the 2Mbit/s upload limit.

Sorry, I did mess up there a bit.  From the author's first comment after the initial question, he stated in number 1, that he was transfering a 40GB file, not a 45GB file.

Oops - I goofed a bit on that one too, I was confused with the 4.5GB DVD image download mentioned earlier on. So your use of the 2Mbit/s limit in your calculation is correct.

I think the point we're both making is that the throughput is pretty much what is expected.

Somewhat offtopic - I don't know the nature of the data that needs to be transferred, but maybe rsync can help reduce how much needs to be transferred in the first place?

;the information that has been provided already (Traceroutes) indicateds high latency and the author indicated packet loss; which this information packet loss alone will degrade performance and also cause the network to appear as if there are throughput issues as TCP will have to retransmit.
in my previous comment:
 
yeah 432ms is high latency for 2 miles that is what your issue is more than likely, can you provide a traceroute from the other side?

You can also see that at hop 1 you are at 126ms, too high for the first hop.


Also, from each site can you please provide bandwidth test from:

http://129.119.200.191:7123/

After each test, click on statistics and copy and paste the information please.

Thanks
Billy

Comcast is throttling the traffic or something on there end is causing this. Talk to them about it. I think this beyond the help anyone here can provide.

So you ever figure out what the problem was?

Yes, so sorry for the long delay.  Turns out, as expected, powerboost was responsible for my "16x2"  circuit showing "32x8" speeds.  oh well...

So the 250kbps was EXACTLY the 2Mbit they provided me :/