DS3 dedicated pipe will not do 45Mbs will only go 4.5Mbs per stream

Seems a little odd to me too... It may be that your DS3 is configured channelized and not clear, and their routing config will only alow one flow per channel at a time, effectively limiting the data rate of any single flow ot the capacity of a channel; problem with that theory being that a DS3 channel is generally one DS1, roughly 1.5mbit/s, and it makes no sense that they'd treat the DS3 as bundles of 3 DS1 channels...

I would open a ticket with the carrier, and ask for a specific technical document that explains what causes this per-flow bandwidth cap. I wouldn't expect actually getting the full 45mbit/s out of a full pipe T3, there is some overhead, but still 7mb is ridiculous.

If you want to paste back in that doco they provide (you'll probably have to escalate and press them consistently to get it), I'd be happy to review it and translate their network-guy-eese into english for you :)

Unfortunately, I suspect that if they had a pat answer all ready for you, many many many others have complained, this is probably a limitation of their network engineering in their cloud, and your only way to get around it may be another carrier.

I spoke with Cisco and they explain it as the ISP using Perflow not per Link, We have been waiting for this connection for 6 months and when it gets here we run into this. I just want ot make sure that if everyone is in agreement then if I copy a 10MByte file It should do better then 30 Seconds an diff I copy a 100Mbyte file is around 5 minutes. I calcualte this at under 5Mbits/sec. This is a dedicated Pipe I figured I should see at least 30Mbits on a single stream with no other traffic on the IPVPN.

We tested until 10pm last night and excluding Our side and the remote side they tested from Co to CO they saw the &.5mbs using Iperf but drop it an dwent back to testing end to end mutithread to claim they are giving us 45Mbs..

Other then ranting I gues I am asking what should I expect for a IPVPN point to point with nothing else on it.

Here is from Router to Router

transmit or receive [receive]: r
perform tcp half close [n]:
receive buflen [8192]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
rcvwndsize [512360]:
delayed ACK [y]:
show tcp information at end [n]: yes

ttcp-r: buflen=8192, align=16384/0, port=5001
rcvwndsize=512360, delayedack=yes  tcp
ttcp-r: accept from 74.203.95.50 (mss 536, sndwnd 512360, rcvwnd 512360)
ttcp-r: 16777216 bytes in 18628 ms (18.628 real seconds) (~878 kB/s) +++
ttcp-r: 19520 I/O calls
ttcp-r: 0 sleeps (0 ms total) (0 ms average)
Connection state is CLOSEWAIT, I/O status: 7, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255
Local host: 192.168.130.1, Local port: 5001
Foreign host: 74.203.95.50, Foreign port: 64785
Connection tableid (VRF): 0

Enqueued packets for retransmit: 0, input: 0  mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x5620C06C):
Timer          Starts    Wakeups            Next
Retrans             1          0             0x0
TimeWait            0          0             0x0
AckHold         19049          0             0x0
SendWnd             0          0             0x0
KeepAlive       32770          0      0x5621AAC8
GiveUp              0          0             0x0
PmtuAger            0          0             0x0
DeadWait            0          0             0x0
Linger              0          0             0x0
ProcessQ            0          0             0x0

iss:  773075009  snduna:  773075010  sndnxt:  773075010     sndwnd: 512360
irs: 1703317821  rcvnxt: 1720095039  rcvwnd:     512208  delrcvwnd:    152

SRTT: 37 ms, RTTO: 1837 ms, RTV: 1800 ms, KRTT: 0 ms
minRTT: 28 ms, maxRTT: 300 ms, ACK hold: 200 ms
Status Flags: passive open, retransmission timeout, gen tcbs
Option Flags: keepalive running, win-scale
IP Precedence value : 0

Datagrams (max data segment is 536 bytes):
Rcvd: 32771 (out of order: 0), with data: 32768, total data bytes: 16777216
Sent: 30722 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 0, total data bytes: 0
 Packets received in fast path: 0, fast processed: 0, slow path: 0
 fast lock acquisition failures: 0, slow path: 0

if you were using a clear channel T3 you should see a 10M Byte file go across in roughly 2 seconds, best case. Likewise, a 100M byte file should be 20 seconds best case. Real world with other traffic, maybe 2-3x that long.

Can you expand a little on what the Cisco rep told you about "per flow" versus "per link"? Do you control the routers at each end so you could paste in a config ?

Yes we have 2851's at both ends. Cisco did not go into detil other then to say that the configuration we are using should not cause anytype of rate limiting. Basically the problem is not the routers.. Its going a little better this morning 10 meg file going to the remote side was 18 seconds and coming back was about 12 secnds..

Is your carrier providing you with an external CSU/DSU on each end, or are you using integrated DSUs in the 2800s?

Etherernet handoff on both ends from a ONS 15310 on one side and oveture on the other side.

Ah ok, then as long as you have no config in your 2800's doing any kind of flow throttling, you're definitely looking at a carrier config issue or engineering design limitation.

As equalizer pointed out (i think) in his comment, you should expect something under full line rate on a file transfer due to protocol and application overhead. FTP on a huge system (some big unix box for instance) is probably your best bet for a simulation of a real world best case transfer.

I would still ask the carrier to provide documentation as to "what is the highest data rate transfer that they will be able to support in a single flow", and an explanation as to why if its anything less than nearly full line rate. If they cannot or will not provide this info, of if they cannot change their config to permit you to fully use your purchased bandwidth, I would then begin preparing to change carriers, and ask the question up front of the next round of bidders.

I called our rep over at CDW and they are goign to get find me a couple of outside vars to come in at do some testing. the ISP is claiming that they are getting 40Mbs straight up end to end test one stream via TCP. if I test using cisco's TTCP, Iperf or just a speed test I cannot get a sigle stream past 7Mbits up and 10Mbs down.. They told me that I would not get 40Mbs throughput so my reply was well give me half, they responded with well sometimes you might not even get half. This is on a dedicated IPVPN/MPLS from site to site so if that is the case then why rate teh connection as DS3 how about pot luck or DS3 5-45Mbs...lol... thanks for the help folks it  is greatly appreciated.