OK here is a good one. I'v ebeen experiencing slow internet access form my site for a while now. One site in particular will not load....
www.good.com See the tecxt below of a sniffer trace (etherreal) any ideas? I've attached some troubleshooting that has happened already...We are looking for fresh ideas now.
OK, I've been looking into xxxx ths evening, and I found something that isn't right.
First, as summary of what you have in NJ.. As you may be aware, NJEDS has three (3) T1's for it's site. The three ones are aggregated into a PPP Multilink bundle, giving a total of ~4.5Mb/s. Currently, each connection is limited to 1.5Mb/s given how things are configured...more on that in a bit.
Here are the three interfaces:
NJEDS-ER-Site#
NJEDS-ER-Site#
NJEDS-ER-Site#show int ser0/0/0:0
Serial0/0/0:0 is up, line protocol is up
Hardware is GT96K Serial
Description: T1 to Paetec (CID# xx)
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 88/255, rxload 5/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 6d22h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 34000 bits/sec, 32 packets/sec
5 minute output rate 535000 bits/sec, 98 packets/sec
22848457 packets input, 2888792823 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
62672123 packets output, 2924484279 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Timeslot(s) Used:1-24, SCC: 0, Transmitter delay is 0 flags
NJEDS-ER-Site#
NJEDS-ER-Site#
NJEDS-ER-Site#
NJEDS-ER-Site#show int ser0/0/1:0
Serial0/0/1:0 is up, line protocol is up
Hardware is GT96K Serial
Description: T1 to Paetec (CID# xxxxx)
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 89/255, rxload 5/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 6d22h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 32000 bits/sec, 30 packets/sec
5 minute output rate 537000 bits/sec, 98 packets/sec
22880386 packets input, 2887525686 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
62690729 packets output, 2936854900 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags
NJEDS-ER-Site#
NJEDS-ER-Site#
NJEDS-ER-Site#
NJEDS-ER-Site#show int ser0/1/0:0
Serial0/1/0:0 is up, line protocol is up
Hardware is GT96K Serial
Description: T1 to Paetec (CID# xxxx..NJ)
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 88/255, rxload 5/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 6d22h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 32000 bits/sec, 30 packets/sec
5 minute output rate 535000 bits/sec, 97 packets/sec
22813342 packets input, 2862958091 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
107811 input errors, 82285 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
62693039 packets output, 2933002928 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Timeslot(s) Used:1-24, SCC: 0, Transmitter delay is 0 flags
NJEDS-ER-Site#
NJEDS-ER-Site#
Those three physical interfaces above are bundled into a single logical interface called a Multilink interface. Here is the Multilink info on it:
Multilink1, bundle name is Px
Endpoint discriminator is Pxx1
Bundle up for 11w6d, 90/255 load
Receive buffer limit 36000 bytes, frag timeout 1000 ms
0/0 fragments/bytes in reassembly list
41895 lost fragments, 6878661 reordered
3186/3540062 discarded fragments/bytes, 3186 lost received
0xAC53EB received sequence, 0x9D314F sent sequence
Member links: 3 active, 0 inactive (max not set, min not set)
Se0/1/0:0, since 11w6d
Se0/0/0:0, since 11w6d
Se0/0/1:0, since 9w1d
When you have a multilink path, reordering is normal and expected as packets can be sent by the other side of the link "out of order", at which point we have to hold onto the packet(s) in memory, and reorder them. But, to me, the number seems a tad bit high given the amount of time the link has been up, but I'm just guessing.
With all that said, here's what I think is happening, and what I need you to do:
1) Notice the errors on Serial0/1/0:0. A while back we were getting errors on one of the serial interfaces, and we called Paetec about it, but they said they were clean on their side. Testing on our side AT THAT POINT IN TIME showed no errors, so we couldn't continue forward on it. I kept an eye on it for a little bit, and it seemed fine... But, looks like the errors have returned, and I need you to work with Paetec and TelCo to get this resolved. The above output of the interfaces should show Paetec the errors, which is why I included them. That I'm aware of, the Circuit ID's (CID#) should be accurate, assuming the cables haven't been swaped. As I see it, this is Paetec to solve, so put the work on them (if possible), but they will need your assistance to let TelCo in, etc...
2) Because of the errors in Serial0/1/0:0, some packets are being dropped. These packets are causing retries, thus introducing additional delays. Do they account for the large amount of delays that Seth experienced?...I doubt it, but it's not helping... The good thing, and this is by design, the three T1's are in a PPP Multilink. As such, if we think the errors are really causing the issues, we can disconnect that T1, and the PPP Multilink will continue to function.
3) Once we get this T1 resolved, we can see about changing the way your PPP Multilink sends packets out. Right now, each session/stream of traffic is bound to a SINGLE T1. We can change it so that traffic is load balanced not by session, but rather, on a per packet basis. This essentially will fully maximize your total bandwidth. But, I don't want to do this until the Serial interface is fixed, and not taking on any greater percentage of errors over your other links. Enabling this before that would be bad as every 3rd packet could have a problem. But, in order for this to be fully useful, we need the ISP to do the same thing on their end. So, once we get this corrected, we'll need to have Paetec do the same. I'd highly suggest that you communicate our request to them sooner rather than later, and get from them in an e-mail that they can/will do it upon request. I say this because ISP's are VERY reluctant to do this as it increases the CPU on their routers, as each packet needs to be inspected by the CPU. Hence, they will often fight doing it...and for good reason (from their point of view). FYI, the command that does this is called "ip load-sharing per-packet", but they should know that. I'm mentioning it in case they are not familiar with it, and want to look it up in the Cisco documentation.
x
220 "9.149825" "10.14.66.0" "65.200.201.183" "TCP" "3461 > http [ACK] Seq=551 Ack=2720 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
221 "9.149881" "65.200.201.189" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
222 "9.149924" "65.200.201.189" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
223 "9.149934" "10.14.66.0" "65.200.201.189" "TCP" "3463 > http [ACK] Seq=263 Ack=1461 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
224 "9.150675" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
225 "9.150692" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
226 "9.150713" "10.14.66.0" "65.200.201.183" "TCP" "3461 > http [ACK] Seq=551 Ack=4180 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
227 "9.150791" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
228 "9.151453" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
229 "9.151478" "10.14.66.0" "65.200.201.183" "TCP" "3461 > http [ACK] Seq=551 Ack=5640 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
230 "9.151664" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
231 "9.152200" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
232 "9.152228" "10.14.66.0" "65.200.201.183" "TCP" "3461 > http [ACK] Seq=551 Ack=7100 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
233 "9.169350" "65.200.201.189" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
234 "9.169372" "65.200.201.189" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
235 "9.169395" "10.14.66.0" "65.200.201.189" "TCP" "3463 > http [ACK] Seq=263 Ack=2921 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
236 "9.169412" "65.200.201.189" "10.14.66.0" "HTTP" "HTTP/1.1 200 OK (application/x-javascript)
"
237 "9.209488" "65.200.201.189" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
238 "9.209523" "65.200.201.189" "10.14.66.0" "HTTP" "HTTP/1.1 200 OK (application/x-javascript)
"
239 "9.209542" "10.14.66.0" "65.200.201.189" "TCP" "3464 > http [ACK] Seq=269 Ack=1266 Win=64270 [TCP CHECKSUM INCORRECT] Len=0"
240 "9.229354" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
241 "9.229385" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
242 "9.229408" "10.14.66.0" "65.200.201.183" "TCP" "3461 > http [ACK] Seq=551 Ack=8560 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
243 "9.229467" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
244 "9.229480" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
245 "9.229490" "10.14.66.0" "65.200.201.183" "TCP" "3461 > http [ACK] Seq=551 Ack=10020 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
246 "9.265050" "10.14.66.0" "10.14.64.254" "DNS" "Standard query A us.bc.yahoo.com"
247 "9.270274" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
248 "9.270299" "65.200.201.183" "10.14.66.0" "TCP" "[TCP segment of a reassembled PDU]"
249 "9.270318" "10.14.66.0" "65.200.201.183" "TCP" "3461 > http [ACK] Seq=551 Ack=11480 Win=65535 [TCP CHECKSUM INCORRECT] Len=0"
