Client doesn't respond to SYN-ACK

We have been seeing an issue where the tcp handshake is not being established properly that we are losing data.  Using wireshark to monitor our network, we have seen the client sending SYN and our server will send SYN, ACK, however the client doesn't seem to be responding to the SYN, ACK.  We only have one client connecting at a time since our server it's still being tested and it happens intermittently but one packet loss is critical.

Time                                          Source     Destination      Info
2009-04-28 10:23:00.352914    Client        Server           50883 > gap [SYN] Seq=0 Win=32768 Len=0 MSS=1460 WS=0 TSV=3505842492 TSER=0

2009-04-28 10:23:00.352942   Server       Client             gap > 50883 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=0 TSER=0

2009-04-28 10:23:02.532878   Server       Client            gap > 50883 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=0 TSER=0

2009-04-28 10:23:08.568226   Server       Client           gap > 50883 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=0 TSER=0

2009-04-28 10:23:09.752464   Client         Server         50883 > gap [SYN] Seq=0 Win=32768 Len=0 MSS=1460

2009-04-28 10:23:12.952809   Client         Server         50883 > gap [SYN] Seq=0 Win=32768 Len=0 MSS=1460

2009-04-28 10:23:16.152570   Client         Server         50883 > gap [SYN] Seq=0 Win=32768 Len=0 MSS=1460

2009-04-28 10:23:22.353056   Client         Server         50883 > gap [SYN] Seq=0 Win=32768 Len=0 MSS=1460

2009-04-28 10:23:22.353086   Server      Client           [TCP Previous segment lost] gap > 50883 [SYN, ACK] Seq=5733720 Ack=1 Win=16384 Len=0 MSS=1460

2009-04-28 10:23:24.662605   Server      Client          gap > 50883 [SYN, ACK] Seq=5733720 Ack=1 Win=16384 Len=0 MSS=1460

2009-04-28 10:23:30.697935   Server      Client         gap > 50883 [SYN, ACK] Seq=5733720 Ack=1 Win=16384 Len=0 MSS=1460