jmjames
asked on
OCS 2007 R2 A/V NAT - outside client trying internal IPs
Hello -
I have an existing OCS 2007 R2 install (single server internal, single Edge Server). It's been running for a while, and now I am trying to change its configuration. We have been using a NIC directly connected to the public network for A/V Edge, as R1 required, and I am changing it so that the A/V Edge is connected to the public Internet via NAT, behind our ISA server.
As required, we have our internal DNS server configured so that the A/V Edge machine pulls up the public IP address when it queries the A/V Edge FQDN. In addition, I went into the A/V Edge configuration, and told it to enable NAT.
Internal clients can use the A/V fine. However, external clients are doing something very strange. A packet capture on the external client machine show that it sets up the conversation with the usual chatter to the access and conferencing edge servers. Suddenly, instead of trying to access the A/V Edge's public IP, it tries accessing our base domain name's IP address via HTTPS. In other words, instead of looking up av.domain.com, it looks up domain.com. Then, after that, it does actually try the correct public IP address of the A/V Edge Server via port 3478 (TURN).
But here's where it gets weird. After a little more back-and-forth with the access and conferencing edge servers, it then tries the STUN conversation (over ports in the 50,000+ range) with the internal private IP address of our OCS server. Needless to say, these queries go nowhere, and the client is unable to get the audio/video.
Any suggestions?
Thanks!
J.Ja
I have an existing OCS 2007 R2 install (single server internal, single Edge Server). It's been running for a while, and now I am trying to change its configuration. We have been using a NIC directly connected to the public network for A/V Edge, as R1 required, and I am changing it so that the A/V Edge is connected to the public Internet via NAT, behind our ISA server.
As required, we have our internal DNS server configured so that the A/V Edge machine pulls up the public IP address when it queries the A/V Edge FQDN. In addition, I went into the A/V Edge configuration, and told it to enable NAT.
Internal clients can use the A/V fine. However, external clients are doing something very strange. A packet capture on the external client machine show that it sets up the conversation with the usual chatter to the access and conferencing edge servers. Suddenly, instead of trying to access the A/V Edge's public IP, it tries accessing our base domain name's IP address via HTTPS. In other words, instead of looking up av.domain.com, it looks up domain.com. Then, after that, it does actually try the correct public IP address of the A/V Edge Server via port 3478 (TURN).
But here's where it gets weird. After a little more back-and-forth with the access and conferencing edge servers, it then tries the STUN conversation (over ports in the 50,000+ range) with the internal private IP address of our OCS server. Needless to say, these queries go nowhere, and the client is unable to get the audio/video.
Any suggestions?
Thanks!
J.Ja
ASKER
I have checked the "Use NAT" button.
I have set up NAT (to the best of my knowledge) on ISA; I used the "publish non-Web server".
Users are connecting fine for everything except A/V.
I have set up NAT (to the best of my knowledge) on ISA; I used the "publish non-Web server".
Users are connecting fine for everything except A/V.
The problem tends to be with ISA is that it does not support SNAT.
One of the Caveats with using Private IP Addresses is that SNAT is a requirment, in that there is a 1 to 1 mapping between the Private and Public IPs; this can not be done with ISA.
ISA will use the 1st IP Address assigned to the External NIC Card for all outbound traffic which causes confusion for Communicator.
Adam
One of the Caveats with using Private IP Addresses is that SNAT is a requirment, in that there is a 1 to 1 mapping between the Private and Public IPs; this can not be done with ISA.
ISA will use the 1st IP Address assigned to the External NIC Card for all outbound traffic which causes confusion for Communicator.
Adam
ASKER
I don't think that the ISA server possibly not working with this would cause the client to try connecting to the private IP of the OCS server *in the LAN*. If it were trying to access the private IP of the A/V Edge server in the DMZ, I would be more likely to consider ISA the culprit, or if it were trying to access the ISA server's primary IP address, I would consider ISA to be the culprit. But that's not the behavior that I am seeing.
In addition, "Communicator" is *not* the client application. The client application is Live Meeting. That may make a difference.
J.Ja
In addition, "Communicator" is *not* the client application. The client application is Live Meeting. That may make a difference.
J.Ja
The Internal LAN address is offered as part of the SDP invite which is sent to the client.
It will be tried as part of the ICE process to locate a viable path to route the voice.
Are you able to share the wireshark trace from the client machine, the SIP Strack trace from the Front End and Edge Server would be useful as well.
Adam
It will be tried as part of the ICE process to locate a viable path to route the voice.
Are you able to share the wireshark trace from the client machine, the SIP Strack trace from the Front End and Edge Server would be useful as well.
Adam
ASKER
Ah, that makes a lot more sense then; it is unable to get through for whatever reason, and as a result it is failing over to the internal IP address. I can provide a Microsoft Network Monitor packet capture from the client machine. How do I get the SIP stack trace that you would like?
Thanks!
J.Ja
Thanks!
J.Ja
Due to OCS being encrypted you need to use the logging tools on OCS to capture the data received and send by the Front End Server, this allows you to see what is going on.
If you go to the Admin Tool and right click the pool and select Logging Tool then New Debug Session
Under Component select SIPStack and then under Level select Verbose and under Flags select All Flags.
Select Start Logging, try and get livemeeting to connect, and try and start audio and video sharing, one done select Stop Logging, then View Log Files, then View; save the document to somewhere were you can grab it.
If you do not want to post the traces here you can email them to adamgent@gmail.com
Adam
If you go to the Admin Tool and right click the pool and select Logging Tool then New Debug Session
Under Component select SIPStack and then under Level select Verbose and under Flags select All Flags.
Select Start Logging, try and get livemeeting to connect, and try and start audio and video sharing, one done select Stop Logging, then View Log Files, then View; save the document to somewhere were you can grab it.
If you do not want to post the traces here you can email them to adamgent@gmail.com
Adam
Sorry the last commnet was from me, was logged in under the work account.
Adam
Adam
ASKER
I just emailed it out, from jjames _at_ levitjames _dot_ com.
Thanks!
J.Ja
Thanks!
J.Ja
I have had a look at the SIP Stack trace, it looks like it will try the following addresses for a valid voice path:
192.168.32.69 (Front End)
72.XXX.XXX.102 (AV Edge)
192.168.3.39 (Client outside corp net??)
I have put my assumptions as to what they are in brackets, does that look correct.
Do you also have the network trace for the client?
Ideally the trace needs to be at the same time as the SIP Stack trace as it is easier to marry up details.
Adam
192.168.32.69 (Front End)
72.XXX.XXX.102 (AV Edge)
192.168.3.39 (Client outside corp net??)
I have put my assumptions as to what they are in brackets, does that look correct.
Do you also have the network trace for the client?
Ideally the trace needs to be at the same time as the SIP Stack trace as it is easier to marry up details.
Adam
ASKER
Yes, your assumptions are correct. I will rerun the stack SIP stack trace and get a packet capture from the client (Microsoft Network Monitor) at the same time, and re-email it to you.
J.Ja
J.Ja
Is the ISA Server direct on the Internet or is it behind a firewall or NATed router?
Also would it be possible for you to get a network trace from the ISA Server as well? A SIP Stack trace is not required.
Adam
Also would it be possible for you to get a network trace from the ISA Server as well? A SIP Stack trace is not required.
Adam
ASKER
It is direct on the Internet. I will get a packet capture later tonight.
ASKER
I just emailed it out to you. It is a 30+ MB attachment, let me know if you do not get it.
Thanks!
J.Ja
Thanks!
J.Ja
ASKER
Never mind, your email account rejected attachments that large. I emailed you a link to the file, please let me know when you receive it so I can remove the file from the server. In the capture, 192.168.34.56 is the inside the LAN device (it is on a VPN, by the by), and 24.199.XXX.XXX is the external client.
J.Ja
J.Ja
Hi,
I have the file.
It he inside edge of OCS direct on your internal line or is it NATed behind ISA?
Adam
I have the file.
It he inside edge of OCS direct on your internal line or is it NATed behind ISA?
Adam
Also is there a VPN connection between the Client and ISA.
If there is, have you tried testing this without the VPN?
Adam
If there is, have you tried testing this without the VPN?
Adam
ASKER
The ISA server is in a 3 leg configuration, so my DMZ (where the Edge server is), the LAN (where the OCS Front End is) and the two clients (one on the LAN via VPN, the other with the external address, not coming through the VPN) are all sending traffic through ISA. I have tried taking the machine that is usually on the VPN off of the VPN (and connecting remotely), and it still cannot connect to the A/V services. The firewall rules for the VPN are identical to the LAN, so anyone on the VPN is in the LAN for all intents and purposes.
J.Ja
J.Ja
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
The relationship between the internal network and the ISA server is a "Route" relationship, not a "NAT" relationship.
I too have not seen any of the STUN requests from the external clients. It is almost like they are getting lost before they reach ISA at all. Maybe I need to get an ISP involved?
172.16.32.99 is the DMZ address of the Edge Server, on the NIC that points "inside" the LAN and talks to the internal OCS pool.
J.Ja
I too have not seen any of the STUN requests from the external clients. It is almost like they are getting lost before they reach ISA at all. Maybe I need to get an ISP involved?
172.16.32.99 is the DMZ address of the Edge Server, on the NIC that points "inside" the LAN and talks to the internal OCS pool.
J.Ja
Is there anything in the ISA logs showing that it is being blocked?
It would be unusual for an ISP to block STUN traffic. The only other thing to try would be to capture the traffic before it hits ISA to see if it is even getting to you.
Adam
It would be unusual for an ISP to block STUN traffic. The only other thing to try would be to capture the traffic before it hits ISA to see if it is even getting to you.
Adam
ASKER
Nothing in ISA shows that it is being blocked. I agree that an ISP blocking STUN would be odd, especially since this client machine is the one I used to test the A/V service in the past. There is nothing between the ISA server and the ISP on that end. It is a FiOS line connected to a fairly dumb switch and the ISA machine (as well as a few other items that we want to be directly exposed to the Internet, like the original A/V Edge configuration) are also attached to that switch. No way for me to capture packets before they get to ISA, without talking to the ISP.
J.Ja
J.Ja
The only thing left (aside from an ISA issue) and it is something I have seen before is that the ARP mapping for the IP Address is still pointing directly to the MAC Address of the Edge Server.
Do you have a route onside that terminates the FIOS line? If you have access to it, it may be worth checking the arp mapping if you can; or ask your ISP to check it.
Although a reboot usually clears up the ARP table.
I can not see anything else that would do this....
Adam
Do you have a route onside that terminates the FIOS line? If you have access to it, it may be worth checking the arp mapping if you can; or ask your ISP to check it.
Although a reboot usually clears up the ARP table.
I can not see anything else that would do this....
Adam
ASKER
I considered the same thing for a moment. However, when i made these changes, the IP address for that AV Edge changed as well, so I do not think that it is a cached ARP entry either.
I'll see if I can get the ISP involved.
J.Ja
I'll see if I can get the ISP involved.
J.Ja
Silly question but did you update the External DNS entry for the new IP?
Adam
Adam
ASKER
Yup.
J.Ja
J.Ja
Can you ping the IP Address from another machine that you have direct on the Internet?
Even if ISA blocks it, it would show in the logs.
This would at least show if the issue lies with ISA or the net connection.
Adam
Even if ISA blocks it, it would show in the logs.
This would at least show if the issue lies with ISA or the net connection.
Adam
ASKER
Yup, Network Monitor shows ping going back & forth. Bizarre, isn't it?
J.Ja
J.Ja
It is sounding more like a general routing issue with the IP Address.
I guess speaking to the ISP is the only option really; although I would still be tempted to restart the router!
Adam
I guess speaking to the ISP is the only option really; although I would still be tempted to restart the router!
Adam
ASKER
Futher tests (using telnet from the external client machine) show that TCP/IP packets reach the ISA server just fine. Somehow, impossibly, UDP packets are being blocked somewhere but TCP isn't? My next test is to take another machine on our "cloud" switch (the switch that the FiOS connects to), assign it the same IP address, take that IP address off of the ISA server, and monitor on that other machine to see if the packets reach it.
J.Ja
J.Ja
Do the ISA rules cover allowing UDP through?
It would be unusual for UDP packets to be blocked somewhere....
It would be unusual for UDP packets to be blocked somewhere....
ASKER
Yes, they do, and I am checking the packet capture on the ISA server that will show me what is happening before ISA even has a chance to decide what to do with it.
J.Ja
J.Ja
ASKER
The plot thickens. I put the public IP address of a different device into the AV Edge's hosts file, restarted A/V Edge, and retested, this time putting the monitor on the other device. It too did not show any of the traffic. Clearly, the issue is upstream!
J.Ja
J.Ja
ASKER
My mistake, the traffic actually did reach the test machine, once I turned off the firewall.
J.Ja
J.Ja
ASKER
This issue is so odd, but at this point, it is clear that the problem is not in OCS. I'm marking the thread as answered, since you did effectively answer why my A/V isn't working. Sadly, the traffic I am seeing now is the same traffic I was seeing before, but it is good to get an external confirmation of the issue, and work through step-by-step what is happening. We've decided to leave it "as is" for now, and worry about it later.
Thanks for all of the help!
J.Ja
Thanks for all of the help!
J.Ja
ASKER
In this case, the analysis was spot-on correct, the traffic was never reaching the ISA server to be sent to the A/V Edge Server. We never did locate the root cause or resolve the problem, but at this point, the original question has been answered.
We seem to have the exact same problem.
Did you ever resolve this issue?
M.
Did you ever resolve this issue?
M.
ASKER
Nope, not at all, and not happy about it, either. We spent so much time on this issue it wasn't funny. It was easier to just leave it as-is.
J.Ja
J.Ja
A couple of questions.
Have you checked the box saying use NAT on the Edge Server?
Have you set-up one to one NAT on your firewall?
Are users able to connect in fine for IM etc?
Also there are known issues with using ISA for the Edge Server due to it having issues with SNAT.
Adam