Solved

Packet Capture email around firewall

Posted on 2013-11-27
23
582 Views
Last Modified: 2014-01-25
I'm having trouble with a Juniper SRX240 firewall that is occasionally blocking email - I believe only on receive.

- email comes from an outside server
- It appears that the blocking can be client specific.  Some clients can get mail while others cannot.    I won't guarantee 100% that this is correct but I believe it is.
- Usually, email just works.  But, every few weeks we get this blocking behavior.  It lasts for a day or so and then, due to one thing or another (including my interventions) it heals itself.
- Rebooting the SRX doesn't seem to be a fix.
- The SRX is doing AV scanning on the email traffic and is supposed to be well-capable of handling that load.
- So far we have not been able to determine what's causing the blocking.
- Juniper Networks has not been able to instrument and see anything untoward during periods that blocking is occurring.

Here is my plan of attack now:

- I'm going to capture packets on both sides of the SRX.  This is up and running now using switch mirror ports.
- I'm going to set capture filters on the LAN side to focus on a failing client IP address AND the server public IP address.  This works fine.
- I would like to set up capture or display filters on the public side of the SRX but, with NAT involved, I'm not sure how to limit the capture to the particular client.  Obviously one can't see the LAN IP addresses on the public side.  I guess the port number would be an idea but I don't know if port numbers tend to change by email session or....?
- In the end I want to analyze the packets on both sides of the SRX to see what might be getting blocked.  So, having a minimum of packets would be good.
- I would like to merge the two capture files and sort by time of day.  Presumably this would be possible but I've not done it before.  Perhaps simply appending one file to the other and sorting, eh?

Any thoughts, comments, approaches would be appreciated.
0
Comment
Question by:Fred Marshall
  • 12
  • 10
23 Comments
 
LVL 61

Expert Comment

by:btan
ID: 39683387
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39684077
breadtan:

 Juniper support has done these kinds of things or given me CLI command sets to capture things and they have found nothing.  So I don't imagine that I'm going to learn to do a better job on this than they have done.  Of course, I could be wrong about this but I'm trying to prioritize how I use my time.  Learning is always of value but I need results right now.

But, I can reasonably use Wireshark and I do have switch ports on both sides of the SRX that can be mirrored and monitored on a workstation.   I'm already getting somewhat interesting results that way.  I figure that I can show Juniper support folks the results of blocked transactions once I can reasonlby *see* them.

I can filter the capture on the LAN side to include a failing client IP AND the public email server IP.  So, this narrows down the capture pretty well.

I can filter the capture on the public / untrust side to include the public email server IP.
And I can play around with port numbers to try to segregate various client packets.
That's about where I am now and thus this question....
I'd like to improve the filtering on the public / untrust side in the Wireshark switch port capture.
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39684296
It occurs to me that perhaps I could get something out of the SRX:
The internal IP address / port addition or "socket" is internal to the SRX.
So, how might one extract those lineups as a function of time from the SRX?
That might be useful in interpreting the Untrust side packet capture.
0
 
LVL 61

Expert Comment

by:btan
ID: 39684388
Https and SMTP are for the mail server if that is you running one internally. Pretty tough if https is not terminated and open up for inspection. You cannot inspect what you don't see. Would it be even be easier to disable all policy and try out traffic follow...not sure if there is staging to ease thing a little.
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 39685383
Does the SRX240 log the blocking of emails?  Packet capture tools may not see anything unusual if the firewall simply drops traffic.
0
 
LVL 61

Expert Comment

by:btan
ID: 39685483
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39686221
breadtran: The antispam feature is turned off.

eeRoot: If there were intentional blocking then I'd expect the Juniper folks to have seen that.  They didn't.  The packet capture tools would see the packets being dropped presumably.  That's really what I'm trying to do at the ports and the reason for the original question.

I should think I could merge the inside/outside packet capture files and see:
- a pair of outgoing packets
somewhat matched to
- a pair of incoming packets.
If any expected pairs are NOT pairs but, rather, a single incoming at Untrust  that's not accompanied by a single outgoing at Trust or vice versa then that would be an interesting indicator of what's going on: an indicator of packets dropped in the SRX.
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39686222
Does anyone know how to get the IP/port list?
0
 
LVL 61

Expert Comment

by:btan
ID: 39686287
Was sketching to see we pipe traffic log in syslog for the srx to remote syslog server
 
set system syslog host <IP-Address> any any
set system syslog host <IP-Address> match "RT_FLOW_SESSION"
 
in order to test, before sending to remote server first check it locally through
 
set system syslog file traffic-logs any any
set system syslog file traffic-logs match "RT_FLOW_SESSION"
0
 
LVL 61

Expert Comment

by:btan
ID: 39686293
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39687093
My problem is that I'm conservative when messing around with a device that's in production.  I really don't want to mess up what's already working!!

When I start setting these things up inside a Juniper box, I'm always concerned about the files becoming too large or the cpu load being too much.  I've seen the warnings.  But, I have NO practical experience really in knowing what numbers of milliseconds / seconds / hours make sense.  Any guidance?  This goes for logs, debugs, etc.
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 61

Expert Comment

by:btan
ID: 39687522
If you worried on cpu in production, there should always be staging environment prior to launching any policy in production. Just that you need tp replicate that traffic dropped,  I know it is not easy. Hence support shd advice and the in term of log for high cpu of the additional logging to remote syslog server...you may consider grabbing the cureent cpu log and ask for support assessment, personally the cpu ranges 60% to 80% as per norm and dependent of environment.  There are other log in the overall link too.

http://kb.juniper.net/InfoCenter/index?page=content&id=KB21781#HighCPU

The concern is why support sya they are seeing nothing and didn't any policy trigger as you shared with traffic dropping. Something dropped packet may be default if traffic throughput exceeded that interface throughput,  some silently dropped. Also logging session based help to see if there are anomalous behaviour,  support should advice e.g. filter for RT_FLOW_SESSION as that will capture both session-init and session-close.

http://forums.juniper.net/t5/SRX-Services-Gateway/SRX-Logging/td-p/20233
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39688500
there should always be staging environment prior to launching any policy in production
That would be great but clearly we are from "different worlds".  No such luxury here....
An how does one create the real traffic in a staging environment?

I started this question with the idea that I would treat the SRX as a "black box".  We are now getting into SRX-based "solutions" which I had intended to avoid.  And now, we are only digging that hole deeper.  

If I have received the answer:
Does anyone know how to get the IP/port list?
I have not either seen, or perhaps, understood the answer.
I should think it would be simple and understandable even though in the SRX.
I don't want to go beyond that in the SRX.
0
 
LVL 61

Expert Comment

by:btan
ID: 39688839
For staging it deoend hiw you define production traffic. Cloud based apps cam staged in such that UAT is done by user group delegated. Agree, the exact cannot be replicated in the traffic but minimally whether changes break the system can be validated offline before putting it to the production.

Noted that and if looking from wireshark capture, probably from pcap filter we can specific the range of known ports (e.g. tcp.port >= 1 && tcp.port <= 1024) or viewing Statistics > Endpoints and looking at TCP and UDP gives me a list of ports used.
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39689140
Regarding the ports, I've at least been able to go that far.  I was wondering about something a little different / better.
0
 
LVL 61

Expert Comment

by:btan
ID: 39689191
Unless there is specific keywords I don't think we can extend as far the filter since it also is based on traffic pattern string. Hence which is why it is best that the box can have some log remotely. Also will need to make sure we tapping at span port
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39761277
I'm back to where I started.  I want to use Wireshark on the ports.
0
 
LVL 61

Expert Comment

by:btan
ID: 39761987
Probably some note to be aware using wireshark then...

NetDMA and TCP Chimney Offload. Both technologies offload TCP processing to the NIC thereby bypassing the WinPCAP driver. Instructions for disabling: http://support.microsoft.com/kb/951037#LetMeFixItMyselfAlways


CaptureSetup/InterferingSoftware - Software that's working together with the network protocol stack at a low level can cause problems together with WinPcap
http://wiki.wireshark.org/CaptureSetup/InterferingSoftware

I was thinking simplistic in using a Color Rule (under View -> Colorize Conversation -> New Coloring Rule) and add ip.src == your.ip as a color and ip.dst == your.ip as another color. Just to override the defaults and help in making the comparison.
0
 
LVL 61

Expert Comment

by:btan
ID: 39762009
Saw some useful commands
http://blog.spiderlabs.com/2012/12/pcap-files-are-great-arnt-they.html
3) Show conversations and destination ports using tshark

This is useful to see what ports are open on hosts without scanning, and
see who is talking to it.

tshark -o column.format:'"Source", "%s", "Destination",
"%d", "dstport", "%uD"' -r file.pcap |sort|uniq
6) Find emails using ngrep

As an example of ngrep's regular expression power, you can make up your regex, and then use grep to display it in a nice format. In this example, I'm searching anything that looks like an email address and output the result

ngrep -q -I file.pcap '[a-zA-Z0-9.]+\.?@[a-zA-Z0-9.]+\.[a-zA-Z0-9]+' |grep -Eo '[a-zA-Z0-9.]+\.?@[a-zA-Z0-9.]+\.[a-zA-Z0-9]+'|sort|uniq
8) Show all TCP streams using tshark

This is useful to show all the TCP sessions on the network

tshark -r file.pcap -q -z conv,tcp
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39763176
Well, I can probably use such things but I'm not such an expert on contructing them.

My thought was to do this sort of thing:

1) Determine a client which is having the trouble.
2) Capture Trust and Untrust side traffic.
3) on the Trust side, the capture can be filtered for the client IP.
4) Find the port number being used on the public side to address a private IP on the LAN side.
5) Filter the Untrust side for that port.
6 Merge the filtered files in order to look for dropped traffic sequences.
0
 
LVL 61

Accepted Solution

by:
btan earned 500 total points
ID: 39763845
Maybe to list out the traffic for that machine IP and from the capture see the available port used then. maybe mapped the port to the machine netstat -ano (or process hacker tool) process pid to see the application interfacing using that port. Isolate the pcap on the interested traffic from that port and compare - eyeball may be easier if traffic is not too huge, probably has to run short moment to see if make sense worth
0
 
LVL 25

Author Comment

by:Fred Marshall
ID: 39780346
Yes, that's exactly the idea....
0
 
LVL 61

Expert Comment

by:btan
ID: 39781019
Hmm..let hear if better option but looks like the only manual means out and tedious to eyeball. If it can be contain in test environment will be better or separate oit a vlan for that
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Suggested Solutions

If your business is like most, chances are you still need to maintain a fax infrastructure for your staff. It’s hard to believe that a communication technology that was thriving in the mid-80s could still be an essential part of your team’s modern I…
If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now