Juniper VPN tunnels flapping up and down after firmware upgrade

Experts,

We recently upgraded our branch office SRX650  firmware from 10.2.3 to 10.4.3 a couple of weeks ago.  Afterwards we've seen VPN tunnels flap up and down on LAN side.  During which the WAN side never goes down.  Funny thing is when you login into one of the SRX220 you do show interfaces terse the LAN side shows up/up.

We have site to site VPN tunnels from our branch SRX650s to field SRX220s.  

I've tried rebooting the SRX220's, lowering tcp-mss and ipsec-vpn from 1300 to 1360 but to no avail.  My SRX650 CPU utilization is low.  The monitoring software I'm using to monitor the tunnels test time has been upped to test less frequent to no avail.

We have plenty of other VPN tunnels on the SRX650 that aren't flapping. I'm kind of lost and need some help.
Greggor25Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

dpk_walCommented:
Can you upgrade SRX650 to the recommended version of 11.4R10.3 and see if you see the issue again.

http://kb.juniper.net/KB21476

Please update and update.

Thank you.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Greggor25Author Commented:
I have to get a maintenance window for outage to push firmware update.

Last time I copied the firmware to both nodes via USB stick and rebooted them simultaneously to not break the cluster.

Is this the recommended upgrade procedure?
QuoriCommented:
You can do an in-service upgrade which has minimal impact on live traffic.

Follow this:

http://kb.juniper.net/InfoCenter/index?page=content&id=KB17947
Get Blueprints for Increased Customer Retention

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

dpk_walCommented:
As per Juniper documentation In service Software Upgrade or ISSU is not supported on branch SRX; but there are few customers who have reported to have successfully done ISSU on branch SRX boxes.

http://forums.juniper.net/t5/SRX-Services-Gateway/For-branch-SRX-running-HA-will-ISSU-be-supported-Perhaps-in-the/td-p/180885

Please follow the KB posted by expert Quori; also look at above link for more information.

Please implement and update.

Thank you.
Greggor25Author Commented:
The In service upgrade and documentation provided is very good. How long of maintenance window would I need to complete the upgrade?

Hour or two hours?
dpk_walCommented:
Take max of two hours to be on safer side; ideally both the boxes would get upgraded and operational in max 40 minutes.

Thank you.
Greggor25Author Commented:
I will perform the upgrade in two weeks and let you know the result. In the mean time I have a ticket open with JTAC to see if they have any answers.
dpk_walCommented:
Sure; please update us with your findings.
Greggor25Author Commented:
Updated the SRX650 cluster to Juniper recommended 11.4R10.3.  Appears to have fixed the issue.  

I have noticed after updating the JWeb interface pops policies and configuration a lot faster.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Security

From novice to tech pro — start learning today.