Cisco PIX 515e dropping IPSEC tunnels to ASA 5505 over time

We have a Head-Office/Branch-Office WAN like this,

Server LAN <-> Cisco PIX 515e <-VPN tunnel-> Cisco ASA 5505 <-> Client LAN 1
                              <-VPN tunnel-> Cisco ASA 5505 <-> Client LAN 2
                              <-VPN tunnel-> Cisco ASA 5505 <-> Client LAN 3
                               ...  
                              <-VPN tunnel-> Cisco ASA 5505 <-> Client LAN 66

Problem:
5% of these VPN tunnels degrade over time.

Symptoms:

• Clients respond to PING, but not to RPC or RDP.
• On the ASA, VPN tunnels goes from 1 x IKE, 2 x IPSec down to 1 x IKE, 1 x IPSec.
• A restart of the ASA resolves the problem temporarily.

This PIX has been unreliable, and will probably be replaced with a more modern bit of gear. Although usually under 10%, the CPU on the PIX periodically hits 80-90% with traffic spikes, but I can't say I've been able to correlate dropped tunnels with these loads.

I have a few specific questions, but am grateful for any and all insights.

1. Can I monitor (via SNMP) the total IPSec tunnels on the PIX? This should always be (at least?) twice the number of branch offices, and (at least?) twice the total IKE - if it drops then I probably have a problem.

2. Is there an event I can alarm on in the PIX's own logging, when one of these tunnels is dropped? Maybe,

   snmp-server enable traps ipsec start stop
   

3. Is there anything I can do to keep this tunnel alive, until the PIX can be replaced? I was thinking of scriptable keep-alive traffic, PING doesn't seem to cut it. I am also looking at idle time-out values, maybe re-keying intervals, any other ideas?

---

PIX515E# show run isakmp
crypto isakmp identity address
crypto isakmp enable outside
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
crypto isakmp nat-traversal  20


PIX515E# show run ipsec
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac


PIX515E# show version

Cisco PIX Security Appliance Software Version 7.2(4)
Device Manager Version 5.2(4)

From serverfault nray

• I am seeing the same thing myself. I just setup a PIX515 running 8.04 IPSEC to my ASA5510 8.2. It works great and then the tunnel just dumps everyone. During this time, the internet keeps going just fine. So, it's just the tunnel that is having problems.

  From $pageUsers[$post.author].name

• 1) You absolutely can monitor the number of IPSec tunnels, but we’ve found that not to be a truly reliable way of determining if connectivity is working. It’s always best to send and receive traffic via the tunnel to confirm connectivity (e.g. ping monitor).

  2) Same as #1 – it can be done, but may not give you usable information. Tunnels will start and stop in the normal course of operation depending on timeout intervals.

  3) While it’s not supposed to be necessary, we have seen improvement with tunnel connectivity in some situations by running a ping at frequent intervals (3-5 minutes). Hard to say whether that would help in this situation without in-depth analysis.

  Generally speaking, issues like this occur frequently due to VPN config mismatches between the head end and remote end VPN peers. Differing ACLs are often a problem.

  From Jim B

• Do the tunnel ever come back up by itself or do you manually intervene to get the tunnel back up?

  What is the lifetime set on the ASA?

  And do you have keepalives enabled/disabled on both devices?

  I've seen this issue before between a Cisco 6500 running IOS and an ASA where the IOS is happy to run without an SA (if it expires for whatever reason) where a ASA is not and the tunnel dies for a random period of time until it renegotiates and the tunnel comes back up until the SA expires again.

  Ewan Leith : I've seen the same kind of thing as Conor, it certainly sounds like an SA timeout is different somewhere along the line
  From Conor