Tuesday, August 7, 2007

Cisco ACE redundancy issues

Once you've configured redundancy on the ACEs, there's an active one, and a stand-by one. Ok, this is simple. However, there are some times configuration synch fails, and here's what I observed.

Once redundancy is configured this way :

ACTIVE ACE

ft group 1
peer 1
priority 200
peer priority 101
associate-context CONTEXT1
inservice

STANDBY ACE

ft group 1
peer 1
priority 101
peer priority 200
associate-context CONTEXT1
inservice


As usual, the active one has the highest priority. Now I want this redundancy to be HOT, i.e. sessions remain up during a switchover as they are mantained in sync by the peers.
Typing a show ft group det on the master ACE you could (as I did) see two types of redundancy :

Peer State : FSM_FT_STATE_STANDBY_HOT

or

Peer State : FSM_FT_STATE_STANDBY_COLD

Cold standby state means that sessions during the switchover will be dropped, and that, for some reason, configuration sync failed, so configurations are not even equal between the two peers, and further changes on the master will not be sent to the slave.

Typical reasons for configurations' sync to fail are :
  • A scripted probe needs its script file on the ACE's disk0:, the standby ACE may not have this file on his disk0:
  • Interfaces are not configured the same way (missing some interface vlan?)
  • Svcl groups on the Catalysts hosting the ACE may not pass the same vlans to the two peers.
However, if you made one of these mistakes, as I did, you have your standby ACE in COLD standby state, what to do now ?
Even copying manually the configuration on the second ACE, it will never switch by itself in HOT standby state.

The solution is quite easy :
  1. Solve all the issues that caused the configuration sync to fail (see above).
  2. On the standby ACE, switch off and then on (rapidly) the ft group of the context :

ACE-02/Admin#conf t
ACE-02/Admin(config)#ft group 1
ACE-02/Admin(config-ft-group)#no inservice
ACE-02/Admin(config-ft-group)#inservice

Now you will see the standby ACE erase all of its configuration and then start back to copy it from the master ACE. At the end, you should see on the master :

FT Group : 1
Configured Status : in-service
Maintenance mode : MAINT_MODE_OFF
My State : FSM_FT_STATE_ACTIVE
My Config Priority : 200
My Net Priority : 200
My Preempt : Enabled
Peer State : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority : 101
Peer Net Priority : 101
Peer Preempt : Enabled
Peer Id : 1
Last State Change time : Fri Aug 3 06:22:17 2007

Running cfg sync enabled : Enabled
Running cfg sync status : Running configuration sync has completed
Startup cfg sync enabled : Enabled
Startup cfg sync status : Startup configuration sync has completed
No. of Contexts : 1

Context Name : CONTEXT1
Context Id : 2

Note : During this process configuration is inhibited even on the master ACE.

PS : Thanks to Francesco for helping me in troubleshooting and summarizing the events of that night.

2 comments:

Paulo Lagoia said...

Hello!

I am network engineer from Brazil,
Ive just started work with cisco ACESM and at moment I am implementing redundancy resources.
I am looking forward your next post.

Good blog.

regards

Paulo

Pat D. said...

Thanks for the tip. I was having the same issue where the standby ace was in a COLD state. After logging into the admin and issuing an no inservice/inservice it went to hot.