Tuesday, August 12, 2008

Asymmetric Server Normalization on Cisco ACE

Summertime, I've some time to spend in the lab to try some new scenarios, optimization or try something read on the release notes.
This scenario is mostly a proof of concept as in our DataCenter many applications use server to server load balancing and it requires a too messy configuration to make it work with ASN.
From the Cisco doc :
Asymmetric Server Normalization (ASN) allows the ACE to load balance an initial request from the client to a real server; however, the server directly responds to the client bypassing the ACE. This behavior allows the acceleration of server to client communications and is transparent to the client. When the ACE operates in ASN, it does not perform any network translation when receiving packets destined to the VIP address. Traffic from the client hits the VIP address and the ACE uses the address as the destination address but rewrites the destination MAC address to the address of the real server.
So the match is speed vs. capabilities.
Since the ACE do not control the entire flow (it sees only the packets vs. the VIP) most of the advanced load balancing features aren't usable.
However, in some cases, where speed is the most important thing and the application flow is quite simple it could be a good choice.
Here's my implementation :

On the ACE :

access-list ACL01 line 8 extended permit ip any any

! The probe should check for the health status of the loopback interface
! on the real server.
probe icmp ICMP
ip address
interval 5

rserver host SERVER-10.1
ip address
rserver host SERVER-10.2
ip address

! The transparent command prevents the ACE to make a destination NAT,
! sending the packet as is to the real server.
serverfarm host LAB_20.1:80
probe ICMP
rserver SERVER-10.1 80
rserver SERVER-10.2 80

class-map match-all L4-LAB_20.1:80
2 match virtual-address tcp eq www

policy-map type loadbalance first-match L7-LAB_20.1:80
class class-default
serverfarm LAB_20.1:80
policy-map multi-match L4-POLICYMAPMULTI
class L4-LAB_20.1:80
loadbalance vip inservice
loadbalance policy L7-LAB_20.1:80
loadbalance vip icmp-reply active

service-policy input L4-POLICYMAPMULTI
access-group input ACL01

!The ACE is used as a bare load balancer, diasbling normalization (to make ASN work),
! and icmp-guard.
interface vlan 100
description SERVERSIDE
ip address
no normalization
no icmp-guard
no shutdown
interface vlan 150
description FIREWALLSIDE
ip address
no normalization
no icmp-guard
no shutdown

ip route

On the router :

ip route vrf LAB

This way everything works fine. Now it's time to make server to server load balancing work in this scenario.
What I wanted to work is a load balanced server that calls for its VIP being load balanced on itself or on another real server in the same area. Note that this scenario is different since here real server's are one-armed.

The choice to have a point-to-point vlan between the router and the ACE comes with the esigence to control traffic for some feature I needed described further. For simple ASN scenarios the ACE could easily have only one arm on the real servers' vlan (100).

Here's some observations I made :
1) Without any further configuration, server to server loadbalancing doesn't work as there's a loopback interface on the server with the VIP configured on it, you'll never get out of it.
2) You will need both source NAT and destination NAT to make it work.
3) You will have to make the ACE see all of the flow to make the NAT work.
4) You can't use ASN for server to server load balancing

I will not enumerate all of the tries I made to make it work (from which come the observations), here's my solution : double NAT (source and destination) on the router (6513 in this case) so that the ACE treats the connection as a normal external client request.

Here's the router's configuration :

interface Vlan100
ip vrf forwarding LAB
ip address
ip nat outside
interface Vlan150
ip vrf forwarding LAB
ip address
ip nat inside

ip nat inside source static
ip nat outside source static

ip route vrf LAB
ip route vrf LAB
ip route vrf LAB

The last route is a little tricky : as returning packets come from an outside zone to an inside one, packets need to be first routed and the natted. So you'll have to put a "fake" route versus an inside endpoint to make the packet then match the nat rule.

On the real servers you have a route to make all NATted request return by the ACE and not by the default gateway (router). These requests are easily identified by the source address (

Note that ASN and server to server load balancing with double NAT (source and destination) can be implemented both on the same box.

Thanks to Daniele and Marco for helping me in the lab.


Shafqat said...

Hey Alberto, Thanks for very informative blog. I found good info on ACE and ASN here. I have a question, if you can answer. We have a topology, where there is a routed Hop(an Alcatel router) between our ACE and the real servers. We want the Servers to do DSR, after the initial ACE loadblance to the real servers. I know that the Cisco ASN states that there cannot be a routed hop between ACE and the real server. My question is, is there a work around or a hack ?

Alberto said...

Hi Shafqat,
can you provide me a simple network diagram to better understand which scenario are you trying to implement ? I'll try to see if I can help you.


Charles said...

I tried the ASN config. I think I'm missing something. How does the ICMP probe work? The probe pings the loopback of the real server, but it's also a VIP. I can't ping it. Anyway, I have a "probe-failed" on status in the farm.

Alberto said...

Hi Charles,
have you tried the simpler thing, that is to set the probe to ping the real servers' production ip address and not the loopback ?

The VIP should pingable from client side because the ACE knows at least one rserver is up and running, and the response should come from the ACE.


Charles said...

Thanks for replying. You know I applied the l7 multi-match policy on the interface where clients come in. I see that you apply it globally. Is there potential to interfere with other vips? I assume this supercedes anything applied to interfaces. I have a lot of policies, so I need to be careful.

Kursat said...

How to enable windows server 2008 loopback interface to answer requests?

I achieve to work in linux but not windows

Subhadip Kumar said...

Can we configure VIP at same subnet where Real server placed in ASN mode of ACE

Alexis Cedeño said...
This comment has been removed by the author.
Alexis Cedeño said...

Alberto how are you, I am currently implementing ASN.

I have a fault condition, I would like to consult you.

I am migrating an application that has four servers.

Two servers (ASP16 and ASP17) are accessed through the virtual address tcp port 7777.

The other two servers (WSP1 and WSP2) are accessed through the virtual address tcp port 7780.

They set up four servers in an address alias to receive traffic from the load balancer. However, there is a specific query that is not working. The query is generated from the servers (ASP16 and ASP17) to the virtual address port 7780 tcp servers (WSP1 and WSP2).

Since there alias addresses configured on the servers the request is unable to leave the server. Do you know a similar case? Do you know any way to get traffic out from the servers to address on the load balancers?.

The operating system is now the servers are solaris 10.

The alias address configuration on the servers is:

ifconfig lo0: 1 plumb
ifconfig lo0: 1 netmask up

I've been reading and I think I can generate a static ARP entry with the address with the MAC address of the load balancer so that traffic can leave the server.

Let me know your opinion please.
A lot.