Monday, June 11, 2007

Dual armed server to server load balancing con Cisco ACE

Let's say you have a large data center, let's say that in this data center you have lots of dual-armed load balanced serverfarms. It could happen that these servers need to call each other's balanced services. Here's how this could be accomplished with very light configuration on real servers.

Scenario :



The BLUE-SERVERFARM real servers needs to query a web service located on the YELLOW-SERVERFARM, on tcp port 2000.
All of the real servers use the "upper" interface (vlan 101) to act as servers, i.e. to answer clients' queries coming from the ACE.
The "internal" interface (vlan 102) is used by the servers when they act as client of someone else's service.
Easy to configure this, matter of routes on the servers. The default gateway is always the ACE, there's a static route on the internal interface for all the ips the server could query acting as a client.


Without configuring Source NAT con the ACE, all connections fail, because of asymmetric response from servers of the YELLOW-SERVERFARM.
When a connection arrives from the ACE, the source ip is the internal interface of the client server. As this ip is on a lan directly connected on the destination server, the response will return over the INTERNAL, not over the same route of the request.



Solution:
Source natting this requests on the ACE will cause the destination server not to know as directly connected the source ip, answering on the default gateway (ACE) and so following the same path of the request.
The simplest way I've found is to reserve a new virtual address only for requests coming from the servers on the same lan, as described above. So clients will continue query the service on the VIP 10.20.0.2 port 2000, while servers on the same lan will query the same service on the same port but on VIP 10.20.0.20, being Source-NATted with an IP from the SNATPOOL.



The real server of the YELLOW-SERVERFARM responding to the request, seeing it from a SNATted address will route the response via the default gateway (ACE) which will send back packets on the same path of the request.

probe tcp 2000-TCP
port 2000
interval 10
passdetect interval 5
passdetect count 1

probe icmp ICMP
interval 10
passdetect interval 5
passdetect count 1

rserver host SERVER-10.1
ip address 10.0.10.1
inservice

rserver host SERVER-10.2
ip address 10.0.10.2
inservice


rserver host SERVER-10.3
ip address 10.0.10.3
inservice

rserver host SERVER-10.4
ip address 10.0.10.4
inservice


serverfarm host BLUE-SERVERFARM_20.1:80
failaction purge
probe ICMP
rserver
SERVER-10.1 80
inservice
rserver SERVER-10.2 80
inservice

serverfarm host YELLOW-SERVERFARM_20.2:2000
failaction purge
probe 2000-TCP
rserver
SERVER-10.3 2000
inservice
rserver SERVER-10.4 2000
inservice


class-map match-all L4-MAP-BLUE-SERVERFARM_20.1:80
2 match virtual-address 10.0.20.1 tcp eq www

class-map match-all L4-MAP-YELLOW-SERVERFARM_20.2:2000
2 match virtual-address 10.0.20.2 tcp eq 2000

class-map match-all L4-SNAT-MAP-YELLOW-SERVERFARM_20.20:2000
2 match virtual-address 10.0.20.20 tcp eq 2000

policy-map type loadbalance first-match L7-BLUE-SERVERFARM_20.1:80
class class-default
serverfarm
BLUE-SERVERFARM_20.1:80

policy-map type loadbalance first-match L7-YELLOW-SERVERFARM_20.2:2000
class class-default
serverfarm
YELLOW-SERVERFARM_20.2:2000

policy-map multi-match L4-POLICYMAPMULTI
class L4-MAP-BLUE-SERVERFARM_20.1:80
loadbalance vip inservice
loadbalance policy
L7-BLUE-SERVERFARM_20.1:80
class L4-MAP-YELLOW-SERVERFARM_20.2:2000
loadbalance vip inservice
loadbalance policy
L7-YELLOW-SERVERFARM_20.2:2000
class L4-MAP-YELLOW-SERVERFARM_20.20:2000
loadbalance vip inservice
loadbalance policy
L7-YELLOW-SERVERFARM_20.2:2000
nat dynamic 1 vlan 101

interface vlan 101
description SERVERSIDE
ip address 10.0.10.199 255.255.255.0
nat-pool 1 10.0.21.1 10.0.21.254 netmask 255.255.255.0
no shutdown
interface vlan 151
description FIREWALLSIDE
ip address 10.0.0.2 255.255.255.240
service-policy input L4-POLICYMAPMULTI
no shutdown



2 comments:

Stuart said...

Is there any way to do this on a single armed server, where the default gateway of the server is the ACE.

I.e. An internal web server accessing the vip on the outside.

Thanks
Stu

Al said...

yes, you should source-nat all of the traffic from the servers that goes through vips :

I have to look for an old lab configuration I made however it should work creating a class map that matches both source address (servers) and destination address (vips).

This class map should be put on top of the policy map multi-match and should source nat with a pool defined in the interfaces configuration.

Starting from this article's example should work this way :

class-map match-all L4-SNAT-ONEARM
2 match source-address 10.0.10.0 255.255.255.0
3 match destination-address 10.0.20.0 255.255.255.0

policy-map multi-match L4-POLICYMAPMULTI
class L4-SNAT-ONEARM
nat dynamic 1 vlan 101

I'll try to get that configuration out of my old labs and write a more complete article on this.

Hope it helpes