Wednesday, December 12, 2007

Source NAT on Cisco ACE

I've already described the scenario for this issue here.


My first approach on Source NAT for this scenario was a little too CSS-minded, so wasting vip addresses and not considering some new features on the ACE.

For each serverfarm which had to be reached from the same subnet I created two vips, one for everybody, one for SNATted connections from the same subnet.

Now I changed approach, SNATting everything coming from the INTERNAL subnet, no matter which vip is looking for. This way I have not to use different vips depending on source address.




class-map match-all L4-MAP-SNAT-INTERNAL
2 match source address 10.0.11.0 255.255.255.0


class-map match-all L4-MAP-YELLOW-SERVERFARM_20.2:80
2 match virtual-address 10.0.20.2 tcp eq www


policy-map type loadbalance first-match L7-FARM_20.2:80
class class-default
serverfarm
YELLOW-SERVERFARM_20.2:80

policy-map multi-match L4-POLICYMAPMULTI-LOADBALANCE
class L4-MAP-SNAT-INTERNAL
nat dynamic 100 vlan 101

class L4-MAP-FARM_20.1:21
loadbalance vip inservice
loadbalance policy
L7-FARM_20.2:80
loadbalance vip icmp-reply active

service-policy input L4-POLICYMAPMULTI-LOADBALANCE

interface vlan 101
description SERVERSIDE
ip address 10.0.10.199 255.255.255.0
nat-pool 100 10.0.21.100 10.0.21.115 netmask 255.255.255.0 pat

no shutdown

interface vlan 151
description FIREWALLSIDE
ip address 10.0.0.2 255.255.255.240
no shutdown


In this scenario each connection coming from any server on the 10.0.11.x subnet is subnetted. Then, going down on the multimatch policy, it reaches the vip class which loadbalance on the real servers.
Note that the L4-MAP-SNAT-INTERNAL must be the first class defined in the policy for have things work well.

Tuesday, September 11, 2007

Tcl tcp check script

Here's my script I use for TCP port checking on Cisco ACE.

The probe is simple :

probe scripted 8080-TCP-SCRIPT
port 8080
script tcp_check.tcl


and here's the script :


#Procedures
proc
opensocket {ip port}
{
set sock [socket $ip $port] }

# Get the IP address of the real server from a predefined global array

set
ip $scriptprobe_env(realIP)


# Get port from command line arguments (may be different from real service's one)
if
{$argc <> 1 )
{ set
port $scriptprobe_env(realPort)
}
else

{ set port [lindex $argv 0] }

# Open a socket to the server. This creates a TCP connection to the
# real server
set
EXIT_MSG "opening socket..."

if
{ [catch { opensocket $ip $port } sock] }
{

set EXIT_MSG "$ip:$port ERR : $sock"

set expCode1 "timeout"

set expCode2 "connection timed out"

set expCode3 "connection refused"

set expCode4 "Unknown host"

set expCode5 "o route to host"

set expCode6 "nable to connect"

set expCode7 "host is unreachable"

if { [ regexp $expCode1 $sock ] }
{
set EXIT_MSG "probe fail. $ip:$port ERR : <$expCode1>"
exit 30002
}

if { [ regexp $expCode2 $sock ] }
{
set EXIT_MSG "probe fail. $ip:$port ERR : <$expCode2>"
exit 30002
}

if { [ regexp $expCode3 $sock ] }
{
set EXIT_MSG "probe fail. $ip:$port ERR : <$expCode3>"
exit 30002
}

if { [ regexp $expCode4 $sock ] }
{
set EXIT_MSG "probe fail. $ip:$port ERR : <$expCode4>"
exit 30002
}

if { [ regexp $expCode5 $sock ] }
{
set EXIT_MSG "probe fail. $ip:$port ERR : <$expCode5>"
exit 30002
}

if { [ regexp $expCode6 $sock ] }
{
set EXIT_MSG "probe fail. $ip:$port ERR : <$expCode6>"
exit 30002
}

if { [ regexp $expCode7 $sock ] }
{
set EXIT_MSG "probe fail. $ip:$port ERR : <$expCode7>"
exit 30002
}

} else {

set EXIT_MSG "probe success"

close $sock

exit 30001

}

Thursday, August 9, 2007

C# IP utilities : Decimal to binary notation

public string IPdec2binary(string ip)
{

string ret = "";
try
{
string[] theIP = ip.Split('.');

long na = Convert.ToInt64(theIP[0]);

long nb = Convert.ToInt64(theIP[1]);

long nc = Convert.ToInt64(theIP[2]);

long nd = Convert.ToInt64(theIP[3]);

string sa = Convert.ToString(na, 2);

string sb = Convert.ToString(nb, 2);

string sc = Convert.ToString(nc, 2);

string sd = Convert.ToString(nd, 2);


long a = Convert.ToInt64(sa);

long b = Convert.ToInt64(sb);

long c = Convert.ToInt64(sc);

long d = Convert.ToInt64(sd);



ret = Convert.ToString("" + a).PadLeft(8, '0') + ".";

ret += Convert.ToString("" + b).PadLeft(8, '0') + ".";

ret += Convert.ToString("" + c).PadLeft(8, '0') + ".";

ret += Convert.ToString("" + d).PadLeft(8, '0');
}
catch
{
return "";
}

return ret;
}

Tuesday, August 7, 2007

C# IP utilities : Integer to dotted notation


public string IPint2dotted(long ip)
   {
     string ret = "";
     try
     {
       string hex = ip.ToString("X");

       if (hex.Length>=8)
         hex = hex.Substring(hex.Length-8);
       else
         hex = hex.PadLeft(8,'0');

       for (int i=hex.Length;i>0;i-=2)
       {
         ret += long.Parse(Convert.ToString(hex[i-2].ToString() + hex[i-1].ToString()), System.Globalization.NumberStyles.HexNumber).ToString()+".";
       }

       if (ret.Length>0)
return ret.Substring(0,ret.Length-1);
else
return "";
}
catch (Exception exc)
{
return "";
}

}

Cisco ACE redundancy issues

Once you've configured redundancy on the ACEs, there's an active one, and a stand-by one. Ok, this is simple. However, there are some times configuration synch fails, and here's what I observed.

Once redundancy is configured this way :

ACTIVE ACE

ft group 1
peer 1
priority 200
peer priority 101
associate-context CONTEXT1
inservice

STANDBY ACE

ft group 1
peer 1
priority 101
peer priority 200
associate-context CONTEXT1
inservice


As usual, the active one has the highest priority. Now I want this redundancy to be HOT, i.e. sessions remain up during a switchover as they are mantained in sync by the peers.
Typing a show ft group det on the master ACE you could (as I did) see two types of redundancy :

Peer State : FSM_FT_STATE_STANDBY_HOT

or

Peer State : FSM_FT_STATE_STANDBY_COLD

Cold standby state means that sessions during the switchover will be dropped, and that, for some reason, configuration sync failed, so configurations are not even equal between the two peers, and further changes on the master will not be sent to the slave.

Typical reasons for configurations' sync to fail are :
  • A scripted probe needs its script file on the ACE's disk0:, the standby ACE may not have this file on his disk0:
  • Interfaces are not configured the same way (missing some interface vlan?)
  • Svcl groups on the Catalysts hosting the ACE may not pass the same vlans to the two peers.
However, if you made one of these mistakes, as I did, you have your standby ACE in COLD standby state, what to do now ?
Even copying manually the configuration on the second ACE, it will never switch by itself in HOT standby state.

The solution is quite easy :
  1. Solve all the issues that caused the configuration sync to fail (see above).
  2. On the standby ACE, switch off and then on (rapidly) the ft group of the context :

ACE-02/Admin#conf t
ACE-02/Admin(config)#ft group 1
ACE-02/Admin(config-ft-group)#no inservice
ACE-02/Admin(config-ft-group)#inservice

Now you will see the standby ACE erase all of its configuration and then start back to copy it from the master ACE. At the end, you should see on the master :

FT Group : 1
Configured Status : in-service
Maintenance mode : MAINT_MODE_OFF
My State : FSM_FT_STATE_ACTIVE
My Config Priority : 200
My Net Priority : 200
My Preempt : Enabled
Peer State : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority : 101
Peer Net Priority : 101
Peer Preempt : Enabled
Peer Id : 1
Last State Change time : Fri Aug 3 06:22:17 2007

Running cfg sync enabled : Enabled
Running cfg sync status : Running configuration sync has completed
Startup cfg sync enabled : Enabled
Startup cfg sync status : Startup configuration sync has completed
No. of Contexts : 1

Context Name : CONTEXT1
Context Id : 2

Note : During this process configuration is inhibited even on the master ACE.

PS : Thanks to Francesco for helping me in troubleshooting and summarizing the events of that night.

Tuesday, July 10, 2007

Static NAT on Cisco ACE

This one's simple. Here's how I managed my first NAT to make a load balanced dual armed farm exit through the ACE towards the Internet. I had to make all of the traffic NATted on a single ip in order to make firewalls' life easy.

Scenario :


Then the configuration :

rserver host SERVER-10.1
ip address 10.0.10.1
inservice

rserver host SERVER-10.2
ip address 10.0.10.2
inservice


serverfarm host FARM_20.1:80
failaction purge
rserver
SERVER-10.1 21
inservice
rserver SERVER-10.2 21
inservice


class-map match-any L4-MAP-NAT
2 match source address 10.0.10.1 255.255.255.255
3 match source address 10.0.10.2 255.255.255.255

class-map match-all L4-MAP-FARM_20.1:21
2 match virtual-address 10.0.20.1 tcp eq www


policy-map type loadbalance first-match L7-FARM_20.1:80
class class-default
serverfarm
FARM_20.1:80

policy-map multi-match L4-POLICYMAPMULTI-LOADBALANCE
class L4-MAP-FARM_20.1:21
loadbalance vip inservice
loadbalance policy
L7-FARM_20.1:80
loadbalance vip icmp-reply active

policy-map multi-match L4-POLICYMAPMULTI-NAT
class L4-MAP-NAT
nat static 10.0.21.1 netmask 255.255.255.255 vlan 151


service-policy input
L4-POLICYMAPMULTI-LOADBALANCE

interface vlan 101
description SERVERSIDE
ip address 10.0.10.199 255.255.255.0
service-policy input L4-POLICYMAPMULTI-NAT
no shutdown

interface vlan 151
description FIREWALLSIDE
ip address 10.0.0.2 255.255.255.240
no shutdown

Tuesday, June 19, 2007

Sorted Hashtable

Need an HashTable uh? Tired of that messy foreach statement ? Here's a simple implementation of a Dictionary collection, but sortable.

namespace snippets101.Collections
{
public enum sortType
{
NoSort = 1,
Keys = 2,
Values = 3
}
public class sHash
{
private ArrayList _alVals;
private ArrayList _alKeys;
private sortType _sort;

public sHash()
{

_alKeys = new ArrayList();
_alVals = new ArrayList();

_sort = sortType.NoSort;

}

public sortType SortType
{
get { return _sort; }
set { _sort = value; }

}

public void Sort()
{
Array k = _alKeys.ToArray(typeof(object));
Array v = _alVals.ToArray(typeof(object));

if (_sort == sortType.Keys)
Array.Sort(k, v, new sHashComparer());
if (_sort == sortType.Values)
Array.Sort(v, k, new sHashComparer());

_alKeys = ArrayList.Adapter(k);
_alVals = ArrayList.Adapter(v);

}

public void Add(object key, object value)
{
if (_alKeys.Contains(key))
throw new Exception("Key already present.");

_alKeys.Add(key);
_alVals.Add(value);

}

public void Clear()
{
_alKeys.Clear();
_alVals.Clear();
}

public bool Contains(object key)
{
return _alKeys.Contains(key);
}

public IDictionaryEnumerator GetEnumerator()
{
return new sHashDictionaryEnumerator(_alKeys, _alVals);
}

public bool IsFixedSize
{
get { return false; }
}

public bool IsReadOnly
{
get { return false; }
}

public ArrayList Keys
{
get { return _alKeys; }
}

public ArrayList Values
{
get { return _alVals; }
}

public void Remove(object key)
{
_alVals.RemoveAt(_alKeys.IndexOf(key));
_alKeys.Remove(key);
}

public object this[object key]
{
get
{
return _alVals[_alKeys.IndexOf(key)];
}
set
{
if (_alKeys.Contains(key))
{
_alVals[_alKeys.IndexOf(key)] = value;
}
else
{
_alKeys.Add(key);
_alVals.Add(value);
}
}
}

public int Count
{
get { return _alKeys.Count; }
}

private class sHashDictionaryEnumerator : IDictionaryEnumerator
{
DictionaryEntry[] items;
Int32 index = -1;

public sHashDictionaryEnumerator(ArrayList keys, ArrayList mappings)
{
items = new DictionaryEntry[keys.Count];
for (int i = 0; i <> {
items[i] = new DictionaryEntry(keys[i],mappings[i]);
}
}

public Object Current { get { ValidateIndex(); return items[index]; } }

public DictionaryEntry Entry
{
get { return (DictionaryEntry)Current; }
}

public Object Key { get { ValidateIndex(); return items[index].Key; } }

public Object Value { get { ValidateIndex(); return items[index].Value; } }

public Boolean MoveNext()
{
if (index < style="color: rgb(51, 51, 255);"> return true; }
return false;
}

private void ValidateIndex()
{
if (index <>= items.Length)
throw new InvalidOperationException("Enumerator is before or after the collection.");
}

public void Reset()
{
index = -1;
}
}

private class sHashComparer : IComparer
{
private bool isNumeric(Type t)
{
switch (t.Name.ToLower())
{
case "int32":
return true;
case "single":
return true;
case "double":
return true;
default:
return false;
}
}


public int Compare(object x, object y)
{

if ((isNumeric(x.GetType())) && (isNumeric(y.GetType())))
{
double d = Math.Truncate(Convert.ToSingle(x) - Convert.ToSingle(y));
if (d <>
return -1;
if (d == 0)
return 0;
if (d > 0)
return 1;

return 0;
}
if ((isNumeric(x.GetType())) && (!isNumeric(y.GetType())))
return -1;
if ((!isNumeric(x.GetType())) && (isNumeric(y.GetType())))
return 1;

return String.Compare(x.ToString(), y.ToString());
}


}


}
}

Monday, June 11, 2007

Dual armed server to server load balancing con Cisco ACE

Let's say you have a large data center, let's say that in this data center you have lots of dual-armed load balanced serverfarms. It could happen that these servers need to call each other's balanced services. Here's how this could be accomplished with very light configuration on real servers.

Scenario :



The BLUE-SERVERFARM real servers needs to query a web service located on the YELLOW-SERVERFARM, on tcp port 2000.
All of the real servers use the "upper" interface (vlan 101) to act as servers, i.e. to answer clients' queries coming from the ACE.
The "internal" interface (vlan 102) is used by the servers when they act as client of someone else's service.
Easy to configure this, matter of routes on the servers. The default gateway is always the ACE, there's a static route on the internal interface for all the ips the server could query acting as a client.


Without configuring Source NAT con the ACE, all connections fail, because of asymmetric response from servers of the YELLOW-SERVERFARM.
When a connection arrives from the ACE, the source ip is the internal interface of the client server. As this ip is on a lan directly connected on the destination server, the response will return over the INTERNAL, not over the same route of the request.



Solution:
Source natting this requests on the ACE will cause the destination server not to know as directly connected the source ip, answering on the default gateway (ACE) and so following the same path of the request.
The simplest way I've found is to reserve a new virtual address only for requests coming from the servers on the same lan, as described above. So clients will continue query the service on the VIP 10.20.0.2 port 2000, while servers on the same lan will query the same service on the same port but on VIP 10.20.0.20, being Source-NATted with an IP from the SNATPOOL.



The real server of the YELLOW-SERVERFARM responding to the request, seeing it from a SNATted address will route the response via the default gateway (ACE) which will send back packets on the same path of the request.

probe tcp 2000-TCP
port 2000
interval 10
passdetect interval 5
passdetect count 1

probe icmp ICMP
interval 10
passdetect interval 5
passdetect count 1

rserver host SERVER-10.1
ip address 10.0.10.1
inservice

rserver host SERVER-10.2
ip address 10.0.10.2
inservice


rserver host SERVER-10.3
ip address 10.0.10.3
inservice

rserver host SERVER-10.4
ip address 10.0.10.4
inservice


serverfarm host BLUE-SERVERFARM_20.1:80
failaction purge
probe ICMP
rserver
SERVER-10.1 80
inservice
rserver SERVER-10.2 80
inservice

serverfarm host YELLOW-SERVERFARM_20.2:2000
failaction purge
probe 2000-TCP
rserver
SERVER-10.3 2000
inservice
rserver SERVER-10.4 2000
inservice


class-map match-all L4-MAP-BLUE-SERVERFARM_20.1:80
2 match virtual-address 10.0.20.1 tcp eq www

class-map match-all L4-MAP-YELLOW-SERVERFARM_20.2:2000
2 match virtual-address 10.0.20.2 tcp eq 2000

class-map match-all L4-SNAT-MAP-YELLOW-SERVERFARM_20.20:2000
2 match virtual-address 10.0.20.20 tcp eq 2000

policy-map type loadbalance first-match L7-BLUE-SERVERFARM_20.1:80
class class-default
serverfarm
BLUE-SERVERFARM_20.1:80

policy-map type loadbalance first-match L7-YELLOW-SERVERFARM_20.2:2000
class class-default
serverfarm
YELLOW-SERVERFARM_20.2:2000

policy-map multi-match L4-POLICYMAPMULTI
class L4-MAP-BLUE-SERVERFARM_20.1:80
loadbalance vip inservice
loadbalance policy
L7-BLUE-SERVERFARM_20.1:80
class L4-MAP-YELLOW-SERVERFARM_20.2:2000
loadbalance vip inservice
loadbalance policy
L7-YELLOW-SERVERFARM_20.2:2000
class L4-MAP-YELLOW-SERVERFARM_20.20:2000
loadbalance vip inservice
loadbalance policy
L7-YELLOW-SERVERFARM_20.2:2000
nat dynamic 1 vlan 101

interface vlan 101
description SERVERSIDE
ip address 10.0.10.199 255.255.255.0
nat-pool 1 10.0.21.1 10.0.21.254 netmask 255.255.255.0
no shutdown
interface vlan 151
description FIREWALLSIDE
ip address 10.0.0.2 255.255.255.240
service-policy input L4-POLICYMAPMULTI
no shutdown



Friday, June 8, 2007

FTP serverfarm on Cisco ACE

This is a mix from Cisco forums and hundreds of trials, being the Cisco ACE Server Load Balancing configuration guide not so clear...

Scenario :



Active FTP :
This is the simpler and it's only a matter of ACE configuration :

probe tcp 21-TCP
port 21
interval 10
passdetect interval 5
passdetect count 1

rserver host SERVER-10.1
ip address 10.0.10.1
inservice

rserver host SERVER-10.2
ip address 10.0.10.2
inservice


serverfarm host FTPFARM_20.1:21
failaction purge
probe 21-TCP
rserver
SERVER-10.1 21
inservice
rserver SERVER-10.2 21
inservice

sticky ip-netmask 255.255.255.255 address source
STICKY-FTPFARM_20.1:21
timeout 20
timeout activeconns
replicate sticky
serverfarm
FTPFARM_20.1:21


class-map match-all L4-MAP-FTPFARM_20.1:21
2 match virtual-address 10.0.20.1 any

policy-map type loadbalance first-match L7-
FTPFARM_20.1:21
class class-default
sticky-serverfarm STICKY-
FTPFARM_20.1:21

policy-map multi-match L4-POLICYMAPMULTI-FTP
class L4-MAP-FTPFARM_20.1:21
loadbalance vip inservice
loadbalance policy
L7-FTPFARM_20.1:21
inspect ftp


interface vlan 101
description SERVERSIDE
ip address 10.0.10.199 255.255.255.0
no normalization
no shutdown
interface vlan 151
description FIREWALLSIDE
ip address 10.0.0.2 255.255.255.240
no normalization
service-policy input L4-POLICYMAPMULTI-FTP
no shutdown


Passive FTP :
In order to make Passive FTP connection work, with the firewall checking consistency of source and destination addresses, youll need to change the FTP server configuration.
On the frox server there's a configuration parameter "PASV Reply Address" that should be set to the VIP (10.0.20.1) in order to have the FTP server call back the client (passive mode) with the same address the firewall see for the active client-server communication.

dotted notation to number

Writing my ip address management software I needed something to order my ips in sql.The dotted notation wasn't so efficient, so, I needed to convert it in an integer.
Wandering on the Net, and with the CCNA study guide open, here's the result, starting from a script read somewhere...


CREATE FUNCTION dbo.IPDottedToNumber( @IPAddress varchar(15))
RETURNS bigint
AS
BEGIN
DECLARE
@biOctetA bigint,
@biOctetB bigint,
@biOctetC bigint,
@biOctetD bigint,
@biIP bigint

DECLARE @tbl TABLE

(
OctetNo smallint,
Octet bigint
)

INSERT INTO @tbl
SELECT ElementID, CONVERT(bigint,Element) FROM dbo.Split(@IPAddress, '.')

IF (SELECT COUNT(*) FROM @tblArray WHERE Octet BETWEEN 0 AND 255) = 4
BEGIN

SET @biOctetA = (SELECT (Octet * 256 * 256 * 256) FROM @tblArray WHERE OctetNo = 1)

SET
@biOctetB = (SELECT (Octet * 256 * 256 ) FROM @tblArray WHERE
OctetNo = 2)

SET
@biOctetC = (SELECT (Octet * 256 ) FROM @tblArray WHERE
OctetNo = 3)

SET
@biOctetD = (SELECT (Octet) FROM @tblArray WHERE
OctetNo = 4)

SET
@biIP = @biOctetA + @biOctetB + @biOctetC + @biOctetD


END


RETURN(@biIP)
END

Thursday, June 7, 2007

You're welcome

As a network engineer and a software developer, I must face technical issues every day, of various complexity. Most of them come from my laziness in reading all of the manual....

However, since the 80% of this issues found a solution by reading something on a forum, a blog, or some code exchange site, I thought it were high time to pay back the favour to Internet.

In this blog I'll put all of the solutions I find (by myself or not), code snippets, configurations, strange things that happens to people like me.

Enjoy.

PS: I'm sure my english is one of the worst you can find around the 'net, but hey! I'm an italian guy, please be patient, I know much better how to cook...