I use racktables to keep track of our devices and ip space. To prevent duplicate work and differences in naming I wrote (as all sysadmins 😉 ) a script to export a rancid config file from Racktables. To be able to enable or disable configuration backup via Rancid, I created a Dictionary ‘chapter’ called Rancid, with a Yes and No option. I added this Dictionary as an Attribute and mapped this to the Firewall, Router and Switch objects.

Now I can set the Rancid backup from the properties of the object. To create the Rancid config file I created the following script:


#!/usr/bin/perl

use DBI;

$db="racktables";
$host="localhost";
$user="XXXX";
$passwd="XXXX";
$connectinfo="dbi:mysql:$db;$host";
$filename="racktables-rancid-devices.txt";
$dbh = DBI->connect($connectinfo,$user,$passwd);

$query = "select inet_ntoa(IPBonds.ip), RackObject.name from RackObject JOIN AttributeValue JOIN IPBonds ON RackObject.id=AttributeValue.object_id AND AttributeValue.object_id=IPBonds.object_id WHERE AttributeValue.attr_id=10003 AND AttributeValue.uint_value=50030 group by RackObject.name";

open FILE, ">", "$filename" or die $!;

$sth=$dbh->prepare($query);
$sth->execute();
$sth->bind_columns(\$IP, \$Name);
while($sth->fetch()) {
print FILE "# $Name \n$IP:cisco:up\n";
}

$sth->finish();

$dbh->disconnect;

This script creates the rancid ‘router.db’ configuration format. I created a keypair and used ssh-agent to be able to run the following script to copy over the file to our rancid server.


#!/bin/sh
/home/rancid/export-rancid.pl
scp racktables-rancid-devices.txt rancidserver:.
ssh rancidserver 'cp router-manual.db router.db'
ssh rancidserver 'cat racktables-rancid-devices.txt >> router.db'
ssh rancidserver 'mv router.db /usr/local/rancid/var/networking/router.db'

OpenQRM uses dropbear for the communication and exchange of messages between the server and the appliances. When something goes wrong in this communication OpenQRM can’t function correctly. It can’t access the applicances for status updates and commands. These communication problems are often caused by a misconfiguration in dropbear. The most seen problem is a misconfiguration in the the public and private dropbear key.

The keys should be synchronized between the server and the appliance. On the server grep the public key with the following command:

[root@localhost log]# /usr/lib/openqrm/bin/dropbearkey -t rsa -f /usr/lib/openqrm/etc/dropbear/dropbear_rsa_host_key -y
Public key portion is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCBvwSO7vBBL2avDMds...pVn root@localhost.localdomain
Fingerprint: md5 65:ca:5b:3b:05:c3:61:6d:fb:75:2f:c0:d2:7e:02:cf

Copy the ssh-rsa public key in /root/.ssh/authorized_keys on the appliance.

Now communication should be established.

OpenQRM event log with example of error message caused by communication problem:

openqrm-cmd-queue ERROR executing command with token 64d478dcac6670e5fb000e7c4954863b : /usr/lib/openqrm/bin/dbclient


Aug 26 23:19:45 localhost httpd: openQRM resource-monitor: (update_info) Processing statistics from resource 2
Aug 26 23:19:48 localhost logger: openQRM-cmd-queu: Running Command with token 64d478dcac6670e5fb000e7c4954863b 1. retry : /usr/lib/openqrm/bin/dbclient -I 0 -K 10 -y -i /usr/lib/openqrm/etc/dropbear/dropbear_rsa_host_key -p 1667 root@192.168.42.243 "/usr/lib/openqrm/bin/openqrm-cmd /usr/lib/openqrm/plugins/xen/bin/openqrm-xen post_vm_list -u openqrm -p openqrm"
Aug 26 23:19:52 localhost logger: openQRM-cmd-queu: ERROR executing command with token 64d478dcac6670e5fb000e7c4954863b 2. retry : /usr/lib/openqrm/bin/dbclient -I 0 -K 10 -y -i /usr/lib/openqrm/etc/dropbear/dropbear_rsa_host_key -p 1667 root@192.168.42.243 "/usr/lib/openqrm/bin/openqrm-cmd /usr/lib/openqrm/plugins/xen/bin/openqrm-xen post_vm_list -u openqrm -p openqrm" -----
Aug 26 23:19:52 localhost logger: Host '192.168.42.243' key accepted unconditionally.
Aug 26 23:19:52 localhost logger: (fingerprint md5 64:d5:c7:8e:7a:11:08:3f:43:bc:3c:2b:bf:4a:c8:ce)
Aug 26 23:19:52 localhost logger: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: root@192.168.42.243's password: /usr/lib/openqrm/bin/dbclient: connection to root@192.168.42.243:1667 exited: remote closed the connection

There are couple of differences between IPv6 and IPv4 address allocation.

  • The prefix length for an IPv6 subnet will always be /64; no more, no less. It allows you to place as many IPv6 devices as the underlying network medium allows. The 128 bit IPv6 address is automatically created from the /64 bit subnet extended with the 64 bit MAC-address of the NIC.

With IPv4, prefix length varies between subnets to subnets, and it caused painful costs when renumbering subnets (for example, imagine when you renumber an IPv4 subnet from /28 to /29 or vice versa).

  • An ordinary leaf site will always get /48 of address space. This will be sufficient for most small to medium sized networks.

With IPv4, the allocation varies by the size of the site, and made it very painful when you migrated from one ISP to another, for example.

A SAN is often implemented as a dedicated network that is considered to be a secure network. However, the nature of a SAN is that it is a shared network. This involves some serious security risks, that should be evaluated when using an iSCSI based SAN. Some vendors consider an iSCSI network save when it is implemented as a dedicated switches network (Dell EqualLogic. Securing storage area networks with iSCSI. EqualLogic Inc., 2008.). They consider it virtually impossible to snoop or inject packets in a switched network. We all know this is not the case. If this is true, why do we use firewalls, ids and tons of other security measures? Even if iSCSI runs on an isolated network, and only the management interface of the storage devices are connected to a shared/general-purpose network, security is just as good as the hosts that are connected to the dedicated network. A single compromised host connected to the dedicated iSCSI network can attack the storage devices to get access to LUNs for other hosts.

When implementing an iSCSI network you should be aware of the security risks that this imposes on the environment. To estimate the risk, awareness of the methods that can be used to secure iSCSI is paramount. The iSCSI protocol allows for the following security measures to prevent unintended or unauthorized access to storage resources:

  • Authorization
  • Authentication
  • Encryption

Because iSCSI setups are generally shared environments access to the storage elements (LUNs) by unauthorized initiators should be blocked. Authorization is implemented by means of the iQN. The iQN is the initiator node name (iSCSI Qualified Name), this can be seen as a mac-address. During an audit, storage systems must demonstrate controls to ensure that a server under one regime cannot access the storage assets of a server under another.
Typically, iSCSI storage arrays explicitly map initiators to specific target LUNs; an initiator authenticates not to the storage array, but to the specific storage asset it intends to use.

As an added security method, the iSCSI protocol allows initiators and targets to use CHAP to authenticate each other. This prevents simple access by spoofing the iQN. And last, because iSCSI runs on IP, IPSec can be used to secure and encrypt the data flowing between the client (initiator) and the storage server (target).

Now that we know there are multiple ways to secure access to the storage resouces, you might conclude that iSCSI must be safe and secure to use. Unfortunately this is not evident. There are several flaws in the iSCSI security design:

  • iQN’s are trusted, but are easy to spoof, sniff and guessed
  • iSCSI authorization is the only required security method, and this uses only the iQN
  • Authentication is disabled by default
  • Authentication is (mostly) only implemented as CHAP
  • IPSec is difficult to implement

Because iQN’s are manually configured in the iSCSI driver on the client, it is easy to change them. To get access to a LUN that is only protected by a iQN restriction, you can sniff the communication to get the iQN, or guess the iQN as it is often a default string (eg.: iqn.1991-05.com.microsoft.hostname), configure the iscsi driver to use this name and get access to the LUN.

The CHAP protocol is basically the only authentication mechanism that is supported by iSCSI vendors. The protocol allows for other mechanisms like Kerberos. The CHAP protocol is not a protocol know for its strong security on shared networks. The CHAP protocol is vulnerable to dictionary attacks, spoofing, or reflection attacks. Because the security issues with CHAP are well known, the RFC even mentions ways to deal with the limitations of CHAP (http://tools.ietf.org/html/rfc3720#section-8.2.1).

While IPSec could stop or reduce most of the security issues outlined above, it is hard to implement and manage. Therefor not many administrators will feel the need to use it. It should not only be possible to make a secure network, it should also be made easy.

To reduce the risk, and make your iSCSI network as safe as possible, you should do the following:

  • Enable mutual (incoming/outgoing) authentication
  • Follow advice to secure CHAP
  • Enable CRC checksums
  • Do not only rely on iQN for authorization
  • Enable IPSec (if performance allows it)

Also vendors/distributors should enable authentication by default, and add other authentication mechanisms to the iSCSI target and initiator software.

References:
http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-Dwivedi-update.pdf
http://en.wikipedia.org/wiki/ISCSI#Authentication
http://weird-hobbes.nl/reports/iSCSI%20security/

We have a combination of Cisco 2500 terminal server (oldies) and some Avocent ACS terminal servers. All our cisco kit authenticates against a tacacs server (tac_plus) and I want to include the Avocent in the same central user-management infrastructure.

The Avocent manual includes some commands to configure it to authenticate against different back-ends. The tacacs commands and options are all explained, but these commands did not give me a working setup. Below I outline the steps in a small how-to to setup Tacacs authentication on an Avocent terminal server.

  1. Login to the avocent as root
  2. enter the command line interface:

    [root@hostname root]# CLI

    - Thanks for using the CLI -

    This interface allows you to easily modify configurations to customize
    and define the functionality of your unit.

    Some basic and useful keys are:
    up/down arrow - navigates up/down in the command history
    tab (once/twice) - shows the next possible option(s)

    Other hints:
    Put quotes around strings that contain spaces.

    Please refer to the Reference Guide for other special keys and
    additional information on how to use this interface.

    Press TAB to see the list of available options.

    cli>

  3. Configure the ACS to use tacacs for physical ports:

    cli>config physicalports all access authtype TacacsPlusDownlocal

  4. Configure the ACS to use tacacs for authentication to the terminal server itself:

    cli>config security authentication authtype tacasdownlocal

  5. Configure the tacacs parameters:

    cli>config security authentication tacplusauthsvr1 10.x.x.x
    cli>config security authentication tacplussecret T@C@CSk3y

  6. Commit the config:

    cli>config runconfig

  7. Save the config to flash:

    cli>config savetoflashadf

With the setup described above I was not able to succesfully login to the Avocent with a valid tacacs user. The following entries was written in the tacacs log file:

Thu Jul 30 18:29:16 2009 [23176]: pap-login query for 'testuser' ssh from hostname.domain rejected
Thu Jul 30 18:29:16 2009 [23176]: login failure: testuser hostname.domain (10.x.x.x) ssh

The cause of the problem was that the Avocent uses ‘PAP’ authentication and this needs to be configured separately for the tacacs user. See the example below:


user = testuser {
default service = permit
name = "Test User"
login = cleartext "password"
pap = cleartext "password"
service = exec {
priv-lvl = 15
}
}

I encountered a problem  when logging in with a ‘restricted’ rancid user (see article). This user does not have the proper authorization, since this is only setup to backup Cisco configuration. In my setup this is not a problem, but be aware of this issue. The problem can be seen in the tacacs log file:


Thu Jul 30 18:47:12 2009 [1541]: authorization query for 'rancid' ssh from cltsp-ts01.ams-spa rejected

Due to the shortage of IPv4 IP addresses, we will run out of IPv4 some day in the near future (approx. 699 days from now, http://inetcore.com/project/ipv4ec/index_en.html). To be prepared for this we are experimenting with IPv6. We got a /32 allocated by RIPE, and are currently preparing a live network to connect some services via native  IPv6 to the internet. While reviewing the IPv6 capabilities of the network equipment we use, I found out that Cisco ASA/Pix does not support failover when running IPv6. For details on this matter please see the following links:

  • http://www.v4tov6.com/2009/06/cisco-asa-ipv6-failover-update.html
  • http://v4tov6.com/2008/11/cisco-asa-ipv6-failover.html
  • http://forums.cabling-design.com/cisco/Re-PIX-IPv6-Failover-bug-4167-.htm

This failover support seems to be lacking for some time now, and is still missing in the latest release. I’ve notified our sales rep. at Cisco about this. Personally I think this is quite a serious issue, as failover is a essential part of building serious infrastructures. I hope Cisco will see that this feature should be implemented as soon as possible.

If you consider this a serious issue as well, I recommend you notify your Cisco contact. Also leave a comment, just because I’m wondering how many people think this is a problem 🙂

06 april 2010: Cisco released ASA version 8.3 which solves this problem: http://www.networkworld.com/community/node/58537

When using rancid you have to store the username and password in a text file. When you don’t want to give the user privilege level 15 you have to store the enable password as well. Tacacs with authorization is the best solution to restrict access for the rancid user. Since the rancid user doesn’t need to change any configuration on the network devices, you can restrict the commands it is allowed to run.

When using tac_plus (http://www.shrubbery.net/tac_plus/) you can use the following definition for the rancid user:

user = rancid {
#    default service = permit
login = cleartext "XXX"
enable = cleartext "XXX"
name = "Rancid User"
service = exec {
priv-lvl = 15
}
cmd = show {
permit .*
}
cmd = write {
permit term
}
cmd = dir {
permit .*
}
cmd = admin {
permit .*
}
cmd = more {
permit .*
}

}

user = rancid {
#    default service = permit
login = cleartext “R4nc!d”
enable = cleartext “raNc1d_3naB1e”
name = “Rancid User”
service = exec {
priv-lvl = 15
}
cmd = show {
permit .*
}
cmd = write {
permit term
}
cmd = dir {
permit .*
}
cmd = admin {
permit .*
}
}T

The rancid user is automatically in enable mode because the privilege level is set to 15 in tacacs. You have to configure rancid no to enter enable mode. This is configured (for cisco devices) in ~rancid/.cloginrc

Enter the following details:

add user        *       rancid
add password    *       XXX
add method      *       telnet
add autoenable  *       1

I’ve had to configure a couple of ServerIron loadbalancers over the last couple of month’s, and each time it surprised me that there are very little example configurations available. I’ve included a diagram with a sample network and the basic (not secured) configuration of the ServerIron to do basic http loadbalancer in a one-armed setup.

The ‘server source ip’ statement is used to connect the realservers in a different subnet then the loadbalancer is in. You can define source IP addresses on a ServerIron system running switch code to place it in multi-netted environment. These source IP addresses could potentially be used as default gateways for real servers. You can also define source NAT IP addresses while using source NAT. The ‘source nat’ statement changes the source ip addres of each request. This causes that the ip adres of the loadbalancer is shown in the access logs on the webserver. This might cause problems for log analysis or geo-ip processing. The example config does not use the ‘source nat’ function.

I’m aware of the fact that this configuration can be improved in many ways, but as stated before, this is just an example for people that are strugling to get their first ServerIron up and running, and can’t find any working config for their situation.

serveriron-diagram

server source-ip 192.168.11.2 255.255.255.0 192.168.11.1

!

context default

!

server real web01 192.168.11.10

port http

port http url "HEAD /"

!

server real web02 192.168.11.20

port http

port http url "HEAD /"

!

server real web03 192.168.11.30

port http

port http url "HEAD /"

!

!

server virtual wts-virt1 192.168.11.100

predictor least-conn

port http

bind http web01 http web02 http web03 http

!

vlan 1 name DEFAULT-VLAN by port

!

vlan 2 by port

!

aaa authentication web-server default local

no enable aaa console

ip address 192.168.11.254 255.255.255.0

ip default-gateway 192.168.11.1

username admin XXX

!

end

server source-ip 192.168.10.2 255.255.255.0 192.168.10.1
!
context default
!
server real web01 192.168.10.10
port http
port http url "HEAD /"
!
server real web02 192.168.10.20
port http
port http url "HEAD /"
!
server real web03 192.168.10.30
port http
port http url "HEAD /"
!
!
server virtual wts-virt1 192.168.10.100
predictor least-conn
port http
bind http web01 http web02 http web03 http
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 by port
!
aaa authentication web-server default local
no enable aaa console
ip address 192.168.10.254 255.255.255.0
ip default-gateway 192.168.10.1
telnet server
username admin password .....
password-change telnet-cli
snmp-server
!
en