In general it is a good idea to configure password aging as part of your password/security policy. In some cases however, this might cause unexpected problems. I’ve seen cases where an expired password prevented a machine from booting. In this specific case this was caused by a service that ran as the user with the expired password. In general you should not run services as a normal user account, but sometimes you just have to deal with things you can’t change. Generally the documentation states that to disable password aging you have to edit the /etc/shadow file, and remove the part where the password age is stored. This is quite error prone. If you do it this way, be sure to use vipw to prevent errors in this critical file. To disable password aging I recommend just using the command to enable it as well:

# chage -m 0 -M 99999 -E -1 username

Check the before and after:

# chage -l username
Minimum: 7
Maximum: 90
Warning: 7
Inactive: -1
Last Change: Jun 26, 2009
Password Expires: Sep 24, 2009
Password Inactive: Never
Account Expires: Never

After disabling password aging:

# chage -l username
Minimum: 0
Maximum: 99999
Warning: 7
Inactive: -1
Last Change: Jun 26, 2009
Password Expires: Never
Password Inactive: Never
Account Expires: Never

As a note, please only disable password aging when there is no other way to fix the problem.

We have a combination of Cisco 2500 terminal server (oldies) and some Avocent ACS terminal servers. All our cisco kit authenticates against a tacacs server (tac_plus) and I want to include the Avocent in the same central user-management infrastructure.

The Avocent manual includes some commands to configure it to authenticate against different back-ends. The tacacs commands and options are all explained, but these commands did not give me a working setup. Below I outline the steps in a small how-to to setup Tacacs authentication on an Avocent terminal server.

  1. Login to the avocent as root
  2. enter the command line interface:

    [root@hostname root]# CLI

    - Thanks for using the CLI -

    This interface allows you to easily modify configurations to customize
    and define the functionality of your unit.

    Some basic and useful keys are:
    up/down arrow - navigates up/down in the command history
    tab (once/twice) - shows the next possible option(s)

    Other hints:
    Put quotes around strings that contain spaces.

    Please refer to the Reference Guide for other special keys and
    additional information on how to use this interface.

    Press TAB to see the list of available options.

    cli>

  3. Configure the ACS to use tacacs for physical ports:

    cli>config physicalports all access authtype TacacsPlusDownlocal

  4. Configure the ACS to use tacacs for authentication to the terminal server itself:

    cli>config security authentication authtype tacasdownlocal

  5. Configure the tacacs parameters:

    cli>config security authentication tacplusauthsvr1 10.x.x.x
    cli>config security authentication tacplussecret T@C@CSk3y

  6. Commit the config:

    cli>config runconfig

  7. Save the config to flash:

    cli>config savetoflashadf

With the setup described above I was not able to succesfully login to the Avocent with a valid tacacs user. The following entries was written in the tacacs log file:

Thu Jul 30 18:29:16 2009 [23176]: pap-login query for 'testuser' ssh from hostname.domain rejected
Thu Jul 30 18:29:16 2009 [23176]: login failure: testuser hostname.domain (10.x.x.x) ssh

The cause of the problem was that the Avocent uses ‘PAP’ authentication and this needs to be configured separately for the tacacs user. See the example below:


user = testuser {
default service = permit
name = "Test User"
login = cleartext "password"
pap = cleartext "password"
service = exec {
priv-lvl = 15
}
}

I encountered a problem  when logging in with a ‘restricted’ rancid user (see article). This user does not have the proper authorization, since this is only setup to backup Cisco configuration. In my setup this is not a problem, but be aware of this issue. The problem can be seen in the tacacs log file:


Thu Jul 30 18:47:12 2009 [1541]: authorization query for 'rancid' ssh from cltsp-ts01.ams-spa rejected

Yesterday I read an article in the Washington Post about a big security breach at Network Solutions where >500.000 credit and debit cards are stolen. Network Solutions acknowledge this security incident on their site. They claim the cause of this incident was malicious code that was uploaded to a platform supporting their merchants sites. How this was possible and how this could lead to the possible theft of half a million credit card numbers is unclear.

I’ve been involved in several PCI projects to help our customers become PCI compliant as required by the credit card issuing companies. While the need for most of the procedures and measures required by PCI is clear, some seem useless, costly and/or superfluous to implement. But after incidents like these, it is a lot easier to explain customers the point of the measures PCI requires of companies handling credit card data. Basically everything is aimed at protecting the creditcard data, and making sure in case of a security incident all needed audit trails are available to investigate the cause and source of the attack. At least that is how I look at it.

For instance, PCI requires you to audit the integrity of the files present on the system. This should not only include OS files, but also the (web) application code. It is premature to speculate if this ‘malicious code’ could have been detected by running a properly configured host-based IDS on the platform, such as Tripwire or Samhain.

Secondly, PCI requires you to establish roles that have access to a production platform to upload code. Staff members should be part of a role that authorizes them for the access they need to do their job. Access should be restricted to allow only this traffic. Not only should access be locked, but audit trails of all activity should be available upon request. These audit trails not only include who logged in to the systems, but should also include network IDS logs (eg. SNORT), commands that are executed (eg. sudo), output of these commands (eg. rootsh) and the reports of host-based IDS’s. Together these tools should give an auditor a good insight in the activity on a (compromised) server. Also SElinux could be a big help in restricting access. Strangely PCI does not require or advise the use of SElinux, while it does require the use of application level firewalls (eg. mod_security in Apache). But this is a different discussion 🙂

To secure the creditcard data itself, PCI requires that all creditcards should be stored in encrypted from. Manual access to keys to decrypt this data should not be possible. Based on this you can infer that the creditcard data is compromised by some sort of ‘man-in-the-middle’ attack. The malicious code could have intercepted the data after it was decrypted in the webserver, leaving the SSL tunnel, and before it was encrypted and stored in the database. But this is just speculating of course…

I’m looking forward to more details on this incident. I hope this is made publicly available so we can learn from the mistakes that were made.

Mac OSX uses a DNS cache. This cache stores the nameserver responses. While this is useful in general cases, it can be a pain when testing a website that is being moved to a different server. The cache will still serve the old ip address of the site, even if the DNS server already has the new ip address configured. To overrule the cache, and request the latest information with the authoritive DNS server, you can flush the DNS cache. To do this, open a Terminal window and enter the following command:


$ sudo dscacheutil -flushcache

Due to the shortage of IPv4 IP addresses, we will run out of IPv4 some day in the near future (approx. 699 days from now, http://inetcore.com/project/ipv4ec/index_en.html). To be prepared for this we are experimenting with IPv6. We got a /32 allocated by RIPE, and are currently preparing a live network to connect some services via native  IPv6 to the internet. While reviewing the IPv6 capabilities of the network equipment we use, I found out that Cisco ASA/Pix does not support failover when running IPv6. For details on this matter please see the following links:

  • http://www.v4tov6.com/2009/06/cisco-asa-ipv6-failover-update.html
  • http://v4tov6.com/2008/11/cisco-asa-ipv6-failover.html
  • http://forums.cabling-design.com/cisco/Re-PIX-IPv6-Failover-bug-4167-.htm

This failover support seems to be lacking for some time now, and is still missing in the latest release. I’ve notified our sales rep. at Cisco about this. Personally I think this is quite a serious issue, as failover is a essential part of building serious infrastructures. I hope Cisco will see that this feature should be implemented as soon as possible.

If you consider this a serious issue as well, I recommend you notify your Cisco contact. Also leave a comment, just because I’m wondering how many people think this is a problem 🙂

06 april 2010: Cisco released ASA version 8.3 which solves this problem: http://www.networkworld.com/community/node/58537

When using rancid you have to store the username and password in a text file. When you don’t want to give the user privilege level 15 you have to store the enable password as well. Tacacs with authorization is the best solution to restrict access for the rancid user. Since the rancid user doesn’t need to change any configuration on the network devices, you can restrict the commands it is allowed to run.

When using tac_plus (http://www.shrubbery.net/tac_plus/) you can use the following definition for the rancid user:

user = rancid {
#    default service = permit
login = cleartext "XXX"
enable = cleartext "XXX"
name = "Rancid User"
service = exec {
priv-lvl = 15
}
cmd = show {
permit .*
}
cmd = write {
permit term
}
cmd = dir {
permit .*
}
cmd = admin {
permit .*
}
cmd = more {
permit .*
}

}

user = rancid {
#    default service = permit
login = cleartext “R4nc!d”
enable = cleartext “raNc1d_3naB1e”
name = “Rancid User”
service = exec {
priv-lvl = 15
}
cmd = show {
permit .*
}
cmd = write {
permit term
}
cmd = dir {
permit .*
}
cmd = admin {
permit .*
}
}T

The rancid user is automatically in enable mode because the privilege level is set to 15 in tacacs. You have to configure rancid no to enter enable mode. This is configured (for cisco devices) in ~rancid/.cloginrc

Enter the following details:

add user        *       rancid
add password    *       XXX
add method      *       telnet
add autoenable  *       1

Last few weeks I’ve been experimenting with OpenQRM (http://www.openqrm.org). The initial installation did not cause any big problems. After logging in to the interface and trying to enable plugins, this just timed-out. I tried to manually start the plugins on the console (eg. /usr/lib/openqrm/plugins/dns/etc/init.d/openqrm-plugin-dns  start). This command started the plugin without any problem. Next I tried to configure select a volume group in the Plugin section (Storage -> LVM-storage-> Volume Groups). This also timed out with the error below in the http errorlog:

This error was caused by SElinux. After I disabled SElinux and rebooted the server, all functions of the OpenQRM interface worked fine, including enabling plugins and displaying volumegroups.


[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fopen(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 103

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fwrite(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 104

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fclose(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 105

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  unlink(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.unlink'>function.unlink</a>]: No such file or directory in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 1547

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fopen(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 103
[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fwrite(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 104
[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fclose(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 105
[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  unlink(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.unlink'>function.unlink</a>]: No such file or directory in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 1547

I’ve had to configure a couple of ServerIron loadbalancers over the last couple of month’s, and each time it surprised me that there are very little example configurations available. I’ve included a diagram with a sample network and the basic (not secured) configuration of the ServerIron to do basic http loadbalancer in a one-armed setup.

The ‘server source ip’ statement is used to connect the realservers in a different subnet then the loadbalancer is in. You can define source IP addresses on a ServerIron system running switch code to place it in multi-netted environment. These source IP addresses could potentially be used as default gateways for real servers. You can also define source NAT IP addresses while using source NAT. The ‘source nat’ statement changes the source ip addres of each request. This causes that the ip adres of the loadbalancer is shown in the access logs on the webserver. This might cause problems for log analysis or geo-ip processing. The example config does not use the ‘source nat’ function.

I’m aware of the fact that this configuration can be improved in many ways, but as stated before, this is just an example for people that are strugling to get their first ServerIron up and running, and can’t find any working config for their situation.

serveriron-diagram

server source-ip 192.168.11.2 255.255.255.0 192.168.11.1

!

context default

!

server real web01 192.168.11.10

port http

port http url "HEAD /"

!

server real web02 192.168.11.20

port http

port http url "HEAD /"

!

server real web03 192.168.11.30

port http

port http url "HEAD /"

!

!

server virtual wts-virt1 192.168.11.100

predictor least-conn

port http

bind http web01 http web02 http web03 http

!

vlan 1 name DEFAULT-VLAN by port

!

vlan 2 by port

!

aaa authentication web-server default local

no enable aaa console

ip address 192.168.11.254 255.255.255.0

ip default-gateway 192.168.11.1

username admin XXX

!

end

server source-ip 192.168.10.2 255.255.255.0 192.168.10.1
!
context default
!
server real web01 192.168.10.10
port http
port http url "HEAD /"
!
server real web02 192.168.10.20
port http
port http url "HEAD /"
!
server real web03 192.168.10.30
port http
port http url "HEAD /"
!
!
server virtual wts-virt1 192.168.10.100
predictor least-conn
port http
bind http web01 http web02 http web03 http
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 by port
!
aaa authentication web-server default local
no enable aaa console
ip address 192.168.10.254 255.255.255.0
ip default-gateway 192.168.10.1
telnet server
username admin password .....
password-change telnet-cli
snmp-server
!
en