Some time ago I wrote a post on how to get a Cisco IP-Sec VPN working with client certificates on OS-X Lion. Now I upgraded to Mavericks and of course this broke my VPN connection again. Fortunately a friend already had the same problem on Mountain Lion and his fix also worked on Mavericks. To get my connections working on Mavericks I followed the instructions from my previous post: . After that I had to “allow all applications to access this item” on the certificate in Keychain.

Also see the last comment in https://discussions.apple.com/thread/4158642?start=15&tstart=0 saying to allow all access to the cert in Keychain.

I had already set the cert to be always “Trusted” but you have to expand the cert to get to the private key and always “Allow” access to it. It’s a different setting.

See the screenshot below.
VPN cert

As I wrote in my last blog port, I’ve been enabling selinux on some webservers. Last week I updated the Idera CDP agent on one server to support backup and restore of MySQL via the CDP agent. The backup is successful without any issues. Since this integrated MySQL backup was new functionality, I also wanted to test the restore. The restore did not work with selinux enabled. There were a ton of error messages in the audit.log, actually too much to paste in this blog post. I’ve attached the file with error messages cdp-mysql.

To fix the problem I’ve created multiple selinux policies, after the first 4 tries new deny messages appeared in the audit.log. After the fifth version of the policy the restore finished without any error in the log and the database that I dropped and I tried to restore was available and accessible for the sites.

To create the working policy I did the following. I copied the messages that I attached into a separate file (cdp-mysql.se) and used the following command to create a selinux policy:


audit2allow -i cdp-mysql.se -M cdp-mysql

This creates a couple of files (cdp-mysql.pp and cdp-mysql.te) in the current working directory. The cdp-mysql.te contains the plain text policy. The cdp-mysql.pp file can be used to import the selinux policy:


semodule -i cdp-mysql.pp

This activates the cronolog selinux policy that contains the configuration listed below. After this module is activated cronolog is allowed to create directories under the log directory.

module cdp-mysql 1.0;

require {
type bin_t;
type fixed_disk_device_t;
type mysqld_t;
type port_t;
type var_lib_t;
class sock_file { create unlink getattr };
class tcp_socket name_bind;
class chr_file { read write };
class file { write getattr read lock open append };
}

#============= mysqld_t ==============
allow mysqld_t bin_t:file append;
allow mysqld_t fixed_disk_device_t:chr_file { read write };
#!!!! This avc can be allowed using the boolean 'allow_ypbind'

allow mysqld_t port_t:tcp_socket name_bind;
#!!!! The source type 'mysqld_t' can write to a 'file' of the following types:
# mysqld_db_t, hugetlbfs_t, mysqld_tmp_t, mysqld_log_t, mysqld_var_run_t, root_t

allow mysqld_t var_lib_t:file { read write getattr open lock };
allow mysqld_t var_lib_t:sock_file { create unlink getattr };

For those who want to use it I’ve attaced the cdp-mysql.pp module. Make sure to test the md5 checksum (44ec3ec35db17e0adab38ad0ba1fac10 cdp-mysql.pp). You can also recreate the module with the file containing the errors from the audit.log

In the last couple of weeks I’ve started enabling selinux in enforcing mode on my webservers. Besides some expected problems, I encountered a problem with cronolog when I enabled selinux. There was no way that cronolog wanted to create the needed directories. Usually we run cronolog to create separate “Year” and “Month” directories and in every “Month” directory a separate directory and file for each day.

When I started apache with the cronolog log configuration, the following error appeared in the global apache error_log:

piped log program '/usr/sbin/cronolog /var/log/httpd/yyy.xxx.nl/%Y/%m/%d/access_log' failed unexpectedly
/var/log/httpd/wpdev.redbee.nl/2013: Permission denied
piped log program '/usr/sbin/cronolog /var/log/httpd/yyy.xxx.nl/%Y/%m/%d/access_log' failed unexpectedly

There was no year “2013” directory created in the top level log directory /var/log/httpd/yyy.xxx.nl/
As cronolog never gives any problems in that area and file rights were configured correctly, I suspected selinux. And indeed this was the problem.

In /var/log/audit/audit.log the following errors where logged:

type=AVC msg=audit(1361309216.029:25603): avc: denied { create } for pid=10047 comm="cronolog" name="2013" scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1361309216.029:25603): arch=c000003e syscall=83 success=no exit=-13 a0=7fff5ebbf050 a1=1fd a2=ffffffffffffffa8 a3=322f6c6e2e656562 items=0 ppid=9699 pid=10047 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=175 comm="cronolog" exe="/usr/sbin/cronolog" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1361309235.635:25604): avc: denied { create } for pid=10048 comm="cronolog" name="2013" scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1361309235.635:25604): arch=c000003e syscall=83 success=no exit=-13 a0=7fff4cc1efa0 a1=1fd a2=ffffffffffffffa8 a3=322f6c6e2e656562 items=0 ppid=9699 pid=10048 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=175 comm="cronolog" exe="/usr/sbin/cronolog" subj=unconfined_u:system_r:httpd_t:s0 key=(null)

I copied these messages into a separate file (crono.se) and used the following command to create a selinux policy:


audit2allow -i /home/rogierm/crono.se -M crono

This creates a couple of files (crono.pp and crono.te) in the current working directory. The crono.te contains the plain text policy. The crono.pp file can be used to import the selinux policy:


semodule -i crono.pp

This activates the cronolog selinux policy that contains the configuration listed below. After this module is activated cronolog is allowed to create directories under the log directory.


module crono 1.0;

require {
type httpd_log_t;
type httpd_t;
class dir create;
}

#============= httpd_t ==============
allow httpd_t httpd_log_t:dir create;

Since the Cisco VPN client does not work under OSX Lion anymore there was no easy way to connect with certificate authentication. It took some time but I managed to get it working under Lion with the build in VPN Client. Find the steps below to get the certificates imported and working with the VPN Client.

  • Create key: openssl genrsa -des3 -out vpn-cert2.key 1024
  • Create CSR (make sure that the CN is a simple name, no spaces or special characters): openssl req -new -key vpn-cert2.key -out vpn-cert2.csr
  • Request certificate with your CA
  • Create a p12 file from the key and the certificate: openssl pkcs12 -export -inkey vpn-cert2.key -in certnew-3.cer -out vpn.p12
  • Import the p12 file (containing the key and certificate) in the system keychain (not the login keychain, that doesn’t work): sudo security import vpn.p12 -k /Library/Keychains/System.keychain
  • If needed you can import the CA in your keychain and trust the imported certificate: sudo security add-trusted-cert -k /Library/Keychains/System.keychain root.ca.pem
    Note:Make sure that if you import your own CA, that you do it this way. Otherwise the VPN server certificate will not be verified correctly.

To use the certificate for VPN authentication do the following:

  • Open System Preferences
  • Go to Network
  • Click + to add network interface, Select Interface: VPN, VPN Type: Cisco IPSec
  • Click Create
  • In the Server Address type the hostname of the firewall. This is really important. The firewall has a certificate configured on the FQDN. Make sure the server address is the name of the certificate in the firewall. This FQDN can be found in the trustpoint configuration (see below)
  • Enter the username
  • Click Authentication Settings
  • Select Certificate and Click Select
  • Select the correct certificate that you just imported
  • Click OK
  • Click Apply

When you are not able to select the certificate you created the problem is that the CN is not supported. Make sure the CN that you used to create the CSR does not contain spaces or special characters.

Firewall trustpoint config:

crypto ca trustpoint CA1
enrollment terminal
fqdn fw.xxxx.com
subject-name CN=fw.xxxx.com,OU=IT,O=XXX Limited,C=NL,St=NH,L=Amsterdam

Last week I was looking at testing SplashID Enterprise. While a first installation with the MySQL database running on a Mac Mini was working fine, the installation with the Splash Enterprise Admin client failed when the database was running on a default Linux (CentOS) installation. I tried contacting SplashData support, but they could not help me, so I tried to debug myself. I enabled the query log so I could see the queries executed by the Admin client. These queries showed that SplashID ran queries reffering to specific tables in upper case (eg. MYSQL.USER). I manually tried some of these queries and these queries failed with “unknown table” error.

Now I found the problem, I thought it was easy to fix it. I just had to make MySQL case-insensitive. This sounds easier than it actually was 🙂 Lots of articles talked about the character set and the collation, but these only affect the data in the tables, not the actual table name itself. Some googling let met to the lower_case_table_names setting in MySQL. It appears that Windows, Unix and MacOSX all have different default settings, and therefor behave differently.

Setting the following line in the my.cnf in the [mysqld] section solved my case problem with SplashID. MySQL now changes all table names to lower case.

lower_case_table_names=1

Update: I have not tested SplashID Enterprise yet, so I don’t know if it is any good 🙂

I’ve never really liked facebook, but also never really felt annoyed by it. But lately there are so many stories on the internet and in the news about facebooks total disregard for privacy. It might be just me noticing this, but I think not 🙂 Anyway, just for fun I decided to collect the stories that show what impact changes in facebook can have.

Your-Face-Will-Soon-Be-In-Facebook-Ads
facebook-apps-kunnen-meer-privedata-gaan-opvragen
rogue-facebook-apps-access-your-home-address-mobile-phone-number
Facebook-Opens-Up-Home-Addresses-and-Phone-Numbers
facebook-app-permissions.html
Private photo’s public?
Afgeschermde facebook foto’s zichtbaar
facebook exploit macht alben nicht freunden einsehbar

Of course, there are also positive sides on facebook and what its use and users can achieve with it:
The-Inside-Story-of-How-Facebook-Responded-to-Tuni

I know facebook is a free service (as in beer), but that should not mean that they can use your data whichever way they want…

Last month I ran some performance tests over a Cisco ASA 5550 using iperf. There were some performance issues when the ASA was hit with a lot of simultaneous requests. The ASA 5550 is a powerful device so I did not expect any performance problems with 2000 concurrent requests. Our stresstests reported connection problems when the number of concurrent requests increased above 2000 while traffic was way below the maximum supported throughput. To check the wirespeed performance of the ASA I decided to run an iperf test. This test showed expected bandwidth results, but a lower MTU (1408), while all intermediate components are configured at 1500.

Some investigation showed that this was caused by a default maximum MSS setting in the ASA. It appears that the ASA has a default max MSS of 1380. This is set by the command:

sysopt connection tcp-mss MSS_size_in_bytes

The default is 1380 to prevent fragmentation on possible IPSec connections in the path.

To get to MTU 1500 the ASA needs to support an MSS of 1460. This is configured with the following command:

sysopt connection tcp-mss 1460

More information can be found here:

Libvirt is a toolkit to interact with several virtualization platform from a single interface. Considering you can stop and start virtual machines through this API, security is quite important. Libvirt offers several options to give authenticated access from remote machines. By default most distributions disable remote network access for libvirtd. However, I would like to access libvirtd on some of my KVM servers from a single management host to gather some information. The documentation on how to set this up is not too good, so I decided to write up a short how-to.

Step 1: Enable network access for libvirtd
First enable network access for libvirtd on the KVM server(s). On CentOS/RHEL this is done by uncommenting or adding the following line in /etc/sysconfig/libvirtd:

LIBVIRTD_ARGS="--listen"

Step 2: Install a CA on the management server
Install the Perl certificate tools:

yum install openssl-perl

Create Certificate authority:

cd /etc/pki/tls/misc/
./CA.pl -newca

Example output:

./CA.pl -newca
CA certificate filename (or enter to create)

Making CA certificate ...
Generating a 1024 bit RSA private key
..........++++++
.............++++++
writing new private key to '../../CA/private/cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:XX
State or Province Name (full name) [Berkshire]:XX
Locality Name (eg, city) [Newbury]:XXXXX
Organization Name (eg, company) [My Company Ltd]:XXXXX
Organizational Unit Name (eg, section) []:XXXX
Common Name (eg, your name or your server's hostname) []:CA XXX XXX
Email Address []:XXX

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for ../../CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number:
d8:95:24:xx:xx:xx:13:9b
Validity
Not Before: Feb 25 23:14:08 2010 GMT
Not After : Feb 24 23:14:08 2013 GMT
Subject:
countryName = XX
stateOrProvinceName = XX
organizationName = XXXX
organizationalUnitName = XXXX
commonName = CA XXX XXX
emailAddress = XXXXX
X509v3 extensions:
X509v3 Subject Key Identifier:
XXX
X509v3 Authority Key Identifier:
keyid:XXXX
DirName:/C=XX/ST=XX/O=XXX/OU=XXXX/CN=CA XXX XXX/emailAddress=XXX
serial:XXX

X509v3 Basic Constraints:
CA:TRUE
Certificate is to be certified until Feb 24 23:14:08 2013 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated

Step 3: Create CSR’s

openssl genrsa -des3 -out kvm-server1.tmp
openssl rsa -in kvm-server1.tmp -out kvm-server1.key
openssl genrsa -des3 -out mgmt-host.tmp
openssl rsa -in mgmt-host.tmp -out mgmt-host.key
openssl req -new -key kvm-server1.key -out kvm-server1.csr
openssl req -new -key mgmt-host.key -out mgmt-host.csr

Step 4: Sign the certificates

openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -out /root/mgmt-host.crt -infiles /root/mgmt-host.csr
openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -out /root/kvm-server1.crt -infiles /root/kvm-server1.csr

Example output:

Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for /etc/pki/CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number:
d8:95:24:4b:4e:b1:13:9c
Validity
Not Before: Feb 25 23:31:40 2010 GMT
Not After : Feb 25 23:31:40 2011 GMT
Subject:
countryName = XX
stateOrProvinceName = XX
localityName = XX
organizationName = XX
organizationalUnitName = XX
commonName = mgmt-host.xxx.nl
emailAddress = xxxxx
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
6C:EA:8B:C1:D6:XX:B6:6B:5B:18:02
X509v3 Authority Key Identifier:
keyid:C9:36:4A:XXXX:6F:FD:2E:86

Certificate is to be certified until Feb 25 23:31:40 2011 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Step 5: Copy over the certificates to the correct location
On the management host (mgmt-host):

mkdir /etc/pki/libvirt
mkdir /etc/pki/libvirt/private
mkdir /etc/pki/libvirt-vnc

cp /root/mgmt-host.key /etc/pki/libvirt/private/clientkey.pem
cp /root/mgmt-host.key /etc/pki/libvirt-vnc/clientkey.pem
cp /root/mgmt-host.crt /etc/pki/libvirt/clientcert.pem
cp /root/mgmt-host.crt /etc/pki/libvirt-vnc/clientcert.pem

Transfer the key and certificate files to the KVM server (kvm-server1). Ideally, you create the key and CSR on the host itself, so you only have to transfer the certificate. Then, copy the certificates and CA to the correct location on the KVM (libvirtd) server:


mkdir /etc/pki/libvirt
mkdir /etc/pki/libvirt/private
mkdir /etc/pki/libvirt-vnc

cp kvm-server1.key /etc/pki/libvirt/private/serverkey.pem
cp kvm-server1.key /etc/pki/libvirt-vnc/server-key.pem

cp kvm-server1.crt /etc/pki//libvirt/servercert.pem
cp kvm-server1.crt /etc/pki/libvirt-vnc/server-cert.pem

Make sure the CA generated on the management server is available on the KVM server in the following file:
/etc/pki/CA/cacert.pem

Step 6: Reload libvirtd

/etc/init.d/libvirtd reload

Step 7: Test
With these certificates setup, you should be able to access libvirtd on kvm-server1 from mgmt-host. Use the following command to test:

virsh -c qemu://kvm-server1.xxxx.nl/system
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
'quit' to quit

virsh #

Use the list command to see a list of running guests on the server. This only works if these guests have also been created via libvirtd. Manually started KVM guests will not show up in this list.

While migrating the authentication of our ASA firewalls to tacacs, we enabled ‘enable’ authentication to tacacs and tried to switch to enable mode on the console. This did not work, and caused the following message in the tacacs log file:

Wed Jan 13 17:07:42 2010 [25444]: enable query for 'username' 13 from 10.x.x.x rejected

To fix this problem the tacacs configuration for the user needs to include the enable password in the profile, as shown below:

user = username {
login = des "XXXXXXX"
member = admin
acl = mgmt_devices
service = shell {
priv-lvl = 15
}
enable = des "XXXXXXX"
}

We use the following configuration on the ASA to enable AAA to tacacs.

aaa-server tacacs protocol tacacs+
aaa-server tacacs (outside) host 1.1.1.1
key TACACSKEY
aaa-server tacacs (outside) host 2.2.2.2
key TACACSKEY
aaa authentication ssh console tacacs LOCAL
aaa authentication telnet console tacacs LOCAL
aaa authentication serial console tacacs LOCAL
aaa authentication enable console tacacs LOCAL
aaa authentication http console tacacs LOCAL
aaa authorization command tacacs LOCAL

Tacacs is a great way to centralize user authentication, authorization and accounting. While tacacs originally is a Cisco thing, there is an open source server version available, tac_plus (http://www.gazi.edu.tr/tacacs/index.php?page=download). Installing the tacacs server is quite straight forward. Configuring the switch is not difficult either, as long as you think about possible failures. You don’t want to be locked out of your switches when your tacacs server is not available. I use the following configuration that uses two tacacs servers and asks for the enable password when neither of the tacacs servers is available. To enter ‘enable’ mode, the configured enable password suffices. Use the following Cisco configuration for a save AAA authentication.

NOTE: Always be careful when changing authentication and authorization configuration, as this might lock you out of the device. The savest way is to do this on the console of the machine.


aaa new-model
aaa authentication login default group tacacs+ enable
aaa authentication enable default enable
aaa authorization exec default group tacacs+ if-authenticated
aaa authorization commands 15 default group tacacs+ if-authenticated
aaa authorization network default group tacacs+ if-authenticated
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
aaa accounting system default start-stop group tacacs+
aaa session-id common
tacacs-server host 1.1.1.1 single-connection
tacacs-server host 2.2.2.2 single-connection
tacacs-server key TACACSKEY
tacacs-server directed-request

To restrict access to specific devices, you can configure an ACL in the tacacs configuration on the server (tac_plus.conf). See the example below.


user = username {
login = des "XXXX"
member = admin
acl = mgmt_devices
service = shell {
priv-lvl = 15
}
}
group = admin {
default service = permit
service = exec {
priv-lvl = 15
}
}
# acl's

acl = mgmt_devices {
permit = 12.12.12.12
permit = 13.13.13.13
}