Security | Rogierm's Blog

In the last couple of weeks I’ve started enabling selinux in enforcing mode on my webservers. Besides some expected problems, I encountered a problem with cronolog when I enabled selinux. There was no way that cronolog wanted to create the needed directories. Usually we run cronolog to create separate “Year” and “Month” directories and in every “Month” directory a separate directory and file for each day.

When I started apache with the cronolog log configuration, the following error appeared in the global apache error_log:
piped log program '/usr/sbin/cronolog /var/log/httpd/yyy.xxx.nl/%Y/%m/%d/access_log' failed unexpectedly /var/log/httpd/wpdev.redbee.nl/2013: Permission denied piped log program '/usr/sbin/cronolog /var/log/httpd/yyy.xxx.nl/%Y/%m/%d/access_log' failed unexpectedly

There was no year “2013” directory created in the top level log directory /var/log/httpd/yyy.xxx.nl/
As cronolog never gives any problems in that area and file rights were configured correctly, I suspected selinux. And indeed this was the problem.

In /var/log/audit/audit.log the following errors where logged:
type=AVC msg=audit(1361309216.029:25603): avc: denied { create } for pid=10047 comm="cronolog" name="2013" scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:httpd_log_t:s0 tclass=dir type=SYSCALL msg=audit(1361309216.029:25603): arch=c000003e syscall=83 success=no exit=-13 a0=7fff5ebbf050 a1=1fd a2=ffffffffffffffa8 a3=322f6c6e2e656562 items=0 ppid=9699 pid=10047 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=175 comm="cronolog" exe="/usr/sbin/cronolog" subj=unconfined_u:system_r:httpd_t:s0 key=(null) type=AVC msg=audit(1361309235.635:25604): avc: denied { create } for pid=10048 comm="cronolog" name="2013" scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:httpd_log_t:s0 tclass=dir type=SYSCALL msg=audit(1361309235.635:25604): arch=c000003e syscall=83 success=no exit=-13 a0=7fff4cc1efa0 a1=1fd a2=ffffffffffffffa8 a3=322f6c6e2e656562 items=0 ppid=9699 pid=10048 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=175 comm="cronolog" exe="/usr/sbin/cronolog" subj=unconfined_u:system_r:httpd_t:s0 key=(null)

I copied these messages into a separate file (crono.se) and used the following command to create a selinux policy:

audit2allow -i /home/rogierm/crono.se -M crono

This creates a couple of files (crono.pp and crono.te) in the current working directory. The crono.te contains the plain text policy. The crono.pp file can be used to import the selinux policy:

semodule -i crono.pp

This activates the cronolog selinux policy that contains the configuration listed below. After this module is activated cronolog is allowed to create directories under the log directory.

module crono 1.0;


require {

	type httpd_log_t;

	type httpd_t;

	class dir create;

}

#============= httpd_t ============== allow httpd_t httpd_log_t:dir create;

Ten-thousands Belgian credit cards were blocked because hackers copied cardnumbers. This was published by Febelfin (the Federation of the Belgian finance sector) on tuesday October 20th 2009. It was stated that the criminals copied the numbers after they gained access to computer files. The hackers did not succeed in retrieving the pincodes.

I find it very strange that hackers were able to retrieve creditcard numbers from computer files. Specifically since PCI puts great restrictions on storing this stuff:

PCI 3.1 – Keep cardholder data storage to a minimum
PCI 3.2 – Do not store sensitive authentication data after authorization
PCI 3.4 – Render PAN (PAN == cc-number), at minimum unreadable anywhere it is stored by using one of the following: one-way hash, truncation, strong cryptography with associated key-management processes and procedures
PCI 3.4.1 – If dis encryption is used rather then file or database encryption, logical access must be managed independently of native OS access control

There are other requirements but the above give a good idea of the caution you must take when deciding to store this data. The full list of requirements can be downloaded from https://www.pcisecuritystandards.org/security_standards/pci_dss_download.html

Personally, I think these news items should contain more information, or should be evaluated somewhere so people can learn from the mistakes made by others. By being open on the causes of these problems people can easily prevent the holes that others left open. I know, this will probably not happen any time soon… 🙂

Link to original story: http://www.elsevier.nl/web/Nieuws/Internet-Gadgets/248802/Duizenden-creditcards-geblokkeerd-na-aanval-hackers.htm#

A SAN is often implemented as a dedicated network that is considered to be a secure network. However, the nature of a SAN is that it is a shared network. This involves some serious security risks, that should be evaluated when using an iSCSI based SAN. Some vendors consider an iSCSI network save when it is implemented as a dedicated switches network (Dell EqualLogic. Securing storage area networks with iSCSI. EqualLogic Inc., 2008.). They consider it virtually impossible to snoop or inject packets in a switched network. We all know this is not the case. If this is true, why do we use firewalls, ids and tons of other security measures? Even if iSCSI runs on an isolated network, and only the management interface of the storage devices are connected to a shared/general-purpose network, security is just as good as the hosts that are connected to the dedicated network. A single compromised host connected to the dedicated iSCSI network can attack the storage devices to get access to LUNs for other hosts.

When implementing an iSCSI network you should be aware of the security risks that this imposes on the environment. To estimate the risk, awareness of the methods that can be used to secure iSCSI is paramount. The iSCSI protocol allows for the following security measures to prevent unintended or unauthorized access to storage resources:

Authorization
Authentication
Encryption

Because iSCSI setups are generally shared environments access to the storage elements (LUNs) by unauthorized initiators should be blocked. Authorization is implemented by means of the iQN. The iQN is the initiator node name (iSCSI Qualified Name), this can be seen as a mac-address. During an audit, storage systems must demonstrate controls to ensure that a server under one regime cannot access the storage assets of a server under another.
Typically, iSCSI storage arrays explicitly map initiators to specific target LUNs; an initiator authenticates not to the storage array, but to the specific storage asset it intends to use.

As an added security method, the iSCSI protocol allows initiators and targets to use CHAP to authenticate each other. This prevents simple access by spoofing the iQN. And last, because iSCSI runs on IP, IPSec can be used to secure and encrypt the data flowing between the client (initiator) and the storage server (target).

Now that we know there are multiple ways to secure access to the storage resouces, you might conclude that iSCSI must be safe and secure to use. Unfortunately this is not evident. There are several flaws in the iSCSI security design:

iQN’s are trusted, but are easy to spoof, sniff and guessed
iSCSI authorization is the only required security method, and this uses only the iQN
Authentication is disabled by default
Authentication is (mostly) only implemented as CHAP
IPSec is difficult to implement

Because iQN’s are manually configured in the iSCSI driver on the client, it is easy to change them. To get access to a LUN that is only protected by a iQN restriction, you can sniff the communication to get the iQN, or guess the iQN as it is often a default string (eg.: iqn.1991-05.com.microsoft.hostname), configure the iscsi driver to use this name and get access to the LUN.

The CHAP protocol is basically the only authentication mechanism that is supported by iSCSI vendors. The protocol allows for other mechanisms like Kerberos. The CHAP protocol is not a protocol know for its strong security on shared networks. The CHAP protocol is vulnerable to dictionary attacks, spoofing, or reflection attacks. Because the security issues with CHAP are well known, the RFC even mentions ways to deal with the limitations of CHAP (http://tools.ietf.org/html/rfc3720#section-8.2.1).

While IPSec could stop or reduce most of the security issues outlined above, it is hard to implement and manage. Therefor not many administrators will feel the need to use it. It should not only be possible to make a secure network, it should also be made easy.

To reduce the risk, and make your iSCSI network as safe as possible, you should do the following:

Enable mutual (incoming/outgoing) authentication
Follow advice to secure CHAP
Enable CRC checksums
Do not only rely on iQN for authorization
Enable IPSec (if performance allows it)

Also vendors/distributors should enable authentication by default, and add other authentication mechanisms to the iSCSI target and initiator software.

References:
http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-Dwivedi-update.pdf
http://en.wikipedia.org/wiki/ISCSI#Authentication
http://weird-hobbes.nl/reports/iSCSI%20security/

In general it is a good idea to configure password aging as part of your password/security policy. In some cases however, this might cause unexpected problems. I’ve seen cases where an expired password prevented a machine from booting. In this specific case this was caused by a service that ran as the user with the expired password. In general you should not run services as a normal user account, but sometimes you just have to deal with things you can’t change. Generally the documentation states that to disable password aging you have to edit the /etc/shadow file, and remove the part where the password age is stored. This is quite error prone. If you do it this way, be sure to use vipw to prevent errors in this critical file. To disable password aging I recommend just using the command to enable it as well:
# chage -m 0 -M 99999 -E -1 username
Check the before and after:
# chage -l username Minimum: 7 Maximum: 90 Warning: 7 Inactive: -1 Last Change: Jun 26, 2009 Password Expires: Sep 24, 2009 Password Inactive: Never Account Expires: Never
After disabling password aging:
# chage -l username Minimum: 0 Maximum: 99999 Warning: 7 Inactive: -1 Last Change: Jun 26, 2009 Password Expires: Never Password Inactive: Never Account Expires: Never

As a note, please only disable password aging when there is no other way to fix the problem.

Yesterday I read an article in the Washington Post about a big security breach at Network Solutions where >500.000 credit and debit cards are stolen. Network Solutions acknowledge this security incident on their site. They claim the cause of this incident was malicious code that was uploaded to a platform supporting their merchants sites. How this was possible and how this could lead to the possible theft of half a million credit card numbers is unclear.

I’ve been involved in several PCI projects to help our customers become PCI compliant as required by the credit card issuing companies. While the need for most of the procedures and measures required by PCI is clear, some seem useless, costly and/or superfluous to implement. But after incidents like these, it is a lot easier to explain customers the point of the measures PCI requires of companies handling credit card data. Basically everything is aimed at protecting the creditcard data, and making sure in case of a security incident all needed audit trails are available to investigate the cause and source of the attack. At least that is how I look at it.

For instance, PCI requires you to audit the integrity of the files present on the system. This should not only include OS files, but also the (web) application code. It is premature to speculate if this ‘malicious code’ could have been detected by running a properly configured host-based IDS on the platform, such as Tripwire or Samhain.

Secondly, PCI requires you to establish roles that have access to a production platform to upload code. Staff members should be part of a role that authorizes them for the access they need to do their job. Access should be restricted to allow only this traffic. Not only should access be locked, but audit trails of all activity should be available upon request. These audit trails not only include who logged in to the systems, but should also include network IDS logs (eg. SNORT), commands that are executed (eg. sudo), output of these commands (eg. rootsh) and the reports of host-based IDS’s. Together these tools should give an auditor a good insight in the activity on a (compromised) server. Also SElinux could be a big help in restricting access. Strangely PCI does not require or advise the use of SElinux, while it does require the use of application level firewalls (eg. mod_security in Apache). But this is a different discussion 🙂

To secure the creditcard data itself, PCI requires that all creditcards should be stored in encrypted from. Manual access to keys to decrypt this data should not be possible. Based on this you can infer that the creditcard data is compromised by some sort of ‘man-in-the-middle’ attack. The malicious code could have intercepted the data after it was decrypted in the webserver, leaving the SSL tunnel, and before it was encrypted and stored in the database. But this is just speculating of course…

I’m looking forward to more details on this incident. I hope this is made publicly available so we can learn from the mistakes that were made.

Rogierm's Blog

Just another blog with technical stuff

Tag Archives: Security

Cronolog with selinux

Ten-thousands credit cards blocked

iSCSI security, how to secure your LUN?

disable password aging

Creditcard data stolen at Network Solutions