After a yum upgrade of one of our CentOS 5 Xen server, xend would not start properly. The logs contained the following error messages below.
xend-debug.log:

Xend started at Wed Aug 26 18:15:57 2009.
sysctl operation failed -- need to rebuild the user-space tool set?
Exception starting xend: (13, 'Permission denied')

xend.log

[2009-08-26 18:15:57 3310] ERROR (SrvDaemon:347) Exception starting xend ((13, 'Permission denied'))Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line 339, in run servers = SrvServer.create() File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvServer.py", line 251, in create root.putChild('xend', SrvRoot()) File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__ self.get(name) File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 82, in get val = val.getobj() File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 52, in getobj self.obj = klassobj() File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__ self.xn = XendNode.instance()
File "/usr/lib/python2.4/site-packages/xen/xend/XendNode.py", line 752, in instance
inst = XendNode()
File "/usr/lib/python2.4/site-packages/xen/xend/XendNode.py", line 87, in __init__
self.other_config["xen_pagesize"] = self.xeninfo_dict()["xen_pagesize"]
File "/usr/lib/python2.4/site-packages/xen/xend/XendNode.py", line 741, in xeninfo_dict
return dict(self.xeninfo())
File "/usr/lib/python2.4/site-packages/xen/xend/XendNode.py", line 685, in xeninfo
info['xen_scheduler'] = self.xenschedinfo()
File "/usr/lib/python2.4/site-packages/xen/xend/XendNode.py", line 675, in xenschedinfo
sched_id = self.xc.sched_id_get()
Error: (13, 'Permission denied')

After some investigation this was quite easy to solve. The yum upgrade updated the kernel and modified the grub.conf. So after the reboot, the new xen kernel booted. However, this kernel did not match the xen tools installed. This is easily fixed by changing the grub.conf to boot the correct xen kernel. See the examples below for the exact change.

The grub.conf after the yum update that caused the problem:

title CentOS (2.6.18-128.7.1.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.7.1.el5
module /vmlinuz-2.6.18-128.7.1.el5xen ro root=/dev/VolGroup00/LogVol00
module /initrd-2.6.18-128.7.1.el5xen.img

The changed grub.conf after the yum updated that fixed the problem:

title CentOS (2.6.18-128.7.1.el5xen)
root (hd0,0)
kernel /xen.gz-3.3.1
module /vmlinuz-2.6.18-128.7.1.el5xen ro root=/dev/VolGroup00/LogVol00
module /initrd-2.6.18-128.7.1.el5xen.img

Last few weeks I’ve been experimenting with OpenQRM (http://www.openqrm.org). The initial installation did not cause any big problems. After logging in to the interface and trying to enable plugins, this just timed-out. I tried to manually start the plugins on the console (eg. /usr/lib/openqrm/plugins/dns/etc/init.d/openqrm-plugin-dns  start). This command started the plugin without any problem. Next I tried to configure select a volume group in the Plugin section (Storage -> LVM-storage-> Volume Groups). This also timed out with the error below in the http errorlog:

This error was caused by SElinux. After I disabled SElinux and rebooted the server, all functions of the OpenQRM interface worked fine, including enabling plugins and displaying volumegroups.


[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fopen(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 103

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fwrite(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 104

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fclose(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 105

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  unlink(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.unlink'>function.unlink</a>]: No such file or directory in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 1547

[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fopen(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 103
[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fwrite(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 104
[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  fclose(): supplied argument is not a valid stream resource in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 105
[Wed Jul 15 14:35:26 2009] [error] [client 192.168.42.235] PHP Warning:  unlink(/usr/lib/openqrm/web/action/cloud-conf/cloud-monitor.lock) [<a href='function.unlink'>function.unlink</a>]: No such file or directory in /usr/lib/openqrm/plugins/cloud/web/openqrm-cloud-monitor-hook.php on line 1547