| Virtualization Host Configuration and Guest Installation GuideCommon libvirt errors and troubleshootingThis appendix documents common libvirt-related problems and errors along with instructions for dealing with them. Locate the error on the table below and follow the corresponding link under Solution for detailed troubleshooting information. Table B.1. Common libvirt errors | Error | Description of problem | Solution |
|---|
libvirtd failed to start | The libvirt daemon failed to start. However, there is no information about this error in /var/log/messages. | Section B.1, "libvirtd failed to start" | Cannot read CA certificate | This is one of several errors that occur when the URI fails to connect to the hypervisor. | Section B.2, "The URI failed to connect to the hypervisor" | Failed to connect socket ... : Permission denied | This is one of several errors that occur when the URI fails to connect to the hypervisor. | Section B.2, "The URI failed to connect to the hypervisor" | | Other connectivity errors | These are other errors that occur when the URI fails to connect to the hypervisor. | Section B.2, "The URI failed to connect to the hypervisor" | Internal error guest CPU is not compatible with host CPU | The guest virtual machine cannot be started because the host and guest processors are different. | Section B.3, "The guest virtual machine cannot be started: internal error guest CPU is not compatible with host CPU" | Failed to create domain from vm.xml error: monitor socket did not show up.: Connection refused | The guest virtual machine (or domain) starting fails and returns this error or similar. | Section B.4, "Guest starting fails with error: monitor socket did not show up" | Internal error cannot find character device (null) | This error can occur when attempting to connect a guest's console. It reports that there is no serial console configured for the guest virtual machine. | Section B.5, "Internal error cannot find character device (null)" | No boot device | After building a guest virtual machine from an existing disk image, the guest booting stalls. However, the guest can start successfully using the QEMU command directly. | Section B.6, "Guest virtual machine booting stalls with error: No boot device" | The virtual network "default" has not been started | If the default network (or other locally-created network) is unable to start, any virtual machine configured to use that network for its connectivity will also fail to start. | Section B.7, "Virtual network default has not been started" | | PXE boot (or DHCP) on guest failed | A guest virtual machine starts successfully, but is unable to acquire an IP address from DHCP, boot using the PXE protocol, or both. This is often a result of a long forward delay time set for the bridge, or when the iptables package and kernel do not support checksum mangling rules. | Section B.8, "PXE boot (or DHCP) on guest failed" | | Guest can reach outside network, but cannot reach host when using macvtap interface | A guest can communicate with other guests, but cannot connect to the host machine after being configured to use a macvtap (or type='direct') network interface. This is actually not an error - it is the defined behavior of macvtap. | Section B.9, "Guest can reach outside network, but cannot reach host when using macvtap interface" | Could not add rule to fixup DHCP response checksums on network 'default' | This warning message is almost always harmless, but is often mistakenly seen as evidence of a problem. | Section B.10, "Could not add rule to fixup DHCP response checksums on network 'default'" | Unable to add bridge br0 port vnet0: No such device | This error message or the similar Failed to add tap interface to bridge 'br0': No such device reveal that the bridge device specified in the guest's (or domain's) <interface> definition does not exist. | Section B.11, "Unable to add bridge br0 port vnet0: No such device" | Warning: could not open /dev/net/tun: no virtual network emulation qemu-kvm: -netdev tap,script=/etc/my-qemu-ifup,id=hostnet0: Device 'tap' could not be initialized | The guest virtual machine does not start after configuring a type='ethernet' (or 'generic ethernet') interface in the host system. This error or similar appears either in libvirtd.log, /var/log/libvirt/qemu/name_of_guest.log, or in both. | Section B.12, "Guest is unable to start with error: warning: could not open /dev/net/tun" | Unable to resolve address name_of_host service '49155': Name or service not known | QEMU guest migration fails and this error message appears with an unfamiliar hostname. | Section B.13, "Migration fails with Error: unable to resolve address" | Unable to allow access for disk path /var/lib/libvirt/images/qemu.img: No such file or directory | A guest virtual machine cannot be migrated because libvirt cannot access the disk image(s). | Section B.14, "Migration fails with Unable to allow access for disk path: No such file or directory" | | No guest virtual machines are present when libvirtd is started | The libvirt daemon is successfully started, but no guest virtual machines appear to be present when running virsh list --all. | Section B.15, "No guest virtual machines are present when libvirtd is started" | Unable to connect to server at 'host:16509': Connection refused ... error: failed to connect to the hypervisor | While libvirtd should listen on TCP ports for connections, the connection to the hypervisor fails. | Section B.16, "Unable to connect to server at 'host:16509': Connection refused ... error: failed to connect to the hypervisor" | | Common XML errors | libvirt uses XML documents to store structured data. Several common errors occur with XML documents when they are passed to libvirt through the API. This entry provides instructions for editing guest XML definitions, and details common errors in XML syntax and configuration. | Section B.17, "Common XML errors" |
B.1. libvirtd failed to start- Symptom
The libvirt daemon does not start automatically. Starting the libvirt daemon manually fails as well: # /etc/init.d/libvirtd start* Caching service dependencies ... [ ok ]* Starting libvirtd .../usr/sbin/libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info.* start-stop-daemon: failed to start `/usr/sbin/libvirtd' [ !! ]* ERROR: libvirtd failed to start Moreover, there is not 'more info' about this error in /var/log/messages. - Investigation
Change libvirt's logging in /etc/libvirt/libvirtd.conf by uncommenting the line below. To uncomment the line, open the /etc/libvirt/libvirtd.conf file in a text editor, remove the hash (or #) symbol from the beginning of the following line, and save the change: log_outputs="3:syslog:libvirtd" This line is commented out by default to prevent libvirt from producing excessive log messages. After diagnosing the problem, it is recommended to comment this line again in the /etc/libvirt/libvirtd.conf file. Restart libvirt to determine if this has solved the problem. If libvirtd still does not start successfully, an error similar to the following will be shown in the /var/log/messages file: Feb 6 17:22:09 bart libvirtd: 17576: info : libvirt version: 0.9.9Feb 6 17:22:09 bart libvirtd: 17576: error : virNetTLSContextCheckCertFile:92: Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directoryFeb 6 17:22:09 bart /etc/init.d/libvirtd[17573]: start-stop-daemon: failed to start `/usr/sbin/libvirtd'Feb 6 17:22:09 bart /etc/init.d/libvirtd[17565]: ERROR: libvirtd failed to start The libvirtd man page shows that the missing cacert.pem file is used as TLS authority when libvirt is run in Listen for TCP/IP connections mode. This means the --listen parameter is being passed. - Solution
Configure the libvirt daemon's settings with one of the following methods: Install a CA certificate. For more information on CA certificates and configuring system authentication, refer to the Configuring Authentication chapter in the Red Hat Enterprise Linux 6 Deployment Guide. Do not use TLS; use bare TCP instead. In /etc/libvirt/libvirtd.conf set listen_tls = 0 and listen_tcp = 1. The default values are listen_tls = 1 and listen_tcp = 0. Do not pass the --listen parameter. In /etc/sysconfig/libvirtd.conf change the LIBVIRTD_ARGS variable.
B.2. The URI failed to connect to the hypervisorSeveral different errors can occur when connecting to the server (for example, when running virsh). B.2.1. Cannot read CA certificate- Symptom
When running a command, the following error (or similar) appears: $ virsh -c name_of_uri listerror: Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directoryerror: failed to connect to the hypervisor - Investigation
The error message is misleading about the actual cause. This error can be caused by a variety of factors, such as an incorrectly specified URI, or a connection that is not configured. - Solution
- Incorrectly specified URI
When specifying qemu://system or qemu://session as a connection URI, virsh attempts to connect to hostnames system or session respectively. This is because virsh recognizes the text after the second forward slash as the host. Use three forward slashes to connect to the local host. For example, specifying qemu:///system instructs virsh connect to the system instance of libvirtd on the local host. When a hostname is specified, the QEMU transport defaults to TLS. This results in certificates. - Connection is not configured
The URI is correct (for example, qemu[+tls]://server/system) but the certificates are not set up properly on your machine. For information on configuring TLS, see Setting up libvirt for TLS available from the libvirt website.
B.2.2. Failed to connect socket ... : Permission denied- Symptom
When running a virsh command, the following error (or similar) appears: $ virsh -c qemu:///system listerror: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission deniederror: failed to connect to the hypervisor - Investigation
Without any hostname specified, the connection to QEMU uses UNIX sockets by default. If there is no error running this command as root, the UNIX socket options in /etc/libvirt/libvirtd.conf are likely misconfigured. - Solution
To connect as a non-root user using UNIX sockets, configure the following options in /etc/libvirt/libvirtd.conf: unix_sock_group = <group>unix_sock_ro_perms = <perms>unix_sock_rw_perms = <perms> The user running virsh must be a member of the group specified in the unix_sock_group option.
B.2.3. Other connectivity errors- Unable to connect to server at
server:port: Connection refused The daemon is not running on the server or is configured not to listen, using configuration option listen_tcp or listen_tls. - End of file while reading data: nc: using stream socket: Input/output error
If you specified ssh transport, the daemon is likely not running on the server. Solve this error by verifying that the daemon is running on the server.
B.3. The guest virtual machine cannot be started: internal error guest CPU is not compatible with host CPU- Symptom
Running on an Intel Core i7 processor (which virt-manager refers to as Nehalem, or the older Core 2 Duo, referred to as Penryn), a KVM guest (or domain) is created using virt-manager. After installation, the guest's processor is changed to match the host's CPU. The guest is then unable to start and reports this error: 2012-02-06 17:49:15.985+0000: 20757: error : qemuBuildCpuArgStr:3565 : internal error guest CPU is not compatible with host CPU Additionally, clicking Copy host CPU configuration in virt-manager shows Pentium III instead of Nehalem or Penryn. - Investigation
The /usr/share/libvirt/cpu_map.xml file lists the flags that define each CPU model. The Nehalem and Penryn definitions contain this: <feature name='nx'/> As a result, the NX (or No eXecute) flag needs to be presented to identify the CPU as Nehalem or Penryn. However, in /proc/cpuinfo, this flag is missing. - Solution
Nearly all new BIOSes allow enabling or disabling of the No eXecute bit. However, if disabled, some CPUs do not report this flag and thus libvirt detects a different CPU. Enabling this functionality instructs libvirt to report the correct CPU. Refer to your hardware documentation for further instructions on this subject.
B.4. Guest starting fails with error: monitor socket did not show up- Symptom
The guest virtual machine (or domain) starting fails with this error (or similar): # virsh -c qemu:///system create name_of_guest.xml error: Failed to create domain from name_of_guest.xml error: monitor socket did not show up.: Connection refused - Investigation
This error message shows: libvirt is working; The QEMU process failed to start up; and libvirt quits when trying to connect QEMU or the QEMU agent monitor socket.
To understand the error details, examine the guest log: # cat /var/log/libvirt/qemu/name_of_guest.logLC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc -enable-kvm -m 768 -smp 1,sockets=1,cores=1,threads=1 -name name_of_guest -uuid ebfaadbe-e908-ba92-fdb8-3fa2db557a42 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/name_of_guest.monitor,server,nowait -mon chardev=monitor,mode=readline -no-reboot -boot c -kernel /var/lib/libvirt/boot/vmlinuz -initrd /var/lib/libvirt/boot/initrd.img -append method=http://www.example.com/pub/product/release/version/x86_64/os/ -drive file=/var/lib/libvirt/images/name_of_guest.img,if=none,id=drive-ide0-0-0,boot=on -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -device virtio-net-pci,vlan=0,id=net0,mac=52:40:00:f4:f1:0a,bus=pci.0,addr=0x4 -net tap,fd=42,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-gb -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 char device redirected to /dev/pts/1qemu: could not load kernel '/var/lib/libvirt/boot/vmlinuz':Permission denied - Solution
The guest log contains the details needed to fix the error. If a host is shut down while the guest is still running a libvirt version prior to 0.9.5, the libvirt-guest's init script attempts to perform a managed save of the guest. If the managed save was incomplete (for example, due to loss of power before the managed save image was flushed to disk), the save image is corrupted and will not be loaded by QEMU. The older version of libvirt does not recognize the corruption, making the problem perpetual. In this case, the guest log will show an attempt to use -incoming as one of its arguments, meaning that libvirt is trying to start QEMU by migrating in the saved state file. This problem can be fixed by running virsh managedsave-remove name_of_guest to remove the corrupted managed save image. Newer versions of libvirt take steps to avoid the corruption in the first place, as well as adding virsh start --force-boot name_of_guest to bypass any managed save image.
B.5. Internal error cannot find character device (null)- Symptom
This error message appears when attempting to connect to a guest virtual machine's console: # virsh console test2 Connected to domain test2 Escape character is ^] error: internal error cannot find character device (null) - Investigation
This error message shows that there is no serial console configured for the guest virtual machine. - Solution
Set up a serial console in the guest's XML file. Procedure B.1. Setting up a serial console in the guest's XML Add the following XML to the guest virtual machine's XML using virsh edit: <serial type='pty'> <target port='0'/></serial><console type='pty'> <target type='serial' port='0'/></console> Set up the console in the guest kernel command line. To do this, either log in to the guest virtual machine to edit the /boot/grub/grub.conf file directly, or use the virt-edit command line tool. Add the following to the guest kernel command line: console=ttyS0,115200 Run the followings command: # virsh start vm && virsh console vm
B.6. Guest virtual machine booting stalls with error: No boot device- Symptom
After building a guest virtual machine from an existing disk image, the guest booting stalls with the error message No boot device. However, the guest virtual machine can start successfully using the QEMU command directly. - Investigation
The disk's bus type is not specified in the command for importing the existing disk image: # virt-install \--connect qemu:///system \--ram 2048 -n rhel_64 \--os-type=linux --os-variant=rhel5 \--disk path=/root/RHEL-Server-5.8-64-virtio.qcow2,device=disk,format=qcow2 \--vcpus=2 --graphics spice --noautoconsole --import However, the command line used to boot up the guest virtual machine using QEMU directly shows that it uses virtio for its bus type: # ps -ef | grep qemu/usr/libexec/qemu-kvm -monitor stdio -drive file=/root/RHEL-Server-5.8-32-virtio.qcow2,index=0,if=virtio,media=disk,cache=none,format=qcow2 -net nic,vlan=0,model=rtl8139,macaddr=00:30:91:aa:04:74 -net tap,vlan=0,script=/etc/qemu-ifup,downscript=no -m 2048 -smp 2,cores=1,threads=1,sockets=2 -cpu qemu64,+sse2 -soundhw ac97 -rtc-td-hack -M rhel5.6.0 -usbdevice tablet -vnc :10 -boot c -no-kvm-pit-reinjection Note the bus= in the guest's XML generated by libvirt for the imported guest: <domain type='qemu'> <name>rhel_64</name> <uuid>6cd34d52-59e3-5a42-29e4-1d173759f3e7</uuid> <memory>2097152</memory> <currentMemory>2097152</currentMemory> <vcpu>2</vcpu> <os> <type arch='x86_64' machine='rhel5.4.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/root/RHEL-Server-5.8-64-virtio.qcow2'/> <emphasis role="bold"><target dev='hda' bus='ide'/></emphasis> <address type='drive' controller='0' bus='0' unit='0'/> </disk> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='54:52:00:08:3e:8c'/> <source bridge='br0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target port='0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/> <video> <model type='cirrus' vram='9216' heads='1'/> </video> </devices> </domain> The bus type for the disk is set as ide, which is the default value set by libvirt. This is the incorrect bus type, and has caused the unsuccessful boot for the imported guest. - Solution
Procedure B.2. Correcting the disk bus type Undefine the imported guest, then re-import it with bus=virtio and the following: # virsh destroy rhel_64# virsh undefine rhel_64# virt-install \--connect qemu:///system \--ram 1024 -n rhel_64 -r 2048 \--os-type=linux --os-variant=rhel5 \--disk path=/root/RHEL-Server-5.8-64-virtio.qcow2,device=disk,bus=virtio,format=qcow2 \ --vcpus=2 --graphics spice --noautoconsole --import Edit the imported guest's XML using virsh edit and correct the disk bus type.
B.7. Virtual network default has not been started- Symptom
Normally, the configuration for a virtual network named default is installed as part of the libvirt package, and is configured to autostart when libvirtd is started. If the default network (or any other locally-created network) is unable to start, any virtual machine configured to use that network for its connectivity will also fail to start, resulting in this error message: Virtual network default has not been started - Investigation
One of the most common reasons for a libvirt virtual network's failure to start is that the dnsmasq instance required to serve DHCP and DNS requests from clients on that network has failed to start. To determine if this is the cause, run virsh net-start default from a root shell to start the default virtual network. If this action does not successfully start the virtual network, open /var/log/libvirt/libvirtd.log to view the complete error log message. If a message similar to the following appears, the problem is likely a systemwide dnsmasq instance that is already listening on libvirt's bridge, and is preventing libvirt's own dnsmasq instance from doing so. The most important parts to note in the error message are dnsmasq and exit status 2: Could not start virtual network default: internal errorChild process (/usr/sbin/dnsmasq --strict-order --bind-interfaces--pid-file=/var/run/libvirt/network/default.pid --conf-file=--except-interface lo --listen-address 192.168.122.1--dhcp-range 192.168.122.2,192.168.122.254--dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases--dhcp-lease-max=253 --dhcp-no-override) status unexpected: exit status 2 - Solution
If the machine is not using dnsmasq to serve DHCP for the physical network, disable dnsmasq completely. If it is necessary to run dnsmasq to serve DHCP for the physical network, edit the /etc/dnsmasq.conf file. Add or uncomment the first line, as well as one of the two lines following that line. Do not add or uncomment all three lines: bind-interfacesinterface=name_of_physical_interfacelisten-address=chosen_IP_address After making this change and saving the file, restart the systemwide dnsmasq service. Next, start the default network with the virsh net-start default command. Start the virtual machines.
B.8. PXE boot (or DHCP) on guest failed- Symptom
A guest virtual machine starts successfully, but is then either unable to acquire an IP address from DHCP or boot using the PXE protocol, or both. There are two common causes of this error: having a long forward delay time set for the bridge, and when the iptables package and kernel do not support checksum mangling rules. - Long forward delay time on bridge
- Investigation
This is the most common cause of this error. If the guest network interface is connecting to a bridge device that has STP (Spanning Tree Protocol) enabled, as well as a long forward delay set, the bridge will not forward network packets from the guest virtual machine onto the bridge until at least that number of forward delay seconds have elapsed since the guest connected to the bridge. This delay allows the bridge time to watch traffic from the interface and determine the MAC addresses behind it, and prevent forwarding loops in the network topology. If the forward delay is longer than the timeout of the guest's PXE or DHCP client, then the client's operation will fail, and the guest will either fail to boot (in the case of PXE) or fail to acquire an IP address (in the case of DHCP). - Solution
If this is the case, change the forward delay on the bridge to 0, disable STP on the bridge, or both. This solution applies only if the bridge is not used to connect multiple networks, but just to connect multiple endpoints to a single network (the most common use case for bridges used by libvirt). If the guest has interfaces connecting to a libvirt-managed virtual network, edit the definition for the network, and restart it. For example, edit the default network with the following command: # virsh net-edit default Add the following attributes to the <bridge> element: <name_of_bridge='virbr0' delay='0' stp='on'/> delay='0' and stp='on' are the default settings for virtual networks, so this step is only necessary if the configuration has been modified from the default.
If the guest interface is connected to a host bridge that was configured outside of libvirt, change the delay setting. Add or edit the following lines in the /etc/sysconfig/network-scripts/ifcfg-name_of_bridge file to turn STP on with a 0 second delay: STP=onDELAY=0 After changing the configuration file, restart the bridge device: /sbin/ifdown name_of_bridge/sbin/ifup name_of_bridge If name_of_bridge is not the root bridge in the network, that bridge's delay will eventually reset to the delay time configured for the root bridge. In this case, the only solution is to disable STP completely on name_of_bridge.
- The iptables package and kernel do not support checksum mangling rules
- Investigation
This message is only a problem if all four of the following conditions are true: The guest is using virtio network devices. If so, the configuration file will contain model type='virtio' The host has the vhost-net module loaded. This is true if ls /dev/vhost-net The guest is attempting to get an IP address from a DHCP server that is running directly on the host. The iptables version on the host is older than 1.4.10. iptables 1.4.10 was the first version to add the libxt_CHECKSUM extension. This is the case if the following message appears in the libvirtd logs: warning: Could not add rule to fixup DHCP response checksums on network defaultwarning: May need to update iptables package and kernel to support CHECKSUM rule. Unless all of the other three conditions in this list are also true, the above warning message can be disregarded, and is not an indicator of any other problems.
When these conditions occur, UDP packets sent from the host to the guest have uncomputed checksums. This makes the host's UDP packets seem invalid to the guest's network stack. - Solution
To solve this problem, invalidate any of the four points above. The best solution is to update the host iptables and kernel to iptables-1.4.10 or newer where possible. Otherwise, the most specific fix is to disable the vhost-net driver for this particular guest. To do this, edit the guest configuration with this command: virsh edit name_of_guest Change or add a <driver> line to the <interface> section: <interface type='network'> <model type='virtio'/> <driver name='qemu'/> ...</interface> Save the changes, shut down the guest, and then restart it. If this problem is still not resolved, the issue may be due to a conflict between firewalld and the default libvirt network. To fix this, stop firewalld with the service firewalld stop command, then restart libvirt with the service libvirtd restart command.
B.9. Guest can reach outside network, but cannot reach host when using macvtap interface- Symptom
A guest virtual machine can communicate with other guests, but cannot connect to the host machine after being configured to use a macvtap (also known as type='direct') network interface. - Investigation
Even when not connecting to a Virtual Ethernet Port Aggregator (VEPA) or VN-Link capable switch, macvtap interfaces can be useful. Setting the mode of such an interface to bridge allows the guest to be directly connected to the physical network in a very simple manner without the setup issues (or NetworkManager incompatibility) that can accompany the use of a traditional host bridge device. However, when a guest virtual machine is configured to use a type='direct' network interface such as macvtap, despite having the ability to communicate with other guests and other external hosts on the network, the guest cannot communicate with its own host. This situation is actually not an error - it is the defined behavior of macvtap. Due to the way in which the host's physical Ethernet is attached to the macvtap bridge, traffic into that bridge from the guests that is forwarded to the physical interface cannot be bounced back up to the host's IP stack. Additionally, traffic from the host's IP stack that is sent to the physical interface cannot be bounced back up to the macvtap bridge for forwarding to the guests. - Solution
Use libvirt to create an isolated network, and create a second interface for each guest virtual machine that is connected to this network. The host and guests can then directly communicate over this isolated network, while also maintaining compatibility with NetworkManager. Procedure B.3. Creating an isolated network with libvirt Add and save the following XML in the /tmp/isolated.xml file. If the 192.168.254.0/24 network is already in use elsewhere on your network, you can choose a different network. <network> <name>isolated</name> <ip address='192.168.254.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.254.2' end='192.168.254.254' /> </dhcp> </ip></network> Create the network with this command: virsh net-define /tmp/isolated.xml Set the network to autostart with the virsh net-autostart isolated command. Start the network with the virsh net-start isolated command. Using virsh edit name_of_guest, edit the configuration of each guest that uses macvtap for its network connection and add a new <interface> in the <devices> section similar to the following (note the <model type='virtio'/> line is optional to include): <interface type='network'> <source network='isolated'/> <model type='virtio'/></interface> Shut down, then restart each of these guests.
The guests are now able to reach the host at the address 192.168.254.1, and the host will be able to reach the guests at the IP address they acquired from DHCP (alternatively, you can manually configure the IP addresses for the guests). Since this new network is isolated to only the host and guests, all other communication from the guests will use the macvtap interface.
B.10. Could not add rule to fixup DHCP response checksums on network 'default'- Symptom
This message appears: Could not add rule to fixup DHCP response checksums on network 'default' - Investigation
Although this message appears to be evidence of an error, it is almost always harmless. - Solution
Unless the problem you are experiencing is that the guest virtual machines are unable to acquire IP addresses through DHCP, this message can be ignored.
B.11. Unable to add bridge br0 port vnet0: No such device- Symptom
The following error message appears: Unable to add bridge name_of_bridge port vnet0: No such device For example, if the bridge name is br0, the error message will appear as: Unable to add bridge br0 port vnet0: No such device In libvirt versions 0.9.6 and earlier, the same error appears as: Failed to add tap interface to bridge name_of_bridge: No such device Or for example, if the bridge is named br0: Failed to add tap interface to bridge 'br0': No such device - Investigation
Both error messages reveal that the bridge device specified in the guest's (or domain's) <interface> definition does not exist. To verify the bridge device listed in the error message does not exist, use the shell command ifconfig br0. A message similar to this confirms the host has no bridge by that name: br0: error fetching interface information: Device not found
If this is the case, continue to the solution. However, if the resulting message is similar to the following, the issue exists elsewhere: br0 Link encap:Ethernet HWaddr 00:00:5A:11:70:48 inet addr:10.22.1.5 Bcast:10.255.255.255 Mask:255.0.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:249841 errors:0 dropped:0 overruns:0 frame:0 TX packets:281948 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:106327234 (101.4 MiB) TX bytes:21182634 (20.2 MiB)
- Solution
- Edit the existing bridge or create a new bridge with
virsh Use virsh to either edit the settings of an existing bridge or network, or to add the bridge device to the host system configuration. - Edit the existing bridge settings using
virsh Use virsh edit name_of_guest to change the <interface> definition to use a bridge or network that already exists. For example, change type='bridge' to type='network', and <source bridge='br0'/> to <source network='default'/>. - Create a host bridge using
virsh For libvirt version 0.9.8 and later, a bridge device can be created with the virsh iface-bridge command. This will create a bridge device br0 with eth0, the physical network interface which is set as part of a bridge, attached: virsh iface-bridge eth0 br0 Optional: If desired, remove this bridge and restore the original eth0 configuration with this command: virsh iface-unbridge br0
- Create a host bridge manually
B.12. Guest is unable to start with error: warning: could not open /dev/net/tun- Symptom
The guest virtual machine does not start after configuring a type='ethernet' (also known as 'generic ethernet') interface in the host system. An error appears either in libvirtd.log, /var/log/libvirt/qemu/name_of_guest.log, or in both, similar to the below message: warning: could not open /dev/net/tun: no virtual network emulation qemu-kvm: -netdev tap,script=/etc/my-qemu-ifup,id=hostnet0: Device 'tap' could not be initialized - Investigation
Use of the generic ethernet interface type (<interface type='ethernet'>) is discouraged, because using it requires lowering the level of host protection against potential security flaws in QEMU and its guests. However, it is sometimes necessary to use this type of interface to take advantage of some other facility that is not yet supported directly in libvirt. For example, openvswitch was not supported in libvirt until libvirt-0.9.11, so in older versions of libvirt, <interface type='ethernet'> was the only way to connect a guest to an openvswitch bridge. However, if you configure a <interface type='ethernet'> interface without making any other changes to the host system, the guest virtual machine will not start successfully. The reason for this failure is that for this type of interface, a script called by QEMU needs to manipulate the tap device. However, with type='ethernet' configured, in an attempt to lock down QEMU, libvirt and SELinux have put in place several checks to prevent this. (Normally, libvirt performs all of the tap device creation and manipulation, and passes an open file descriptor for the tap device to QEMU.) - Solution
Reconfigure the host system to be compatible with the generic ethernet interface. Procedure B.4. Reconfiguring the host system to use the generic ethernet interface Set SELinux to permissive by configuring SELINUX=permissive in /etc/selinux/config: # This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=permissive# SELINUXTYPE= can take one of these two values:# targeted - Targeted processes are protected,# mls - Multi Level Security protection.SELINUXTYPE=targeted From a root shell, run the command setenforce permissive. In /etc/libvirt/qemu.conf add or edit the following lines: clear_emulator_capabilities = 0 user = "root" group = "root" cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu","/dev/rtc", "/dev/hpet", "/dev/net/tun", Restart libvirtd.
Since each of these steps significantly decreases the host's security protections against QEMU guest domains, this configuration should only be used if there is no alternative to using <interface type='ethernet'>.
For more information on SELinux, refer to the Red Hat Enterprise Linux 6 Security-Enhanced Linux User Guide. B.13. Migration fails with Error: unable to resolve address- Symptom
QEMU guest migration fails and this error message appears: # virsh migrate qemu qemu+tcp://192.168.122.12/system error: Unable to resolve address name_of_host service '49155': Name or service not known For example, if the destination hostname is "newyork", the error message will appear as: # virsh migrate qemu qemu+tcp://192.168.122.12/systemerror: Unable to resolve address 'newyork' service '49155': Name or service not known However, this error looks strange as we did not use "newyork" hostname anywhere. - Investigation
During migration, libvirtd running on the destination host creates a URI from an address and port where it expects to receive migration data and sends it back to libvirtd running on the source host. In this case, the destination host (192.168.122.12) has its name set to 'newyork'. For some reason, libvirtd running on that host is unable to resolve the name to an IP address that could be sent back and still be useful. For this reason, it returned the 'newyork' hostname hoping the source libvirtd would be more successful with resolving the name. This can happen if DNS is not properly configured or /etc/hosts has the hostname associated with local loopback address (127.0.0.1). Note that the address used for migration data cannot be automatically determined from the address used for connecting to destination libvirtd (for example, from qemu+tcp://192.168.122.12/system). This is because to communicate with the destination libvirtd, the source libvirtd may need to use network infrastructure different from that which virsh (possibly running on a separate machine) requires. - Solution
The best solution is to configure DNS correctly so that all hosts involved in migration are able to resolve all host names. If DNS cannot be configured to do this, a list of every host used for migration can be added manually to the /etc/hosts file on each of the hosts. However, it is difficult to keep such lists consistent in a dynamic environment. If the host names cannot be made resolvable by any means, virsh migrate supports specifying the migration host: # virsh migrate qemu qemu+tcp://192.168.122.12/system tcp://192.168.122.12 Destination libvirtd will take the tcp://192.168.122.12 URI and append an automatically generated port number. If this is not desirable (because of firewall configuration, for example), the port number can be specified in this command: # virsh migrate qemu qemu+tcp://192.168.122.12/system tcp://192.168.122.12:12345 Another option is to use tunnelled migration. Tunnelled migration does not create a separate connection for migration data, but instead tunnels the data through the connection used for communication with destination libvirtd (for example, qemu+tcp://192.168.122.12/system): # virsh migrate qemu qemu+tcp://192.168.122.12/system --p2p --tunnelled
B.14. Migration fails with Unable to allow access for disk path: No such file or directory- Symptom
A guest virtual machine (or domain) cannot be migrated because libvirt cannot access the disk image(s): # virsh migrate qemu qemu+tcp://name_of_host/systemerror: Unable to allow access for disk path /var/lib/libvirt/images/qemu.img: No such file or directory For example, if the destination hostname is "newyork", the error message will appear as: # virsh migrate qemu qemu+tcp://newyork/systemerror: Unable to allow access for disk path /var/lib/libvirt/images/qemu.img: No such file or directory - Investigation
By default, migration only transfers the in-memory state of a running guest (such as memory or CPU state). Although disk images are not transferred during migration, they need to remain accessible at the same path by both hosts. - Solution
Set up and mount shared storage at the same location on both hosts. The simplest way to do this is to use NFS: Procedure B.5. Setting up shared storage Set up an NFS server on a host serving as shared storage. The NFS server can be one of the hosts involved in the migration, as long as all hosts involved are accessing the shared storage through NFS. # mkdir -p /exports/images# cat >>/etc/exports <<EOF/exports/images 192.168.122.0/24(rw,no_root_squash)EOF Mount the exported directory at a common location on all hosts running libvirt. For example, if the IP address of the NFS server is 192.168.122.1, mount the directory with the following commands: # cat >>/etc/fstab <<EOF192.168.122.1:/exports/images /var/lib/libvirt/images nfs auto 0 0EOF# mount /var/lib/libvirt/images
It is not possible to export a local directory from one host using NFS and mount it at the same path on another host - the directory used for storing disk images must be mounted from shared storage on both hosts. If this is not configured correctly, the guest virtual machine may lose access to its disk images during migration, because the source host's libvirt daemon may change the owner, permissions, and SELinux labels on the disk images after it successfully migrates the guest to its destination. If libvirt detects that the disk images are mounted from a shared storage location, it will not make these changes.
B.15. No guest virtual machines are present when libvirtd is started- Symptom
The libvirt daemon is successfully started, but no guest virtual machines appear to be present. # virsh list --all Id Name State----------------------------------------------------# - Investigation
There are various possible causes of this problem. Performing these tests will help to determine the cause of this situation: - Verify KVM kernel modules
Verify that KVM kernel modules are inserted in the kernel: # lsmod | grep kvmkvm_intel 121346 0kvm 328927 1 kvm_intel If you are using an AMD machine, verify the kvm_amd kernel modules are inserted in the kernel instead, using the similar command lsmod | grep kvm_amd in the root shell. If the modules are not present, insert them using the modprobe <modulename> command. Although it is uncommon, KVM virtualization support may be compiled into the kernel. In this case, modules are not needed. - Verify virtualization extensions
Verify that virtualization extensions are supported and enabled on the host: # egrep "(vmx|svm)" /proc/cpuinfoflags: fpu vme de pse tsc ... svm ... skinit wdt npt lbrv svm_lock nrip_saveflags: fpu vme de pse tsc ... svm ... skinit wdt npt lbrv svm_lock nrip_save Enable virtualization extensions in your hardware's firmware configuration within the BIOS setup. Refer to your hardware documentation for further details on this. - Verify client URI configuration
Verify that the URI of the client is configured as desired: # virsh urivbox:///system For example, this message shows the URI is connected to the VirtualBox hypervisor, not QEMU, and reveals a configuration error for a URI that is otherwise set to connect to a QEMU hypervisor. If the URI was correctly connecting to QEMU, the same message would appear instead as: # virsh uriqemu:///system This situation occurs when there are other hypervisors present, which libvirt may speak to by default.
- Solution
After performing these tests, use the following command to view a list of guest virtual machines: # virsh list --all
B.16. Unable to connect to server at 'host:16509': Connection refused ... error: failed to connect to the hypervisor- Symptom
While libvirtd should listen on TCP ports for connections, the connections fail: # virsh -c qemu+tcp://host/systemerror: unable to connect to server at 'host:16509': Connection refusederror: failed to connect to the hypervisor The libvirt daemon is not listening on TCP ports even after changing configuration in /etc/libvirt/libvirtd.conf: # grep listen_ /etc/libvirt/libvirtd.conflisten_tls = 1listen_tcp = 1listen_addr = "0.0.0.0" However, the TCP ports for libvirt are still not open after changing configuration: # netstat -lntp | grep libvirtd# - Investigation
The libvirt daemon was started without the --listen option. Verify this by running this command: # ps aux | grep libvirtdroot 27314 0.0 0.0 1000920 18304 ? Sl Feb16 1:19 libvirtd --daemon The output does not contain the --listen option. - Solution
Start the daemon with the --listen option. To do this, modify the /etc/sysconfig/libvirtd file and uncomment the following line: #LIBVIRTD_ARGS="--listen" Then restart the libvirtd service with this command: # /etc/init.d/libvirtd restart
The libvirt tool uses XML documents to store structured data. A variety of common errors occur with XML documents when they are passed to libvirt through the API. Several common XML errors - including misformatted XML, inappropriate values, and missing elements - are detailed below. B.17.1. Editing domain definitionAlthough it is not recommended, it is sometimes necessary to edit a guest virtual machine's (or a domain's) XML file manually. To access the guest's XML for editing, use the following command: # virsh edit name_of_guest.xml This command opens the file in a text editor with the current definition of the guest virtual machine. After finishing the edits and saving the changes, the XML is reloaded and parsed by libvirt. If the XML is correct, the following message is displayed: # virsh edit name_of_guest.xmlDomain name_of_guest.xml XML configuration edited. When using the edit command in virsh to edit an XML document, save all changes before exiting the editor. After saving the XML file, use the xmllint command to validate that the XML is well-formed, or the virt-xml-validate command to check for usage problems: # xmllint --noout config.xml # virt-xml-validate config.xml If no errors are returned, the XML description is well-formed and matches the libvirt schema. While the schema does not catch all constraints, fixing any reported errors will further troubleshooting. - XML documents stored by libvirt
These documents contain definitions of states and configurations for the guests. These documents are automatically generated and should not be edited manually. Errors in these documents contain the file name of the broken document. The file name is valid only on the host machine defined by the URI, which may refer to the machine the command was run on. Errors in files created by libvirt are rare. However, one possible source of these errors is a downgrade of libvirt - while newer versions of libvirt can always read XML generated by older versions, older versions of libvirt may be confused by XML elements added in a newer version.
B.17.2. XML syntax errorsSyntax errors are caught by the XML parser. The error message contains information for identifying the problem. This example error message from the XML parser consists of three lines - the first line denotes the error message, and the two following lines contain the context and location of the XML code containing the error. The third line contains an indicator showing approximately where the error lies on the line above it: error: (name_of_guest.xml):6: StartTag: invalid element name<vcpu>2</vcpu><-----------------^ - Information contained in this message:
- (
name_of_guest.xml) This is the file name of the document that contains the error. File names in parentheses are symbolic names to describe XML documents parsed from memory, and do not directly correspond to files on disk. File names that are not contained in parentheses are local files that reside on the target of the connection. - 6
This is the line number in the XML file that contains the error. - StartTag: invalid element name
This is the error message from the libxml2 parser, which describes the specific XML error.
B.17.2.1. Stray < in the document- Symptom
The following error occurs: error: (name_of_guest.xml):6: StartTag: invalid element name<vcpu>2</vcpu><-----------------^ - Investigation
This error message shows that the parser expects a new element name after the < symbol on line 6 of a guest's XML file. Ensure line number display is enabled in your text editor. Open the XML file, and locate the text on line 6: <domain type='kvm'> <name>name_of_guest</name><memory>524288</memory><vcpu>2</vcpu>< This snippet of a guest's XML file contains an extra < in the document: - Solution
Remove the extra < or finish the new element.
B.17.2.2. Unterminated attribute- Symptom
The following error occurs: error: (name_of_guest.xml):2: Unescaped '<' not allowed in attributes values<name>name_of_guest</name>--^ - Investigation
This snippet of a guest's XML file contains an unterminated element attribute value: <domain type='kvm><name>name_of_guest</name> In this case, 'kvm' is missing a second quotation mark. Strings of attribute values, such as quotation marks and apostrophes, must be opened and closed, similar to XML start and end tags. - Solution
Correctly open and close all attribute value strings.
B.17.2.3. Opening and ending tag mismatch- Symptom
The following error occurs: error: (name_of_guest.xml):61: Opening and ending tag mismatch: clock line 16 and domain</domain>---------^ - Investigation
The error message above contains three clues to identify the offending tag: The message following the last colon, clock line 16 and domain, reveals that <clock> contains a mismatched tag on line 16 of the document. The last hint is the pointer in the context part of the message, which identifies the second offending tag. Unpaired tags must be closed with />. The following snippet does not follow this rule and has produced the error message shown above: <domain type='kvm'> ... <clock offset='utc'> This error is caused by mismatched XML tags in the file. Every XML tag must have a matching start and end tag. - Other examples of mismatched XML tags
The following examples produce similar error messages and show variations of mismatched XML tags. This snippet contains an unended pair tag for <features>: <domain type='kvm'> ... <features> <acpi/> <pae/> ... </domain> This snippet contains an end tag (</name>) without a corresponding start tag: <domain type='kvm'> </name> ...</domain>
- Solution
Ensure all XML tags start and end correctly.
B.17.2.4. Typographical errors in tags- Symptom
The following error message appears: error: (name_of_guest.xml):1: Specification mandate value for attribute ty<domain ty pe='kvm'>-----------^ - Investigation
XML errors are easily caused by a simple typographical error. This error message highlights the XML error - in this case, an extra white space within the word type - with a pointer. <domain ty pe='kvm'> These XML examples will not parse correctly because of typographical errors such as a missing special character, or an additional character: <domain type 'kvm'> <dom#ain type='kvm'> - Solution
To identify the problematic tag, read the error message for the context of the file, and locate the error with the pointer. Correct the XML and save the changes.
B.17.3. Logic and configuration errorsA well-formatted XML document can contain errors that are correct in syntax but libvirt cannot parse. Many of these errors exist, with two of the most common cases outlined below. B.17.3.1. Vanishing parts- Symptom
Parts of the change you have made do not show up and have no effect after editing or defining the domain. The define or edit command works, but when dumping the XML once again, the change disappears. - Investigation
This error likely results from a broken construct or syntax that libvirt does not parse. The libvirt tool will generally only look for constructs it knows but ignore everything else, resulting in some of the XML changes vanishing after libvirt parses the input. - Solution
Validate the XML input before passing it to the edit or define commands. The libvirt developers maintain a set of XML schemas bundled with libvirt which define the majority of the constructs allowed in XML documents used by libvirt. Validate libvirt XML files using the following command: # virt-xml-validate libvirt.xml If this command passes, libvirt will likely understand all constructs from your XML, except if the schemas cannot detect options which are valid only for a given hypervisor. Any XML generated by libvirt as a result of a virsh dump command, for example, should validate without error.
B.17.3.2. Incorrect drive device type- Symptom
The definition of the source image for the CD-ROM virtual drive is not present, despite being added: # virsh dumpxml domain<domain type='kvm'> ... <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdc' bus='ide'/> <readonly/> </disk> ...</domain> - Solution
Correct the XML by adding the missing <source> parameter as follows: <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/path/to/image.iso'/> <target dev='hdc' bus='ide'/> <readonly/></disk> A type='block' disk device expects that the source is a physical device. To use the disk with an image file, use type='file' instead.
|
| |
