Troubleshooters.Com®, Linux Library, and DIY Linux Present:
Using Qemu For Linux DIY
Copyright © 2015 by Steve Litt
See the Troubleshooters.Com Bookstore.
Contents:
Virtual machines (VMs) are a must if you do a lot of Do It Yourself (DIY) and feel the need for speed. Installs go a lot faster on VMs, plus you can install straight off an .iso file instead, which saves a lot of time and fooling around. And, of course, there's the simple fact that switching computers or hard disks is time consuming, and monetarily costly.
I'm not saying VM experimentation is completely authoritative. Things that work in a VM might fail on hardware, and vice versa. Also, networking on a VM can be very complicated, depending on the extent you want to emulate what you'd do with an individual computer. This document discusses only Qemu's default networking, which is very simple.
Note:
Distros that fail with Qemu might succeed with VirtualBox, and vice versa. Try both.
Differences between Virtualbox and Qemu include the following:
Generally speaking, creating and using a Linux hosted, Linux guested Qemu VM occurs in three steps:
This document expresses these three steps using three shellscripts.
This is not a document about Qemu. Qemu is a powerful, flexible, and quite complicated system. In the hands of the most knowledgeable, Qemu can perform miracles. This document doesn't tell you how to optimally use Qemu.
Instead, this document tries to give you one reliable alternative for each activity necessary for Linux distribution DIY. If you want to do things like testing networking, USB or sound, you'll need to either get much more proficiency than this document provides, or, after you see that it's basically working as a host, run the distro on bare metal.
Viewing the code larger:
I've used small font on some of the code in this document. Using small code was the best of a bunch of bad alternatives: Including forcing readers to use horizontal scroll, or kludging in some way of indicating wrapped lines in code. If you're on a desktop and the the code in this document is too small for you to read, please do one of the following:
The biggest DIY benefit of virtual machines is speed. Linux installation is faster on a VM guest than on bare metal, if and only if, the host hardware has hardware VM assist. If not, it's five times slower than bare metal, erasing every possible benefit.
The way you find out whether your host hardware supports hardware VM assist is with one of these commands, depending on whether you have an Intel or an AMD processor:
echo THIS IS FOR INTEL CPUS grep ^flags < /proc/cpuinfo | grep vmx
echo THIS IS FOR AMD CPUS grep ^flags < /proc/cpuinfo | grep svm
If the proper command for your CPU outputs a bunch of text, you have hardware VM assist. If it outputs nothing, your CPU doesn't have hardware VM assist, and this web page will do you no good unless you find other hardware to work with.
As a DIY person, I'd anticipate you'll need knowledge of both Qemu and VirtualBox. There are some Linux setups that would work on one or not the other. Plus the fact that depending on your host OS and distro, you might have access to only one or the other.
I personally go to VirtualBox first, because I've been able to get Linux distros to run on VirtualBox with considerably less experimentation than with Qemu. When I'm DIYing, I like to save my experimentation for the Linux distro, not configuration of the Virtual Machine.
Be careful of VirtualBox and its licensing. The core is GPLv2, but the VirtualBox Guest Additions are a rather benign kind of free beer license that specifies you're using the VirtualBox Guest Additions only for personal use or for business evaluation. I read that to mean I can't base my business's core activities on any processes using VirtualBox Guest Additions. So I'd need to buy a commercial license to run my business on VirtualBox Guest Additions. According to this 2012 notice, the commercial license is reasonable but not negligible, and you need to spend some coin to be eligible for Oracle support.
My finding is that in more cases than not, you need to jump through substantial hoops to make Qemu run a distro with reasonable GUI. Many times Qemu guested distros just hang, unless you jump through those hoops. On the other side of the coin, Qemu has a scripting-friendly CLI interface, and everything about it is Free Software.
Learn both. If you fail with one, you can quickly try the other. Plus, the more you learn about each, the more you learn about Virtual Machines, which helps you on the other. I've written a VirtualBox document similar to this one.
The specifyvm.sh shellscript queries for all information necessary to create a VM image from an .iso file:
#!/bin/sh Copyright (c) 2015 by Steve Litt Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. echo -n "Name of new VM=" read vmname echo "VM name is $vmname" echo disksize="5G" echo -n "Disk size: Append G to number, default 5G=" read junk if test "$junk" != ""; then disksize=$junk;fi disksize=`echo $disksize | tr g G` echo "Disk size is $disksize" echo ramsize="750" echo -n "RAM size in MB, no letter necessary, default 750=" read junk if test "$junk" != ""; then ramsize=$junk;fi echo "RAM size is $ramsize" echo echo -n "ISO to VMize, full path=" read iso echo "ISO path is $iso" echo echo -n "Directory to contain new VM image file=" read vm_dir vm_dir=`echo $vm_dir | sed -e"s+[/[:space:]]*$++"` vm_image="$vm_dir/$vmname.img" echo "VM image at $vm_image" echo ################################################# # WRITE RESULTS TO ENV VAR SHELLSCRIPT ################################################# mkdir -p $vmname ouf=$vmname/envs.sh echo "#!/bin/sh" > $ouf echo "export vmname=$vmname" >> $ouf echo "export disksize=$disksize" >> $ouf echo "export ramsize=$ramsize" >> $ouf echo "export iso=$iso" >> $ouf echo "export vm_image=$vm_image" >> $ouf echo echo "Env script $ouf now contains the following:" cat $ouf
The specifyvm.sh shellscript made it easy for you to specify the VM. The createvm.sh shellscript uses the environment vars set by specifyvm.sh to create a VM and install the Linux from the specified .iso file. Here's the code of the createvm.sh shellscript:
#!/bin/sh Copyright (c) 2015 by Steve Litt Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. dir=$1 . $dir/envs.sh qemu-img create $vm_image $disksize qemu-system-x86_64 -cdrom $iso -hda $vm_image -boot d \ -m $ramsize -localtime --enable-kvm
Once everything is installed, shut down the OS in the normal way, and then re-access the VM with runvm.sh
The runvm.sh shellscript is used to run an already created VM image. Its code follows:
#!/bin/sh Copyright (c) 2015 by Steve Litt Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. dir=$1 . $dir/envs.sh qemu-system-x86_64 -hda $vm_image -boot c \ -m $ramsize -localtime --enable-kvm
This document is an approximation, meant for for conceptual use. It's designed to be used as a sort of block diagram or overview simplification, so there are probably errors on low level details. If you go deep enough, no reference other than the source code will suffice.
But at all the higher levels, I have a feeling you'll find this document very helpful.
Qemu has no facilities for copy and paste, and you certainly need to be able to communicate between host and guest. Imagine transposing long commands. Ugh!
Most Qemu copy and paste substitutes involve shared directories: NFS, SSH, SMB/CIFS, FTP and the like. All are challenging. The least challenging I've found is a directory shared with sshfs, so that's the concentration of this section.
The challenging part of sharing a directory between guest and host is that, in a default network situation, the guest knows the host IP address (default 10.0.2.2), but the host doesn't know the guest's IP address, so the host can't instantiate contact. This is very much like any Network Address Translation situation: Every machine on your local network (192.168.100.0/24, for instance) knows its Internet facing IP address, but nothing from the Internet knows or can contact 192.168.100.3.
Therefore, the guest must reach out to the host for the necessary shared directory. There are many ways of doing this, but the easiest and most straightforward I've seen is using sshfs. Here's how you do it...
You want your Lubuntu 15.04 VM to boot to CLI instead of GUI, so you rename /usr/sbin/lightdm to /usr/sbin/lightdm.org, and reboot. Bad move!
Now, when you boot this VM, it just spins its little dots forever. Oh oh!
If only you could boot it, you could rename /usr/sbin/lightdm.org back to /usr/sbin/lightdm to enable booting the VM. And if only you could boot the VM, you could do the rename. A buried shovel!
I hear a lot on the Internet about loopback mounting your VM image. But invariably, instructions get complicated, and I couldn't get them to work. Then you need to figure out whether your VM is raw, qcow, or something else. Just at the time when you need something easy to get you into a black box, you have to learn a whole new technology. There's got to be a better way.
And there is. It's the same way you bust back into an OS on bare metal, you use System Rescue CD. To set up your System Rescue CD for VM use, download a recent System Rescue CD .iso file and make a note of its name and location. Next, make the following shellscript in the same directory as specifyvm.sh createvm.sh, and runvm.sh. I call mine bustin.sh, and mine looks like the following:
#!/bin/sh ### CHANGE NEXT LINE TO POINT TO YOUR SYS RESC CD ISO ### sysresciso=/scratch/linuxinst/sysrescuecd/systemrescuecd-x86-4.3.1.iso dir=$1 . $dir/envs.sh qemu-system-x86_64 -hda $vm_image -boot d -vga std \ -cdrom $sysresciso -m $ramsize -localtime --enable-kvm
Obviously, change the $sysresciso environment variable to match its name and location on your system. Now just run bustin.sh like this, assuming the name you gave your Lubuntu 15.03 is "lubuntu1504":
sudo ./bustin.sh lubuntu1504
Let System Rescue CD boot as usual, picking the appropriate choices for language, keyboard, etc. Once System Rescue CD has booted to its CLI prompt, run both blkid and lsblk to deduce which device is the VM's root partition. Let's call that partition $root. Now perform the following actions, assuming your VM's root is at /dev/sda1:
root@sysresccd % export root=/dev/sda1 root@sysresccd % mkdir /mnt/root root@sysresccd % mount $root /mnt/root root@sysresccd % chroot /mnt/root bash root@sysresccd:/usr/sbin# root@sysresccd:/usr/sbin# ### Do repair tasks here root@sysresccd:/usr/sbin# root@sysresccd:/usr/sbin# exit exit root@sysresccd/root% umount /mnt/root root@sysresccd/root% poweroff
A few observations about the preceding process:
Qemu makes early stages of Linux DIY much easier and quicker. This document provides a few basic Qemu facilities necessary for early stage DIY:
The preceding three facilities are automated by the following three shellscripts, each of which is in this document:
Networking is complex with Qemu, so this document assumes you selected the default networking, meaning the guest believes the host's IP address to be 10.0.2.2, the host cannot access the guest's network interface, nor can multiple guests access each other.
Qemu currently has no copy and paste. The easiest way I know to compensate for this is to establish a shared directory using an SSH server on the host and sshfs on the guest.
[ Training | Troubleshooters.Com | Email Steve Litt ]