<div id="text-table-of-contents">
<ul>
<li><a href="#sec-1">1. Introduction</a></li>
<li><a href="#sec-2">2. Setting up PCI passthrough</a>
<ul>
<li><a href="#sec-2-1">2.1. Setup machine for virtualization</a></li>
<li><a href="#sec-2-2">2.2. Adding Devices for PCI passthrough</a></li>
<li><a href="#sec-2-3">2.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>
</ul>
</li>
<li><a href="#sec-3">3. Setting up the virtual machines</a>
<ul>
<li><a href="#sec-3-1">3.1. Creating a storage pool</a></li>
<li><a href="#sec-3-2">3.2. Creating a volume</a></li>
<li><a href="#sec-3-3">3.3. Networking</a></li>
<li><a href="#sec-3-4">3.4. Installing an OS</a></li>
<li><a href="#sec-3-5">3.5. Accessing the VMs</a></li>
<li><a href="#sec-3-6">3.6. Cloning the VMs</a></li>
</ul>
</li>
<li><a href="#sec-4">4. Conclusion</a></li>
</ul>
</div>
</div>

<div id="outline-container-sec-1" class="outline-2">
<h4 id="sec-1"><span class="section-number-2">1</span> Introduction</h4>
<div class="outline-text-2" id="text-1">
<p>
A couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems.
Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.
</p>

<p>
Having had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs.
Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment
</p>

<p>
To maximize the number of GPUs we can put in a machine, we ended up getting a <a href="http://exxactcorp.com/index.php/solution/solu_detail/207">Quantum TXR430-0768R</a>
from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.
</p>

<p>
The GPUs used for this build include:
</p>

<ul class="org-ul">
<li><a href="http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100">AMD Firepro W9100</a> x3
</li>
<li><a href="http://www.amd.com/en-us/products/graphics/server/s7150">AMD Firepro S7150</a>
</li>
<li><a href="http://www.nvidia.com/object/tesla-k80.html">NVIDIA Tesla K80</a>
</li>
<li><a href="http://www.nvidia.com/object/tesla-workstations.html">NVIDIA Tesla K40</a>
</li>
<li><a href="http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf">NVIDIA Tesla K10</a> (PDF)
</li>
</ul>

<p>
<b>Special Thanks</b>
</p>
<ul class="org-ul">
<li>To AMD for providing us with Firepro GPUs used in this build and more.
</li>
<li>To NVIDIA for providing us with Tesla GPUs used in this build and more.
</li>
</ul>

<p>
<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.
A short (but by no means comprehensive) list <a href="https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware">can be found here</a>.
</p>
</div>
</div>

<div id="outline-container-sec-2" class="outline-2">
<h4 id="sec-2"><span class="section-number-2">2</span> Setting up PCI passthrough</h4>
<div class="outline-text-2" id="text-2">
<p>
This section describes how to setup PCI passthrough on Ubuntu 16.04. This
is based on <a href="https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF">this arch wiki article</a> about the same topic.
</p>
</div>

<div id="outline-container-sec-2-1" class="outline-3">
<h5 id="sec-2-1"><span class="section-number-3">2.1</span> Setup machine for virtualization</h5>
<div class="outline-text-3" id="text-2-1">
<ul class="org-ul">
<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub
</li>
<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.
</li>
<li>Make sure you create the new <code>grub.cfg</code> file by running the following command
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">sudo grub-mkconfig -o /boot/grub/grub.cfg
</pre>
</div>

<ul class="org-ul">
<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>
</li>
<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines
</li>
</ul>

<pre class="example">
blacklist nouveau
blacklist radeon
blacklist amdgpu
</pre>

<ul class="org-ul">
<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.
</li>

<li>Reboot
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-2-2" class="outline-3">
<h5 id="sec-2-2"><span class="section-number-3">2.2</span> Adding Devices for PCI passthrough</h5>
<div class="outline-text-3" id="text-2-2">
<ul class="org-ul">
<li>Run the following command to get the list of IOMMU groups
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done
</pre>
</div>

<ul class="org-ul">
<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>
</li>

<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.
</li>

<li>A sample configuration is given below.
</li>
</ul>

<pre class="example">
options vfio-pci ids=10de:1024,10de:102d
</pre>

<ul class="org-ul">
<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>
</li>
</ul>

<pre class="example">
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
</pre>

<ul class="org-ul">
<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.
</li>

<li>Reboot
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-2-3" class="outline-3">
<h5 id="sec-2-3"><span class="section-number-3">2.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>
<div class="outline-text-3" id="text-2-3">
<ul class="org-ul">
<li>Install the required virtualization packages
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager
</pre>
</div>

<ul class="org-ul">
<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">sudo usermod -a -G kvm userid
sudo usermod -a -G libvirtd userid
</pre>
</div>

<ul class="org-ul">
<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present
</li>
</ul>

<pre class="example">
nvram = [    "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd"
]
</pre>

<ul class="org-ul">
<li>restart libvrit-bin service
</li>
</ul>

<pre class="example">
sudo systemctl restart libvirt-bin
</pre>

<ul class="org-ul">
<li>You are now ready to pass through the GPU!
</li>
</ul>
</div>
</div>
</div>

<div id="outline-container-sec-3" class="outline-2">
<h4 id="sec-3"><span class="section-number-2">3</span> Setting up the virtual machines</h4>
<div class="outline-text-2" id="text-3">
<p>
This section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.
This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).
</p>
</div>

<div id="outline-container-sec-3-1" class="outline-3">
<h5 id="sec-3-1"><span class="section-number-3">3.1</span> Creating a storage pool</h5>
<div class="outline-text-3" id="text-3-1">
<p>
If you plan on using the default pool, skip this section.
</p>

<ul class="org-ul">
<li>For creating a pool from command line using <code>virsh</code>, <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html">follow these instructions</a>.
</li>
<li>For creating a pool using <code>virt-manager</code>, <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168">follow these instructions</a>.
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-3-2" class="outline-3">
<h5 id="sec-3-2"><span class="section-number-3">3.2</span> Creating a volume</h5>
<div class="outline-text-3" id="text-3-2">
<ul class="org-ul">
<li>For creating a volume from the command line using <code>virsh</code>, use the following commands
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2
</pre>
</div>

<ul class="org-ul">
<li>For creating a volume using <code>virt-manager</code>, double click on your connection,
</li>
</ul>
<p>
navigate to storage, choose your pool, press <code>+</code> and follow the instructions.
</p>
</div>
</div>

<div id="outline-container-sec-3-3" class="outline-3">
<h5 id="sec-3-3"><span class="section-number-3">3.3</span> Networking</h5>
<div class="outline-text-3" id="text-3-3">
<p>
<code>libvirt</code> defaults to creating a NAT interface for networking.
If you want to use networking through a NAT interface, you can skip this section.
</p>

<p>
The rest of the section deals with creating a bridge interface through <code>libvirt</code>
which can later be used by the VMs.
</p>

<ul class="org-ul">
<li>Edit <code>/etc/network/interfaces</code>
</li>
<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>
</li>
</ul>

<pre class="example">
auto br0
iface br0 inet dhcp
    bridge_ports eth1
</pre>

<ul class="org-ul">
<li>Restart the network
</li>
</ul>

<pre class="example">
systemctl restart networking network-manager
</pre>
</div>
</div>

<div id="outline-container-sec-3-4" class="outline-3">
<h5 id="sec-3-4"><span class="section-number-3">3.4</span> Installing an OS</h5>
<div class="outline-text-3" id="text-3-4">
<p>
<b>Installing an OS from command line</b>
</p>

<ul class="org-ul">
<li>Run a command similar to the following
</li>
</ul>

<pre class="example">
virt-install --name test-vm-01 --memory 8192\
--vcpus 4 --disk /my/storage/location/disk-01.qcow2\
--graphics vnc,passowrd=foobar,listen=0.0.0.0\
--cdrom /path/to/os-install.iso\
--network bridge=br0\
--host-device=pci_abcd_vw_xy_z
</pre>

<ul class="org-ul">
<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>
</li>
<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.
</li>
<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>
</li>
</ul>
<pre class="example">
virsh nodedev-list | grep vw_xy_z
</pre>

<ul class="org-ul">
<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.
</li>
<li>change the password to be something more appropriate.
</li>

<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:
</li>
</ul>
<pre class="example">
virsh vncdisplay test-vm-01
</pre>

<ul class="org-ul">
<li>You can then use this information to connect to the server using a vnc client of choice
</li>
</ul>

<p>
<b>Installing an OS using <code>virt-manager</code></b>
</p>

<ul class="org-ul">
<li>Click on create new virtual machine (top left icon)
</li>
<li><code>Local install media</code>. Browse and choose install media
</li>
<li>Optional: Set OS type and version appropriately
</li>
<li>Choose the desired amount of RAM and CPU cores
</li>
<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected
</li>
<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier
</li>
<li>Check the box that says <b>Customize configuration before install</b>
</li>
<li>Click on <code>Add Hardware</code>
</li>
<li>Choose <code>PCI Host Device</code> and select the device you want to pass through
</li>
<li>Click <code>Begin Installation</code> on top left corner.
</li>
</ul>

<p>
<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>
</p>
</div>
</div>

<div id="outline-container-sec-3-5" class="outline-3">
<h5 id="sec-3-5"><span class="section-number-3">3.5</span> Accessing the VMs</h5>
<div class="outline-text-3" id="text-3-5">
<ul class="org-ul">
<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.
</li>
<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.
</li>
<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it's VMs.
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-3-6" class="outline-3">
<h5 id="sec-3-6"><span class="section-number-3">3.6</span> Cloning the VMs</h5>
<div class="outline-text-3" id="text-3-6">
<ul class="org-ul">
<li>From command line run the following command
</li>
</ul>
<pre class="example">
virt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2
</pre>

<ul class="org-ul">
<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.
</li>
</ul>
</div>
</div>
</div>

<div id="outline-container-sec-4" class="outline-2">
<h4 id="sec-4"><span class="section-number-2">4</span> Conclusion</h4>
<div class="outline-text-2" id="text-4">
<p>
Although time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations.
</p>

<p>
We plan on doing interesting things with this setup including integrating it with our <a href="http://ci.arrayfire.org/">continuous integration</a> system.
</p>

<p>
We also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!
</p>
</div>
</div>
</div>