<div id="outline-container-sec-1" class="outline-2">
<h4 id="sec-1">Introduction</h4>
<div class="outline-text-2" id="text-1">
<p>
A couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.
</p>

<p>
Having had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.
</p>

<p>
To maximize the number of GPUs we can put in a machine, we ended up getting a <a href="http://exxactcorp.com/index.php/solution/solu_detail/207">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.
</p>

<table border="0">
<td>
<a href="http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg"><img src="http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg" alt="server_7_gpus_top" width="300" height="225" class="aligncenter size-medium wp-image-8955" /></a>
</td>

<td>
<a href="http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg"><img src="http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg" alt="server_front" width="300" height="225" class="aligncenter size-medium wp-image-8959" /></a>
</td>
</table>

<p>
The GPUs used for this build include:
</p>

<ul class="org-ul">
<li><a href="http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150">AMD Firepro S9150</a> x3
</li>
<li><a href="http://www.amd.com/en-us/products/graphics/server/s7150">AMD Firepro S7150</a>
</li>
<li><a href="http://www.nvidia.com/object/tesla-k80.html">NVIDIA Tesla K80</a>
</li>
<li><a href="http://www.nvidia.com/object/tesla-workstations.html">NVIDIA Tesla K40</a>
</li>
<li><a href="http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf">NVIDIA Tesla K10</a> (PDF)
</li>
</ul>

<p>
<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.
</p>

<p>
<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href="https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware">can be found here</a>.
</p>
</div>
</div>

<div id="outline-container-sec-2" class="outline-2">
<h4 id="sec-2">Setting up PCI passthrough</h4>
<div class="outline-text-2" id="text-2">
<p>
This section describes how to setup PCI passthrough on Ubuntu 16.04. This
is based on <a href="https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF">this arch wiki article</a> about the same topic.
</p>
</div>

<div id="outline-container-sec-2-1" class="outline-3">
<h5 id="sec-2-1">Setup machine for virtualization</h5>
<div class="outline-text-3" id="text-2-1">
<ul class="org-ul">
<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub
</li>
<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.
</li>
<li>Make sure you create the new <code>grub.cfg</code> file by running the following command
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">sudo grub-mkconfig -o /boot/grub/grub.cfg
</pre>
</div>

<ul class="org-ul">
<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>
</li>
<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines
</li>
</ul>

<pre class="example">
blacklist nouveau
blacklist radeon
blacklist amdgpu
</pre>

<ul class="org-ul">
<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.
</li>

<li>Reboot
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-2-2" class="outline-3">
<h5 id="sec-2-2">Adding Devices for PCI passthrough</h5>
<div class="outline-text-3" id="text-2-2">
<ul class="org-ul">
<li>Run the following command to get the list of IOMMU groups
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done
</pre>
</div>

<ul class="org-ul">
<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>
</li>

<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.
</li>

<li>A sample configuration is given below.
</li>
</ul>

<pre class="example">
options vfio-pci ids=10de:1024,10de:102d
</pre>

<ul class="org-ul">
<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>
</li>
</ul>

<pre class="example">
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
</pre>

<ul class="org-ul">
<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.
</li>

<li>Reboot
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-2-3" class="outline-3">
<h5 id="sec-2-3">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>
<div class="outline-text-3" id="text-2-3">
<ul class="org-ul">
<li>Install the required virtualization packages
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager
</pre>
</div>

<ul class="org-ul">
<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">sudo usermod -a -G kvm userid
sudo usermod -a -G libvirtd userid
</pre>
</div>

<ul class="org-ul">
<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present
</li>
</ul>

<pre class="example">
nvram = [    "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd"
]
</pre>

<ul class="org-ul">
<li>restart libvrit-bin service
</li>
</ul>

<pre class="example">
sudo systemctl restart libvirt-bin
</pre>

<ul class="org-ul">
<li>You are now ready to pass through the GPU!
</li>
</ul>
</div>
</div>
</div>

<div id="outline-container-sec-3" class="outline-2">
<h4 id="sec-3">Setting up the virtual machines</h4>
<div class="outline-text-2" id="text-3">
<p>
This section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).
</p>
</div>

<div id="outline-container-sec-3-1" class="outline-3">
<h5 id="sec-3-1">Creating a storage pool</h5>
<div class="outline-text-3" id="text-3-1">
<p>
If you plan on using the default pool, skip this section.
</p>

<ul class="org-ul">
<li>For creating a pool from command line using <code>virsh</code>, <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html">follow these instructions</a>.
</li>
<li>For creating a pool using <code>virt-manager</code>, <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168">follow these instructions</a>.
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-3-2" class="outline-3">
<h5 id="sec-3-2">Creating a volume</h5>
<div class="outline-text-3" id="text-3-2">
<ul class="org-ul">
<li>For creating a volume from the command line using <code>virsh</code>, use the following commands
</li>
</ul>

<div class="org-src-container">

<pre class="src src-sh">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2
</pre>
</div>

<ul class="org-ul">
<li>For creating a volume using <code>virt-manager</code>, double click on your connection,
</li>
</ul>
<p>
navigate to storage, choose your pool, press <code>+</code> and follow the instructions.
</p>
</div>
</div>

<div id="outline-container-sec-3-3" class="outline-3">
<h5 id="sec-3-3">Networking</h5>
<div class="outline-text-3" id="text-3-3">
<p>
<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.
</p>

<p>
The rest of the section deals with creating a bridge interface which can later be used by the VMs.
</p>

<ul class="org-ul">
<li>Edit <code>/etc/network/interfaces</code>
</li>
<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>
</li>
</ul>

<pre class="example">
auto br0
iface br0 inet dhcp
    bridge_ports eth1
</pre>

<ul class="org-ul">
<li>Restart the network
</li>
</ul>

<pre class="example">
systemctl restart networking network-manager
</pre>
</div>
</div>

<div id="outline-container-sec-3-4" class="outline-3">
<h5 id="sec-3-4">Installing an OS</h5>
<div class="outline-text-3" id="text-3-4">
<p>
<b>Installing an OS from command line</b>
</p>

<ul class="org-ul">
<li>Run a command similar to the following
</li>
</ul>

<pre class="example">
virt-install --name test-vm-01 --memory 8192\
--vcpus 4 --disk /my/storage/location/disk-01.qcow2\
--graphics vnc,password=foobar,listen=0.0.0.0\
--cdrom /path/to/os-install.iso\
--network bridge=br0\
--host-device=pci_abcd_vw_xy_z
</pre>

<ul class="org-ul">
<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>
</li>
<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.
</li>
<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>
</li>
</ul>
<pre class="example">
virsh nodedev-list | grep vw_xy_z
</pre>

<ul class="org-ul">
<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.
</li>
<li>change the password to be something more appropriate.
</li>

<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:
</li>
</ul>
<pre class="example">
virsh vncdisplay test-vm-01
</pre>

<ul class="org-ul">
<li>You can then use this information to connect to the server using a vnc client of choice
</li>
</ul>

<p>
<b>Installing an OS using <code>virt-manager</code></b>
</p>

<ul class="org-ul">
<li>Click on create new virtual machine (top left icon)
</li>
<li><code>Local install media</code>. Browse and choose install media
</li>
<li>Optional: Set OS type and version appropriately
</li>
<li>Choose the desired amount of RAM and CPU cores
</li>
<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected
</li>
<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier
</li>
<li>Check the box that says <b>Customize configuration before install</b>
</li>
<li>Click on <code>Add Hardware</code>
</li>
<li>Choose <code>PCI Host Device</code> and select the device you want to pass through
</li>
<li>Click <code>Begin Installation</code> on top left corner.
</li>
</ul>

<p>
<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>
</p>
</div>
</div>

<div id="outline-container-sec-3-5" class="outline-3">
<h5 id="sec-3-5">Accessing the VMs</h5>
<div class="outline-text-3" id="text-3-5">
<ul class="org-ul">
<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.
</li>
<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.
</li>
<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it's VMs.
</li>
</ul>
</div>
</div>

<div id="outline-container-sec-3-6" class="outline-3">
<h5 id="sec-3-6">Cloning the VMs</h5>
<div class="outline-text-3" id="text-3-6">
<ul class="org-ul">
<li>From command line run the following command
</li>
</ul>
<pre class="example">
virt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2
</pre>

<ul class="org-ul">
<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.
</li>
</ul>
</div>
</div>
</div>

<div id="outline-container-sec-4" class="outline-2">
<h4 id="sec-4">Conclusion</h4>
<div class="outline-text-2" id="text-4">
<p>
Although time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href="http://ci.arrayfire.org/">continuous integration</a> system.
</p>

<p>
We also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!
</p>
</div>
</div>
</div>