(8897,18528,'2016-05-25 13:40:22','2016-05-25 17:40:22','Programmers and Data Scientists want to take advantage of fast and parallel computational devices. One of the best ways of doing this is to write vectorized code; however, this is often easier said than done. In today\'s continuation of the blog series \"<a href=\"http://sandbox.arrayfire.com/tag/learning-arrayfire-from-scratch/\">Learning ArrayFire from scratch</a>\", we will discuss the various methods by which you can maximize the performance of your code by using ArrayFire\'s built-in vectorization features.\r\n\r\n<!--more-->\r\n\r\nArrayFire provides many ways to vectorize a given code segment. In this tutorial, we present several methods to vectorize code using ArrayFire and discuss the benefits and drawbacks associated with each method.\r\n<h4><a id=\"GenericDefault_vectorization_10\"></a>Generic/Default vectorization</h4>\r\nBy its very nature, ArrayFire is a vectorized library. Most functions operate on arrays as a whole – on all elements in parallel. Wherever possible, existing vectorized functions should be used opposed to manually indexing into arrays. For example consider the following code:\r\n<pre class=\"lang:default decode:true \">af::array a = af::range(10); // [0,  9]\r\nfor(int i = 0; i &lt; a.dims(0); ++i)\r\n{\r\n    a(i) = a(i) + 1;         // [1, 10]\r\n}</pre>\r\nAlthough completely valid, the code is very inefficient as it results in a kernel that operate on one datum. Instead, the developer should have used ArrayFire\'s overload of the + operator:\r\n<pre><code class=\"language-{.cpp}\">\r\naf::array a = af::range(10);  // [0,  9]\r\na = a + 1;                    // [1, 10]\r\n</code></pre>\r\nThis code will result in a single kernel that operates on all 10 elements of <code>a</code> in parallel.\r\n\r\nMost ArrayFire functions are vectorized. A small subset of these include:\r\n\r\n<strong>Arithmetic operations:</strong> +, -, *, / , %, &gt;&gt;, &lt;&lt;\r\n<strong>Logical operations:</strong> &amp;&amp;, ||, &lt;, &gt;, ==, !=\r\n<strong>Numeric functions:</strong> abs(), floor(), round(), min(), max()…\r\n<strong>Complex operations:</strong> real(), imag(), conj()…\r\n<strong>Exponential and logarithmic functions:</strong> exp(), log(), expm1(), log1p()…\r\n<strong>Trigonometric functions:</strong> sin(), cos(), tan()…\r\n<strong>Hyperbolic functions:</strong> sinh(), cosh(), tanh()…\r\n\r\nIn addition to element-wise operations, many other functions are also vectorized in ArrayFire.\r\n\r\nNotice that even that perform some form of aggregation (e.g. <code>sum()</code> or <code>min()</code>), signal processing (like <code>convolve()</code>), and even image processing functions (i.e. <code>rotate()</code>) all support vectorization on different columns or images. For example, if we have <code>NUM</code> images of size <code>WIDTH</code> by <code>HEIGHT</code>, one could convolve each image in a vector fashion as follows:\r\n<pre class=\"lang:default decode:true \">float g_coef[] = { 1, 2, 1,\r\n                   2, 4, 2,\r\n                   1, 2, 1 };\r\naf::array filter = 1.f/16 * af::array(3, 3, f_coef);\r\naf::array signal = randu(WIDTH, HEIGHT, NUM);\r\naf::array conv = convolve2(signal, filter);</pre>\r\nSimilarly, one can rotate 100 images by 45 degrees in a single call using code like the following:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Construct an array of 100 WIDTH x HEIGHT images of random numbers\r\naf::array imgs = randu(WIDTH, HEIGHT, 100);\r\n// Rotate all of the images in a single command\r\naf::array rot_imgs = rotate(imgs, 45);</pre>\r\n</div>\r\n</div>\r\nAlthough <em>most</em> functions in ArrayFire do support vectorization, some do not. Most notably, all linear algebra functions. Even though they are not vectorized linear algebra operations still execute in parallel on your hardware.\r\n\r\nUsing the built in vectorized operations should be the first and preferred method of vectorizing any code written with ArrayFire.\r\n<h4><a id=\"GFOR_Parallel_forloops_76\"></a>GFOR: Parallel for-loops</h4>\r\nAnother novel method of vectorization present in ArrayFire is the GFOR loop replacement construct. GFOR allows launching all iterations of a loop in parallel on the GPU or device, as long as the iterations are independent. While the standard for-loop performs each iteration sequentially, ArrayFire\'s gfor-loop performs each iteration at the same time (in parallel). ArrayFire does this by tiling out the values of all loop iterations and then performing computation on those tiles in one pass. You can think of gfor as performing auto-vectorization of your code, e.g. you write a gfor-loop that increments every element of a vector but behind the scenes ArrayFire rewrites it to operate on the entire vector in parallel.\r\n\r\nThe original for-loop example at the beginning of this document could be rewritten using GFOR as follows:\r\n<pre><code class=\"language-{.cpp}\">af::array a = af::range(10);\r\ngfor(seq i, n)\r\n    a(i) = a(i) + 1;</code></pre>\r\nIn this case, each instance of the gfor loop is independent, thus ArrayFire will automatically tile out the <code>a</code> array in device memory and execute the increment kernels in parallel.\r\n\r\nTo see another example, you could run an <a class=\"el\" title=\"C++ Interface exclusive sum (cumulative sum) of an array. \" href=\"group__scan__func__accum.htm#gae450135c4ae1bdaa4489804a215f3214\">accum()</a> on every slice of a matrix in a for-loop, or you could \"vectorize\" and simply do it all in one gfor-loop operation:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// runs each accum() in sequence\r\nfor (int i = 0; i &lt; N; ++i)\r\n    B(span,i) = accum(A(span,i));\r\n \r\n// runs N accums in parallel\r\ngfor (seq i, N)\r\n    B(span,i) = accum(A(span,i));</pre>\r\n</div>\r\n</div>\r\nHowever, returning to our previous vectorization technique, <a class=\"el\" title=\"C++ Interface exclusive sum (cumulative sum) of an array. \" href=\"group__scan__func__accum.htm#gae450135c4ae1bdaa4489804a215f3214\">accum()</a> is already vectorized and the operation could be completely replaced with merely:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">B = accum(A);</pre>\r\n</div>\r\n</div>\r\nIt is best to vectorize computation as much as possible to avoid the overhead in both for-loops and gfor-loops. However, the gfor-loop construct is most effective in the narrow case of broadcast-style operations. Consider the case when we have a vector of constants that we wish to apply to a collection of variables, such as expressing the values of a linear combination for multiple vectors. The broadcast of one set of constants to many vectors works well with gfor-loops:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">const static int p=4, n=1000;\r\naf::array consts = af::randu(p);\r\naf::array var_terms = randn(p, n);\r\n \r\ngfor(seq i, n)\r\n    combination(span, i) = consts * var_terms(span, i);</pre>\r\n</div>\r\n</div>\r\nUsing GFOR requires following several rules and multiple guidelines for optimal performance. The details of this vectorization method can be found in the <a class=\"el\" href=\"page_gfor.htm#gfor\">GFOR documentation</a>.\r\n<h4><a id=\"Batching_127\"></a>Batching</h4>\r\nThe <a class=\"el\" href=\"namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b\">batchFunc()</a> function allows the broad application of existing ArrayFire functions to multiple sets of data. Effectively,<a class=\"el\" href=\"namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b\">batchFunc()</a> allows ArrayFire functions to execute in \"batch processing\" mode. In this mode, functions will find a dimension which contains \"batches\" of data to be processed and will parallelize the procedure.\r\n\r\nConsider the following example. Here we create a filter which we would like to apply to each of the weight vectors. The naive solution would be using a for-loop as we have seen previously:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Create the filter and the weight vectors\r\naf::array filter = randn(1, 5);\r\naf::array weights = randu(5, 5);\r\n \r\n// Apply the filter using a for-loop\r\naf::array filtered_weights = constant(0, 5, 5);\r\nfor(int i=0; i&lt;weights.dims(1); ++i){\r\n    filtered_weights.col(i) = filter * weights.col(i);\r\n}</pre>\r\n</div>\r\n</div>\r\nHowever, as we have discussed above, this solution will be very inefficient. One may be tempted to implement a vectorized solution as follows:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Create the filter and the weight vectors\r\naf::array filter = randn(1, 5);\r\naf::array weights = randu(5, 5);\r\n \r\naf::array filtered_weights = filter * weights; // fails due to dimension mismatch</pre>\r\n</div>\r\n</div>\r\nHowever, the dimensions of <code>filter</code> and <code>weights</code> do not match, thus ArrayFire will generate a runtime error.\r\n\r\n<code>batchfunc()</code> was created to solve this specific problem. The signature of the function is as follows:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">array batchFunc(const array &amp;lhs, const array &amp;rhs, batchFunc_t func);</pre>\r\n</div>\r\n</div>\r\nwhere <code>batchFunc_t</code> is a function pointer of the form:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">typedef array (*batchFunc_t) (const array &amp;lhs, const array &amp;rhs);</pre>\r\n</div>\r\n</div>\r\nSo, to use <a class=\"el\" href=\"namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b\">batchFunc()</a>, we need to provide the function we wish to apply as a batch operation. For illustration\'s sake, let\'s \"implement\" a multiplication function following the format.\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">af::array my_mult (const af::array &amp;lhs, const af::array &amp;rhs){\r\nreturn lhs * rhs;\r\n}</pre>\r\n</div>\r\n</div>\r\nOur final batch call is not much more difficult than the ideal syntax we imagined.\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Create the filter and the weight vectors\r\naf::array filter = randn(1, 5);\r\naf::array weights = randu(5, 5);\r\n \r\n// Apply the batch function\r\naf::array filtered_weights = batchFunc( filter, weights, my_mult );</pre>\r\n</div>\r\n</div>\r\nThe batch function will work with many previously mentioned vectorized ArrayFire functions. It can even work with a combination of those functions if they are wrapped inside a helper function matching the <code>__batchFunc_t__</code> signature. One limitation of <code>batchfunc()</code> is that it cannot be used from within a <code><a class=\"el\" href=\"gfor_8h.htm#a6d6cd11d53daad217fd0b92f5a7ceec6\">gfor()</a></code> loop at the present time.\r\n<h4><a id=\"Advanced_Vectorization_181\"></a>Advanced Vectorization</h4>\r\nWe have seen the different methods ArrayFire provides to vectorize our code. Tying them all together is a slightly more involved process that needs to consider data dimensionality and layout, memory usage, nesting order, etc. An excellent example and discussion of these factors can be found in another blog post:\r\n\r\n<a href=\"http://sandbox.arrayfire.com/how-to-write-vectorized-code/\">http://sandbox.arrayfire.com/how-to-write-vectorized-code/</a>\r\n\r\nIt’s worth noting that the whole implementation discussed in the blog has since been transformed into a convenient <strong>af::nearestNeighbour()</strong> function. Before writing something from scratch, check that ArrayFire doesn’t already have an implementation. The default vectorized nature of ArrayFire and an extensive collection of functions will speed things up in addition to replacing dozens of lines of code!','Introduction to Vectorization in ArrayFire','','inherit','closed','closed','','8735-revision-v1','','','2016-05-25 13:40:22','2016-05-25 17:40:22','',8735,'http://sandbox.arrayfire.com/8735-revision-v1/',0,'revision','',0),(8898,18528,'2016-05-25 13:43:54','2016-05-25 17:43:54','Programmers and Data Scientists want to take advantage of fast and parallel computational devices. One of the best ways of doing this is to write vectorized code; however, this is often easier said than done. In today\'s continuation of the blog series \"<a href=\"http://sandbox.arrayfire.com/tag/learning-arrayfire-from-scratch/\">Learning ArrayFire from scratch</a>\", we will discuss the various methods by which you can maximize the performance of your code by using ArrayFire\'s built-in vectorization features.\r\n\r\n<!--more-->\r\n\r\nArrayFire provides many ways to vectorize a given code segment. In this tutorial, we present several methods to vectorize code using ArrayFire and discuss the benefits and drawbacks associated with each method.\r\n<h4><a id=\"GenericDefault_vectorization_10\"></a>Generic/Default vectorization</h4>\r\nBy its very nature, ArrayFire is a vectorized library. Most functions operate on arrays as a whole – on all elements in parallel. Wherever possible, existing vectorized functions should be used opposed to manually indexing into arrays. For example consider the following code:\r\n<pre class=\"lang:default decode:true \">af::array a = af::range(10); // [0,  9]\r\nfor(int i = 0; i &lt; a.dims(0); ++i)\r\n{\r\n    a(i) = a(i) + 1;         // [1, 10]\r\n}</pre>\r\nAlthough completely valid, the code is very inefficient as it results in a kernel that operate on one datum. Instead, the developer should have used ArrayFire\'s overload of the + operator:\r\n<pre><code class=\"language-{.cpp}\">\r\naf::array a = af::range(10);  // [0,  9]\r\na = a + 1;                    // [1, 10]\r\n</code></pre>\r\nThis code will result in a single kernel that operates on all 10 elements of <code>a</code> in parallel.\r\n\r\nMost ArrayFire functions are vectorized. A small subset of these include:\r\n\r\n<strong>Arithmetic operations:</strong> +, -, *, / , %, &gt;&gt;, &lt;&lt;\r\n<strong>Logical operations:</strong> &amp;&amp;, ||, &lt;, &gt;, ==, !=\r\n<strong>Numeric functions:</strong> abs(), floor(), round(), min(), max()…\r\n<strong>Complex operations:</strong> real(), imag(), conj()…\r\n<strong>Exponential and logarithmic functions:</strong> exp(), log(), expm1(), log1p()…\r\n<strong>Trigonometric functions:</strong> sin(), cos(), tan()…\r\n<strong>Hyperbolic functions:</strong> sinh(), cosh(), tanh()…\r\n\r\nIn addition to element-wise operations, many other functions are also vectorized in ArrayFire.\r\n\r\nNotice that even that performs some form of aggregation (e.g. <code>sum()</code> or <code>min()</code>), signal processing (like <code>convolve()</code>), and even image processing functions (i.e. <code>rotate()</code>) all support vectorization on different columns or images. For example, if we have <code>NUM</code> images of size <code>WIDTH</code> by <code>HEIGHT</code>, one could convolve each image in a vector fashion as follows:\r\n<pre class=\"lang:default decode:true \">float g_coef[] = { 1, 2, 1,\r\n                   2, 4, 2,\r\n                   1, 2, 1 };\r\naf::array filter = 1.f/16 * af::array(3, 3, f_coef);\r\naf::array signal = randu(WIDTH, HEIGHT, NUM);\r\naf::array conv = convolve2(signal, filter);</pre>\r\nSimilarly, one can rotate 100 images by 45 degrees in a single call using code like the following:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Construct an array of 100 WIDTH x HEIGHT images of random numbers\r\naf::array imgs = randu(WIDTH, HEIGHT, 100);\r\n// Rotate all of the images in a single command\r\naf::array rot_imgs = rotate(imgs, 45);</pre>\r\n</div>\r\n</div>\r\nAlthough <em>most</em> functions in ArrayFire do support vectorization, some do not. Most notably, all linear algebra functions. Even though they are not vectorized linear algebra operations still execute in parallel on your hardware.\r\n\r\nUsing the built in vectorized operations should be the first and preferred method of vectorizing any code written with ArrayFire.\r\n<h4><a id=\"GFOR_Parallel_forloops_76\"></a>GFOR: Parallel for-loops</h4>\r\nAnother novel method of vectorization present in ArrayFire is the GFOR loop replacement construct. GFOR allows launching all iterations of a loop in parallel on the GPU or device, as long as the iterations are independent. While the standard for-loop performs each iteration sequentially, ArrayFire\'s gfor-loop performs each iteration at the same time (in parallel). ArrayFire does this by tiling out the values of all loop iterations and then performing computation on those tiles in one pass. You can think of gfor as performing auto-vectorization of your code, e.g. you write a gfor-loop that increments every element of a vector but behind the scenes ArrayFire rewrites it to operate on the entire vector in parallel.\r\n\r\nThe original for-loop example at the beginning of this document could be rewritten using GFOR as follows:\r\n<pre><code class=\"language-{.cpp}\">af::array a = af::range(10);\r\ngfor(seq i, n)\r\n    a(i) = a(i) + 1;</code></pre>\r\nIn this case, each instance of the gfor loop is independent, thus ArrayFire will automatically tile out the <code>a</code> array in device memory and execute the increment kernels in parallel.\r\n\r\nTo see another example, you could run an <a class=\"el\" title=\"C++ Interface exclusive sum (cumulative sum) of an array. \" href=\"group__scan__func__accum.htm#gae450135c4ae1bdaa4489804a215f3214\">accum()</a> on every slice of a matrix in a for-loop, or you could \"vectorize\" and simply do it all in one gfor-loop operation:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// runs each accum() in sequence\r\nfor (int i = 0; i &lt; N; ++i)\r\n    B(span,i) = accum(A(span,i));\r\n \r\n// runs N accums in parallel\r\ngfor (seq i, N)\r\n    B(span,i) = accum(A(span,i));</pre>\r\n</div>\r\n</div>\r\nHowever, returning to our previous vectorization technique, <a class=\"el\" title=\"C++ Interface exclusive sum (cumulative sum) of an array. \" href=\"group__scan__func__accum.htm#gae450135c4ae1bdaa4489804a215f3214\">accum()</a> is already vectorized and the operation could be completely replaced with merely:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">B = accum(A);</pre>\r\n</div>\r\n</div>\r\nIt is best to vectorize computation as much as possible to avoid the overhead in both for-loops and gfor-loops. However, the gfor-loop construct is most effective in the narrow case of broadcast-style operations. Consider the case when we have a vector of constants that we wish to apply to a collection of variables, such as expressing the values of a linear combination for multiple vectors. The broadcast of one set of constants to many vectors works well with gfor-loops:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">const static int p=4, n=1000;\r\naf::array consts = af::randu(p);\r\naf::array var_terms = randn(p, n);\r\n \r\ngfor(seq i, n)\r\n    combination(span, i) = consts * var_terms(span, i);</pre>\r\n</div>\r\n</div>\r\nUsing GFOR requires following several rules and multiple guidelines for optimal performance. The details of this vectorization method can be found in the <a class=\"el\" href=\"page_gfor.htm#gfor\">GFOR documentation</a>.\r\n<h4><a id=\"Batching_127\"></a>Batching</h4>\r\nThe <a class=\"el\" href=\"namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b\">batchFunc()</a> function allows the broad application of existing ArrayFire functions to multiple sets of data. Effectively,<a class=\"el\" href=\"namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b\">batchFunc()</a> allows ArrayFire functions to execute in \"batch processing\" mode. In this mode, functions will find a dimension which contains \"batches\" of data to be processed and will parallelize the procedure.\r\n\r\nConsider the following example. Here we create a filter which we would like to apply to each of the weight vectors. The naive solution would be using a for-loop as we have seen previously:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Create the filter and the weight vectors\r\naf::array filter = randn(1, 5);\r\naf::array weights = randu(5, 5);\r\n \r\n// Apply the filter using a for-loop\r\naf::array filtered_weights = constant(0, 5, 5);\r\nfor(int i=0; i&lt;weights.dims(1); ++i){\r\n    filtered_weights.col(i) = filter * weights.col(i);\r\n}</pre>\r\n</div>\r\n</div>\r\nHowever, as we have discussed above, this solution will be very inefficient. One may be tempted to implement a vectorized solution as follows:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Create the filter and the weight vectors\r\naf::array filter = randn(1, 5);\r\naf::array weights = randu(5, 5);\r\n \r\naf::array filtered_weights = filter * weights; // fails due to dimension mismatch</pre>\r\n</div>\r\n</div>\r\nHowever, the dimensions of <code>filter</code> and <code>weights</code> do not match, thus ArrayFire will generate a runtime error.\r\n\r\n<code>batchfunc()</code> was created to solve this specific problem. The signature of the function is as follows:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">array batchFunc(const array &amp;lhs, const array &amp;rhs, batchFunc_t func);</pre>\r\n</div>\r\n</div>\r\nwhere <code>batchFunc_t</code> is a function pointer of the form:\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">typedef array (*batchFunc_t) (const array &amp;lhs, const array &amp;rhs);</pre>\r\n</div>\r\n</div>\r\nSo, to use <a class=\"el\" href=\"namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b\">batchFunc()</a>, we need to provide the function we wish to apply as a batch operation. For illustration\'s sake, let\'s \"implement\" a multiplication function following the format.\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">af::array my_mult (const af::array &amp;lhs, const af::array &amp;rhs){\r\nreturn lhs * rhs;\r\n}</pre>\r\n</div>\r\n</div>\r\nOur final batch call is not much more difficult than the ideal syntax we imagined.\r\n<div class=\"fragment\">\r\n<div class=\"line\">\r\n<pre class=\"lang:default decode:true \">// Create the filter and the weight vectors\r\naf::array filter = randn(1, 5);\r\naf::array weights = randu(5, 5);\r\n \r\n// Apply the batch function\r\naf::array filtered_weights = batchFunc( filter, weights, my_mult );</pre>\r\n</div>\r\n</div>\r\nThe batch function will work with many previously mentioned vectorized ArrayFire functions. It can even work with a combination of those functions if they are wrapped inside a helper function matching the <code>__batchFunc_t__</code> signature. One limitation of <code>batchfunc()</code> is that it cannot be used from within a <code><a class=\"el\" href=\"gfor_8h.htm#a6d6cd11d53daad217fd0b92f5a7ceec6\">gfor()</a></code> loop at the present time.\r\n<h4><a id=\"Advanced_Vectorization_181\"></a>Advanced Vectorization</h4>\r\nWe have seen the different methods ArrayFire provides to vectorize our code. Tying them all together is a slightly more involved process that needs to consider data dimensionality and layout, memory usage, nesting order, etc. An excellent example and discussion of these factors can be found in another blog post:\r\n\r\n<a href=\"http://sandbox.arrayfire.com/how-to-write-vectorized-code/\">http://sandbox.arrayfire.com/how-to-write-vectorized-code/</a>\r\n\r\nIt’s worth noting that the whole implementation discussed in the blog has since been transformed into a convenient <strong>af::nearestNeighbour()</strong> function. Before writing something from scratch, check that ArrayFire doesn’t already have an implementation. The default vectorized nature of ArrayFire and an extensive collection of functions will speed things up in addition to replacing dozens of lines of code!','Introduction to Vectorization in ArrayFire','','inherit','closed','closed','','8735-revision-v1','','','2016-05-25 13:43:54','2016-05-25 17:43:54','',8735,'http://sandbox.arrayfire.com/8735-revision-v1/',0,'revision','',0),(8901,18528,'2016-06-10 07:33:38','2016-06-10 11:33:38','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\n<h4>Use of the Marks without written permission</h4>\r\nSubject to the inclusion of the disclaimer at the bottom of this section, we permit the use of the literal mark, \"ArrayFire\" to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n\r\n&nbsp;\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software that provides interoperability between ArrayFire and a programming language (commonly called \"language wrappers\"), in the format \"ArrayFire.[Language]\" or \"ArrayFire-[Language]\" or [Language] for ArrayFire\" -- provided that such use case otherwise compiles with the rest of the Policy.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</li>\r\n</ul>\r\nIn the above scenarios, the following disclaimer must be prominently displayed:\r\n\r\n<em>\"[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire literal mark is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\",</em>\r\n\r\n&nbsp;\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>','Trademark Policy','','publish','closed','closed','','trademark-policy','','','2016-06-10 07:33:38','2016-06-10 11:33:38','',0,'http://sandbox.arrayfire.com/?page_id=8901',0,'page','',0),(8902,18528,'2016-06-02 09:11:54','2016-06-02 13:11:54','<span style=\"font-weight: 400;\">The literal mark “ArrayFire” and ArrayFire logos are trademarks of AccelerEyes LLC D.B.A. ArrayFire. </span>\r\n\r\n<span style=\"font-weight: 400;\">Trademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.</span>\r\n\r\n<span style=\"font-weight: 400;\">Throughout this Policy, the word “Marks” refers to the following:</span>\r\n<ul>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">The literal mark “ArrayFire”</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">The logos and graphic marks displayed and avaliable for download at  </span><a href=\"http://sandbox.arrayfire.com/trademark/logos\"><span style=\"font-weight: 400;\">http://sandbox.arrayfire.com/trademark</span></a></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">The slogan “If you want faster code, you’ve come to the right place”</span></li>\r\n</ul>\r\n<h2><span style=\"font-weight: 400;\">Use of the marks without written permission</span></h2>\r\n<span style=\"font-weight: 400;\">You may use the Marks without prior written permission to refer to ArrayFire software or an ArrayFire product in a fashion consistent with trademark fair use. This frequently requires that you use only the words, not the font or graphics, </span><a href=\"https://en.wikipedia.org/wiki/Nominative_use\"><span style=\"font-weight: 400;\">in a nominative form</span></a><span style=\"font-weight: 400;\"> to refer to the trademarked product or its source without implying sponsorship or endorsement by the trademark holder. A few examples of use cases that are clearly fair use are:</span>\r\n<ol>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">To refer to the ArrayFire software in substantially unmodified form.</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">To factually refer to the ArrayFire products, software, services, or source.</span></li>\r\n</ol>\r\n<span style=\"font-weight: 400;\">In addition, you may use the Marks to refer to projects, services, or communities outside of the ArrayFire software and ArrayFire Project without written permission in the following contexts:</span>\r\n<ol start=\"4\">\r\n 	<li><span style=\"font-weight: 400;\">When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</span></li>\r\n</ol>\r\n<ol start=\"5\">\r\n 	<li><span style=\"font-weight: 400;\">When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</span></li>\r\n</ol>\r\n<h2><span style=\"font-weight: 400;\">Prohibited usages of the Marks</span></h2>\r\n<span style=\"font-weight: 400;\">You may not use the Marks in the following ways:</span>\r\n<ul>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">In any way likely to cause confusion as to the identity of an ArrayFire project, the provenance of its software, or the software\'s license.</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">In any way that indicates a greater degree of association between you and the ArrayFire Project than actually exists.</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">In any way that implies a designated successor to ArrayFire products, software, services, or source.</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">In any other way that dilutes or otherwise infringes upon the trademark rights of AccelerEyes LLC.</span></li>\r\n</ul>\r\n<span style=\"font-weight: 400;\">In addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from AccelerEyes LLC. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.</span>\r\n<h2><span style=\"font-weight: 400;\">Reservation of Rights</span></h2>\r\n<span style=\"font-weight: 400;\">ArrayFire reserves the sole right to:</span>\r\n<ul>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Determine the compliance with this policy</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Modify the Policy in ways consistent with the mission of protecting the public.</span></li>\r\n 	<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</span></li>\r\n</ul>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 09:11:54','2016-06-02 13:11:54','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8903,18528,'2016-06-02 09:39:43','2016-06-02 13:39:43','The literal mark “ArrayFire” and ArrayFire logos are trademarks of AccelerEyes LLC D.B.A. ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at http://sandbox.arrayfire.com/trademark</li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Products\" refers to software, source code, services, and training content developed by AcclerEyes LLC or otherwise contributed to AccelerEyes LLC by third-parties.\r\n<h4>Use of the marks without written permission</h4>\r\nYou may use the Marks without prior written permission to refer to ArrayFire Products when the Marks are used for the purpose of description or identification of ArrayFire Products where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to the ArrayFire products, software, services, or source.</li>\r\n</ul>\r\nIn addition, we permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Products without written permission in the following contexts:\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy. Such items must display the following disclaimer: \"<em>[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by AccelerEyes LLC the trademark holder in the United States and other countries.\"</em></li>\r\n</ul>\r\n<h4>Prohibited usages of the Marks</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire project, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to ArrayFire products, software, services, or source.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of AccelerEyes LLC.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from AccelerEyes LLC. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n\r\nReservation of Rights\r\n\r\nArrayFire reserves the sole right to:\r\n\r\nDetermine the compliance with this policy\r\n\r\nModify the Policy in ways consistent with the mission of protecting the public.\r\n\r\nGrant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 09:39:43','2016-06-02 13:39:43','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8904,18528,'2016-06-02 09:45:41','2016-06-02 13:45:41','','arrayfire_icon','','inherit','open','closed','','arrayfire_icon','','','2016-06-02 09:45:41','2016-06-02 13:45:41','',8901,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png',0,'attachment','image/png',0),(8905,18528,'2016-06-02 09:45:41','2016-06-02 13:45:41','','arrayfire_logo','','inherit','open','closed','','arrayfire_logo','','','2016-06-02 09:45:41','2016-06-02 13:45:41','',8901,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png',0,'attachment','image/png',0),(8906,18528,'2016-06-02 09:47:11','2016-06-02 13:47:11','The literal mark “ArrayFire” and ArrayFire logos are trademarks of AccelerEyes LLC D.B.A. ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Products\" refers to software, source code, services, and training content developed by AcclerEyes LLC or otherwise contributed to AccelerEyes LLC by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to ArrayFire Products when the Marks are used for the purpose of description or identification of ArrayFire Products where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to the ArrayFire products, software, services, or source.</li>\r\n</ul>\r\nIn addition, we permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Products without written permission in the following contexts:\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy. Such items must display the following disclaimer: \"<em>[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by AccelerEyes LLC the trademark holder in the United States and other countries.\"</em></li>\r\n</ul>\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire project, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to ArrayFire products, software, services, or source.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of AccelerEyes LLC.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from AccelerEyes LLC. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 09:47:11','2016-06-02 13:47:11','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8908,18528,'2016-06-02 09:58:53','2016-06-02 13:58:53','The literal mark “ArrayFire” and ArrayFire logos are trademarks of AccelerEyes LLC D.B.A. ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by AcclerEyes LLC or otherwise contributed to AccelerEyes LLC by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\nIn addition, we permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy. Such items must display the following disclaimer: \"<em>[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by AccelerEyes LLC the trademark holder in the United States and other countries.\"</em></li>\r\n</ul>\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or AccelerEyes LLC than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of AccelerEyes LLC.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from AccelerEyes LLC. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 09:58:53','2016-06-02 13:58:53','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8909,18528,'2016-06-02 10:02:49','2016-06-02 14:02:49','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\nIn addition, we permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy. Such items must display the following disclaimer: \"<em>[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\"</em></li>\r\n</ul>\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 10:02:49','2016-06-02 14:02:49','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8911,18528,'2016-06-02 12:32:44','2016-06-02 16:32:44','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\nIn addition, we permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software provided that such use otherwise complies with the rest of the Policy. Such items must display the following disclaimer: \"<em>[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\"</em></li>\r\n</ul>\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 12:32:44','2016-06-02 16:32:44','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8912,18528,'2016-06-02 12:46:39','2016-06-02 16:46:39','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\nIn addition, we permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy. Such items must display the following disclaimer: \"<em>[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\"</em></li>\r\n</ul>\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 12:46:39','2016-06-02 16:46:39','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8913,18528,'2016-06-02 12:55:38','2016-06-02 16:55:38','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\nIn addition, we permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n<ul>\r\n 	<li>When used in or as part of a mark of third-party software that is released under an <a href=\"https://opensource.org/licenses\">Open-Source Initiative Approved License</a> provided that such use otherwise complies with the rest of the Policy.</li>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</li>\r\n</ul>\r\nHowever, the aforementioned use cases must include the following disclaimer:\r\n\r\n<em>\"[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\"</em>\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 12:55:38','2016-06-02 16:55:38','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8914,18528,'2016-06-02 15:17:38','2016-06-02 19:17:38','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\n<h4>Use of the Marks without written permission</h4>\r\nWe permit the use of the Marks to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts subject to the inclusion of a disclaimer:\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party programming language wrapper that provides interoperability between ArrayFire, in the format \"ArrayFire.[Language]\" or \"ArrayFire-[Language]\" or [Language] for ArrayFire\" -- provided that such use case otherwise compiles with the rest of the Policy.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</li>\r\n</ul>\r\nThese use cases must publicly display the following disclaimer:\r\n\r\n<em>\"[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire logo is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\"</em>\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 15:17:38','2016-06-02 19:17:38','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8915,18528,'2016-06-02 15:25:56','2016-06-02 19:25:56','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\n<h4>Use of the Marks without written permission</h4>\r\nSubject to the inclusion of the disclaimer at the bottom of this section, we permit the use of the literal mark, \"ArrayFire\" to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n\r\n&nbsp;\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software that provides interoperability between ArrayFire and a programming language (commonly called \"language wrappers\"), in the format \"ArrayFire.[Language]\" or \"ArrayFire-[Language]\" or [Language] for ArrayFire\" -- provided that such use case otherwise compiles with the rest of the Policy.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</li>\r\n</ul>\r\nIn the above scenarios, the following disclaimer must be prominently displayed:\r\n\r\n<em>\"[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire literal mark is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\",</em>\r\n\r\n&nbsp;\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nArrayFire Icon\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\"><img class=\"size-full wp-image-8904 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_icon.png\" alt=\"arrayfire_icon\" width=\"32\" height=\"32\" /></a>\r\n\r\n&nbsp;\r\n\r\nArrayFire Logo\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\"><img class=\"size-full wp-image-8905 alignleft\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/arrayfire_logo.png\" alt=\"arrayfire_logo\" width=\"400\" height=\"57\" /></a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 15:25:56','2016-06-02 19:25:56','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8916,18528,'2016-06-02 15:34:51','2016-06-02 19:34:51','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\n<h4>Use of the Marks without written permission</h4>\r\nSubject to the inclusion of the disclaimer at the bottom of this section, we permit the use of the literal mark, \"ArrayFire\" to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n\r\n&nbsp;\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software that provides interoperability between ArrayFire and a programming language (commonly called \"language wrappers\"), in the format \"ArrayFire.[Language]\" or \"ArrayFire-[Language]\" or [Language] for ArrayFire\" -- provided that such use case otherwise compiles with the rest of the Policy.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</li>\r\n</ul>\r\nIn the above scenarios, the following disclaimer must be prominently displayed:\r\n\r\n<em>\"[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire literal mark is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\",</em>\r\n\r\n&nbsp;\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>\r\n\r\n<hr />\r\n\r\nYou may use the following logos and graphic Marks in a way that is consistent with the Policy above.\r\n\r\nPowered by ArrayFire Icon\r\n\r\nPowered by ArrayFire Logo','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-02 15:34:51','2016-06-02 19:34:51','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8917,18528,'2016-06-03 13:10:54','2016-06-03 17:10:54','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\r\n\r\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\r\n\r\nThroughout this Policy, the word “Marks” refers to the following:\r\n<ul>\r\n 	<li>The literal mark “ArrayFire”</li>\r\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\r\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\r\n</ul>\r\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\r\n<h4>Fair Use</h4>\r\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\r\n\r\nA few examples of use cases that are clearly fair use are:\r\n<ul>\r\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\r\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\r\n 	<li>To factually refer to an ArrayFire Product.</li>\r\n</ul>\r\n<h4>Use of the Marks without written permission</h4>\r\nSubject to the inclusion of the disclaimer at the bottom of this section, we permit the use of the literal mark, \"ArrayFire\" to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\r\n\r\n&nbsp;\r\n<ul>\r\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\r\n 	<li>When referring to a third-party software that provides interoperability between ArrayFire and a programming language (commonly called \"language wrappers\"), in the format \"ArrayFire.[Language]\" or \"ArrayFire-[Language]\" or [Language] for ArrayFire\" -- provided that such use case otherwise compiles with the rest of the Policy.</li>\r\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</li>\r\n</ul>\r\nIn the above scenarios, the following disclaimer must be prominently displayed:\r\n\r\n<em>\"[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire literal mark is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\",</em>\r\n\r\n&nbsp;\r\n<h4>Prohibited Use</h4>\r\nYou may not use the Marks in the following ways:\r\n<ul>\r\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\r\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\r\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\r\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\r\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\r\n</ul>\r\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\r\n<h4>Reservation of Rights</h4>\r\nArrayFire reserves the sole right to:\r\n<ul>\r\n 	<li>Determine the compliance with this Policy</li>\r\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\r\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\r\n</ul>\r\n<h4>Questions</h4>\r\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>','Trademark Policy','','inherit','closed','closed','','8901-revision-v1','','','2016-06-03 13:10:54','2016-06-03 17:10:54','',8901,'http://sandbox.arrayfire.com/8901-revision-v1/',0,'revision','',0),(8920,17729,'2016-06-08 01:08:19','2016-06-08 05:08:19','In this post, we would like to introduce our latest &amp; newest language bindings for ArrayFire library: ArrayFire-Rust - Rust bindings for ArrayFire. We have been working on the wrapper and using it in daily production within the company for the last few months and now regard it stable enough for public distribution.\r\n<h4><a id=\"How_to_use_arrayfire_from_cratesio_2\"></a>How to use ArrayFire from <a href=\"http://crates.io\">crates.io</a></h4>\r\nTo use the rust bindings for ArrayFire from <strong><a href=\"http://crates.io\">crates.io</a></strong>, the following requirements are to be met first:\r\n<ol>\r\n	<li>Download and install ArrayFire binaries based on your operating system.</li>\r\n	<li>If you are on Linux or OSX, set the environment variable <code>AF_PATH</code> to point to ArrayFire installation root folder. (This is automatically set for you on Windows.)</li>\r\n	<li>Make sure you add the path to library files to your path environment variables.\r\n<ul>\r\n	<li>On Linux &amp; OSX: do <code>export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$AF_PATH/lib</code></li>\r\n	<li>On Windows: Add <code>%AF_PATH%\\lib</code> to your PATH environment variable.</li>\r\n</ul>\r\n</li>\r\n	<li>Run <code>cargo new --bin helloworld &amp;&amp; helloworld</code></li>\r\n	<li>Edit <code>Cargo.toml</code> to add the line <code>arrayfire = \"3.3.0\"</code> to the <code>[dependencies]</code> section.</li>\r\n	<li>Add the following sample rust code to your source file <code>helloworld/src/main.rs</code>.\r\n<pre><code class=\"language-rust\">\r\nextern crate arrayfire as af;\r\nuse af::*; // Ideally, you should use only names you are about to use, for example purposes * is fine.\r\n    \r\nfn main() {\r\n    let num_rows: u64 = 5;\r\n    let num_cols: u64 = 3;\r\n    let dims = Dim4::new(&[num_rows, num_cols, 1, 1]);\r\n    println!(\"Create a 5-by-3 matrix of random floats on the GPU\");\r\n    let a = randu::<f32>(dims);\r\n    print(&a);\r\n}\r\n</code></pre>\r\n</li>\r\n	<li>Run <code>cargo run</code> and you should see output similar to the one below.\r\n<pre><code class=\"language-bash\">\r\nCreate a 5-by-3 matrix of random floats on the GPU\r\n[5 3 1 1]\r\n    0.7402     0.4464     0.7762\r\n    0.9210     0.6673     0.2948\r\n    0.0390     0.1099     0.7140\r\n    0.9690     0.4702     0.3585\r\n    0.9251     0.5132     0.6814\r\n</code></pre>\r\n</li>\r\n</ol>\r\n<h4><a id=\"Documentation_44\"></a>Documentation</h4>\r\nYou can find the documentation for all the functions and macros available in the crate over <a href=\"http://arrayfire.github.io/arrayfire-rust/arrayfire/index.html\">here</a>. You can also look at the ArrayFire C/C++ API documentation <a href=\"http://sandbox.arrayfire.com/docs/group__func__categories.htm\">on ArrayFire.com</a> which categorizes ArrayFire functions into several specific domains.\r\n<h4><a id=\"Example_Project_47\"></a>Example Project</h4>\r\n<a href=\"https://github.com/jramapuram/hal\">hal</a> project is one of the initial adaptors of our rust wrapper for ArrayFire to enable use of GPUs from Rust via ArrayFire.\r\n<h4><a id=\"Credits_50\"></a>Credits</h4>\r\nWe would like to thank <a href=\"https://github.com/jramapuram\" target=\"_blank\">Jason Ramapuram</a> for helping us in writing and reviewing the rust bindings. We would also like to thank the <a href=\"https://users.rust-lang.org/\" target=\"_blank\">users from rust community</a> who provided their invaluable feedback while working on the bindings for ArrayFire.\r\n\r\n&nbsp;','Rust wrapper for ArrayFire','','inherit','closed','closed','','8769-revision-v1','','','2016-06-08 01:08:19','2016-06-08 05:08:19','',8769,'http://sandbox.arrayfire.com/8769-revision-v1/',0,'revision','',0),(8924,18528,'2016-06-19 14:52:40','2016-06-19 18:52:40','The literal mark “ArrayFire” and ArrayFire logos are trademarks of ArrayFire.\n\nTrademarks are used to identify and distinguish the source of goods of one party from those offered by others. As such, a trademark must be used properly and consistently otherwise the mark risks becoming generic, making it impossible to enforce. The following guidelines govern the use of ArrayFire trademarks whether by ArrayFire or third parties, in order to protect the value of ArrayFire trademarks and provide guidance to users on how they should or should not use ArrayFire marks. Unauthorized use of ArrayFire trademarks, or similarly confusing marks, which do not comply with the guidelines below, is prohibited and may constitute trademark infringement. These guidelines to not interfere with your right to use or modify the underlying ArrayFire open source software as permitted under the applicable software license.\n\nThroughout this Policy, the word “Marks” refers to the following:\n<ul>\n 	<li>The literal mark “ArrayFire”</li>\n 	<li>The logos and graphic marks displayed and available for download at <a href=\"http://sandbox.arrayfire.com/trademark\">http://sandbox.arrayfire.com/trademark</a></li>\n 	<li>The slogan “If you want faster code, you’ve come to the right place”</li>\n</ul>\nThroughout this Policy \"ArrayFire Product\" refers to software, source code, services, and training content developed by ArrayFire or otherwise contributed to ArrayFire by third-parties.\n<h4>Fair Use</h4>\nYou may use the Marks without prior written permission to refer to an ArrayFire Product when the Marks are used for the purpose of description or identification of an ArrayFire Product where there is not suggestion of endorsement or sponsorship by the trademark owner or use of the Marks in such a way to confuse the consumer. Such use is generally regarded as \"fair use\" under trademark law.\n\nA few examples of use cases that are clearly fair use are:\n<ul>\n 	<li>To refer to the ArrayFire software in substantially unmodified form.</li>\n 	<li>To identify the ArrayFire software as a distinct component of a software offering where the Mark is not being used in a way which suggests sponsorship or endorsement of the project.</li>\n 	<li>To factually refer to an ArrayFire Product.</li>\n</ul>\n<h4>Use of the Marks without written permission</h4>\nSubject to the inclusion of the disclaimer at the bottom of this section, we permit the use of the literal mark, \"ArrayFire\", to refer to projects, services, or communities outside of ArrayFire Product without written permission in the following contexts:\n\n&nbsp;\n<ul>\n 	<li>When referring to the ArrayFire software that is not substantially unmodified, to say that such software is a “derivative of” or “based on” ArrayFire.</li>\n 	<li>When referring to a third-party software that provides interoperability between ArrayFire and a programming language (commonly called \"language wrappers\"), in the format \"ArrayFire.[Language]\" or \"ArrayFire-[Language]\" or [Language] for ArrayFire\" -- provided that such use case otherwise compiles with the rest of the Policy.</li>\n 	<li>When referring to a third-party software product and/or service that is interoperable with the ArrayFire software, in the format “[Product Name] for ArrayFire” -- provided that such use otherwise complies with the rest of the Policy.</li>\n</ul>\nIn the above scenarios, the following disclaimer must be prominently displayed:\n\n<em>\"[Your project name] is not affiliated with or endorsed by ArrayFire. The ArrayFire literal mark is used under a limited license granted by ArrayFire the trademark holder in the United States and other countries.\",</em>\n\n&nbsp;\n<h4>Prohibited Use</h4>\nYou may not use the Marks in the following ways:\n<ul>\n 	<li><del></del>In any way likely to cause confusion as to the identity of an ArrayFire Product, the provenance of its software, or the software\'s license.</li>\n 	<li>In any way that indicates a greater degree of association between you and the ArrayFire Project or ArrayFire than actually exists.</li>\n 	<li>In any way that implies a designated successor to an ArrayFire Product.</li>\n 	<li>To refer to the ArrayFire software, modified by a third party to the point of inoperability with ArrayFire software in substantially unmodified form.</li>\n 	<li>In any other way that dilutes or otherwise infringes upon the trademark rights of ArrayFire.</li>\n</ul>\nIn addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., “ArrayFirepedia”) used as a mark for a third-party product or service without explicit written permission from ArrayFire. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service\'s use of ArrayFire code.\n<h4>Reservation of Rights</h4>\nArrayFire reserves the sole right to:\n<ul>\n 	<li>Determine the compliance with this Policy</li>\n 	<li>Modify the Policy in ways consistent with the mission of protecting the public.</li>\n 	<li>Grant exceptions to this Policy, of any kind and for any reason whatsoever, other clauses notwithstanding.</li>\n</ul>\n<h4>Questions</h4>\nIf you have any questions regarding this Policy, with to report any misuse of ArrayFire trademarks, or are unsure on whether you require a trademark license, or wish to obtain a license to use ArrayFire Marks, please contact <a href=\"mailto:sale@arrayfire.com\">sale@arrayfire.com</a>','Trademark Policy','','inherit','closed','closed','','8901-autosave-v1','','','2016-06-19 14:52:40','2016-06-19 18:52:40','',8901,'http://sandbox.arrayfire.com/8901-autosave-v1/',0,'revision','',0),(8927,17728,'2016-06-23 13:32:09','2016-06-23 17:32:09','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found ourselves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<table border=\"0\">\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg\" alt=\"server_front\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8959\" /></a>\r\n</td>\r\n\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n</td>\r\n</table>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being passed through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','publish','open','closed','','using-gpus-kvm-virutal-machines','','','2016-06-23 14:56:12','2016-06-23 18:56:12','',0,'http://sandbox.arrayfire.com/?p=8927',0,'post','',2),(8928,17728,'2016-06-22 22:10:07','2016-06-23 02:10:07','<div id=\"content\">\r\n<h1 class=\"title\">Setting Up A Virtualized Test Server</h1>\r\n<div id=\"table-of-contents\">\r\n<h2>Table of Contents</h2>\r\n<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h2 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h2>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h3 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your machine. Make sure you have an SSH or\r\n</li>\r\n</ul>\r\n<p>\r\nVPN login to your server.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h3 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h3 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h2 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h2>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h3 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h3 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h3 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h3 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h3 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h3 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>\r\n<div id=\"postamble\" class=\"status\">\r\n<p class=\"author\">Author: Pavan Yalamanchili</p>\r\n<p class=\"date\">Created: 2016-06-22 Wed 22:05</p>\r\n<p class=\"creator\"><a href=\"http://www.gnu.org/software/emacs/\">Emacs</a> 25.1.50.1 (<a href=\"http://orgmode.org\">Org</a> mode 8.2.10)</p>\r\n<p class=\"validation\"><a href=\"http://validator.w3.org/check?uri=referer\">Validate</a></p>\r\n</div>','Setting up a virtualized server for testing','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:10:07','2016-06-23 02:10:07','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8929,17728,'2016-06-22 22:10:58','2016-06-23 02:10:58','<div id=\"content\">\r\n<div id=\"table-of-contents\">\r\n<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h2 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h2>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h3 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your machine. Make sure you have an SSH or\r\n</li>\r\n</ul>\r\n<p>\r\nVPN login to your server.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h3 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h3 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h2 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h2>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h3 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h3 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h3 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h3 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h3 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h3 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>\r\n<div id=\"postamble\" class=\"status\">\r\n<p class=\"author\">Author: Pavan Yalamanchili</p>\r\n<p class=\"date\">Created: 2016-06-22 Wed 22:05</p>\r\n<p class=\"creator\"><a href=\"http://www.gnu.org/software/emacs/\">Emacs</a> 25.1.50.1 (<a href=\"http://orgmode.org\">Org</a> mode 8.2.10)</p>\r\n<p class=\"validation\"><a href=\"http://validator.w3.org/check?uri=referer\">Validate</a></p>\r\n</div>','Setting up a virtualized server for testing','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:10:58','2016-06-23 02:10:58','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8930,17728,'2016-06-22 22:12:16','2016-06-23 02:12:16','<div id=\"content\">\r\n<div id=\"table-of-contents\">\r\n<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h2 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h2>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h3 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your machine. Make sure you have an SSH or\r\n</li>\r\n</ul>\r\n<p>\r\nVPN login to your server.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h3 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h3 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h2 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h2>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h3 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h3 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h3 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h3 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h3 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h3 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','Setting up a virtualized server for testing','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:12:16','2016-06-23 02:12:16','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8931,17728,'2016-06-22 22:17:24','2016-06-23 02:17:24','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h2 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h2>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h3 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your machine. Make sure you have an SSH or\r\n</li>\r\n</ul>\r\n<p>\r\nVPN login to your server.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h3 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h3 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h3>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h2 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h2>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h3 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h3 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h3 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h3 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h3 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h3 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h3>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','Setting up a virtualized server for testing','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:17:24','2016-06-23 02:17:24','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8932,17728,'2016-06-22 22:21:25','2016-06-23 02:21:25','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h3 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h3>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h5 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your machine. Make sure you have an SSH or\r\n</li>\r\n</ul>\r\n<p>\r\nVPN login to your server.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h5 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h5 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h3 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h3>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h5 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h5 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h5 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','Setting up a virtualized server for testing','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:21:25','2016-06-23 02:21:25','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8933,17728,'2016-06-22 22:23:01','2016-06-23 02:23:01','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h3 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h3>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h5 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>\r\nThis step may break GUI login on your machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n</ul>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h5 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h5 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h3 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h3>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h5 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h5 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h5 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','Setting up a virtualized server for testing','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:23:01','2016-06-23 02:23:01','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8934,17728,'2016-06-22 22:25:22','2016-06-23 02:25:22','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h3 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h3>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h5 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>\r\nThis step may break GUI login on your machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n</ul>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h5 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h5 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h3 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h3>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h5 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h5 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h5 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','Setting up Multi GPU Server with PCI Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:25:22','2016-06-23 02:25:22','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8935,17728,'2016-06-22 22:27:47','2016-06-23 02:27:47','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h3 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h3>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h5 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>\r\nThis step may break GUI login on your machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n</ul>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h5 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h5 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h3 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h3>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h5 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h5 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h5 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','Virtualized Multi GPU Server using KVM and GPU Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:27:47','2016-06-23 02:27:47','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8936,17728,'2016-06-22 22:28:37','2016-06-23 02:28:37','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h3 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h3>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h5 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>\r\nThis step may break GUI login on your machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n</ul>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h5 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h5 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h3 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h3>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h5 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h5 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h5 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','GPU Virtual Machines using KVM and PCI Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:28:37','2016-06-23 02:28:37','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8937,17728,'2016-06-22 22:29:28','2016-06-23 02:29:28','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-1-1\">1.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-1-2\">1.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-1-3\">1.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-2\">2. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Networking</a></li>\r\n<li><a href=\"#sec-2-4\">2.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-2-5\">2.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-2-6\">2.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h3 id=\"sec-1\"><span class=\"section-number-2\">1</span> Setting up PCI passthrough</h3>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nThis document describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-1\" class=\"outline-3\">\r\n<h5 id=\"sec-1-1\"><span class=\"section-number-3\">1.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>\r\nThis step may break GUI login on your machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n</ul>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-2\" class=\"outline-3\">\r\n<h5 id=\"sec-1-2\"><span class=\"section-number-3\">1.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1-3\" class=\"outline-3\">\r\n<h5 id=\"sec-1-3\"><span class=\"section-number-3\">1.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-1-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h3 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up the virtual machines</h3>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis document describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-4\" class=\"outline-3\">\r\n<h5 id=\"sec-2-4\"><span class=\"section-number-3\">2.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-5\" class=\"outline-3\">\r\n<h5 id=\"sec-2-5\"><span class=\"section-number-3\">2.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-6\" class=\"outline-3\">\r\n<h5 id=\"sec-2-6\"><span class=\"section-number-3\">2.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n</div>','Virtual Machines with GPUs using KVM and PCI Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-22 22:29:28','2016-06-23 02:29:28','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8938,17728,'2016-06-23 00:26:19','2016-06-23 04:26:19','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Introduction</a></li>\r\n<li><a href=\"#sec-2\">2. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-3\">3. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-3-1\">3.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-3-2\">3.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-3-3\">3.3. Networking</a></li>\r\n<li><a href=\"#sec-3-4\">3.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-3-5\">3.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-3-6\">3.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-4\">4. Conclusion</a></li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\"><span class=\"section-number-2\">1</span> Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems.\r\nAround the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs.\r\nBuilding a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a>\r\nfrom Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b>\r\n</p>\r\n<ul class=\"org-ul\">\r\n<li>To AMD for providing us with Firepro GPUs used in this build and more.\r\n</li>\r\n<li>To NVIDIA for providing us with Tesla GPUs used in this build and more.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.\r\nA short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\"><span class=\"section-number-2\">3</span> Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\"><span class=\"section-number-3\">3.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\"><span class=\"section-number-3\">3.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\"><span class=\"section-number-3\">3.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\"><span class=\"section-number-3\">3.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\"><span class=\"section-number-3\">3.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\"><span class=\"section-number-3\">3.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\"><span class=\"section-number-2\">4</span> Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations.\r\n</p>\r\n\r\n<p>\r\nWe plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Virtual Machines with GPUs using KVM and PCI Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 00:26:19','2016-06-23 04:26:19','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8939,17728,'2016-06-23 00:27:24','2016-06-23 04:27:24','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Introduction</a></li>\r\n<li><a href=\"#sec-2\">2. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-3\">3. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-3-1\">3.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-3-2\">3.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-3-3\">3.3. Networking</a></li>\r\n<li><a href=\"#sec-3-4\">3.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-3-5\">3.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-3-6\">3.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-4\">4. Conclusion</a></li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\"><span class=\"section-number-2\">1</span> Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems.\r\nAround the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs.\r\nBuilding a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a>\r\nfrom Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b>\r\n</p>\r\n<ul class=\"org-ul\">\r\n<li><b>To AMD</b> for providing us with Firepro GPUs used in this build and more.\r\n</li>\r\n<li><b>To NVIDIA</b> for providing us with Tesla GPUs used in this build and more.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.\r\nA short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\"><span class=\"section-number-2\">3</span> Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\"><span class=\"section-number-3\">3.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\"><span class=\"section-number-3\">3.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\"><span class=\"section-number-3\">3.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\"><span class=\"section-number-3\">3.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\"><span class=\"section-number-3\">3.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\"><span class=\"section-number-3\">3.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\"><span class=\"section-number-2\">4</span> Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations.\r\n</p>\r\n\r\n<p>\r\nWe plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Virtual Machines with GPUs using KVM and PCI Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 00:27:24','2016-06-23 04:27:24','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0);
INSERT INTO `wp_posts` VALUES (8940,17728,'2016-06-23 00:31:50','2016-06-23 04:31:50','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Introduction</a></li>\r\n<li><a href=\"#sec-2\">2. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-3\">3. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-3-1\">3.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-3-2\">3.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-3-3\">3.3. Networking</a></li>\r\n<li><a href=\"#sec-3-4\">3.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-3-5\">3.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-3-6\">3.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-4\">4. Conclusion</a></li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\"><span class=\"section-number-2\">1</span> Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems.\r\nAround the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs.\r\nBuilding a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a>\r\nfrom Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b>\r\n</p>\r\n<ul class=\"org-ul\">\r\n<li><b>To AMD</b> for providing us with Firepro GPUs used in this build and more.\r\n</li>\r\n<li><b>To NVIDIA</b> for providing us with Tesla GPUs used in this build and more.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.\r\nA short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\"><span class=\"section-number-2\">3</span> Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\"><span class=\"section-number-3\">3.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\"><span class=\"section-number-3\">3.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\"><span class=\"section-number-3\">3.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\"><span class=\"section-number-3\">3.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\"><span class=\"section-number-3\">3.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\"><span class=\"section-number-3\">3.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\"><span class=\"section-number-2\">4</span> Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations.\r\n</p>\r\n\r\n<p>\r\nWe plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Virtual Machines with GPUs using KVM and PCI Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 00:31:50','2016-06-23 04:31:50','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8941,17728,'2016-06-23 00:33:41','2016-06-23 04:33:41','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Introduction</a></li>\r\n<li><a href=\"#sec-2\">2. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-3\">3. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-3-1\">3.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-3-2\">3.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-3-3\">3.3. Networking</a></li>\r\n<li><a href=\"#sec-3-4\">3.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-3-5\">3.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-3-6\">3.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-4\">4. Conclusion</a></li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\"><span class=\"section-number-2\">1</span> Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems.\r\nAround the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs.\r\nBuilding a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a>\r\nfrom Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b>\r\n</p>\r\n<ul class=\"org-ul\">\r\n<li><b>To AMD</b> for providing us with Firepro GPUs used in this build and more.\r\n</li>\r\n<li><b>To NVIDIA</b> for providing us with Tesla GPUs used in this build and more.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.\r\nA short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\"><span class=\"section-number-2\">3</span> Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\"><span class=\"section-number-3\">3.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\"><span class=\"section-number-3\">3.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\"><span class=\"section-number-3\">3.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\"><span class=\"section-number-3\">3.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can query the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\"><span class=\"section-number-3\">3.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\"><span class=\"section-number-3\">3.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\"><span class=\"section-number-2\">4</span> Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations.\r\n</p>\r\n\r\n<p>\r\nWe plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Virtual Machines with GPUs using KVM and PCI Passthrough','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 00:33:41','2016-06-23 04:33:41','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8943,17728,'2016-06-23 10:52:23','2016-06-23 14:52:23','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Introduction</a></li>\r\n<li><a href=\"#sec-2\">2. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-3\">3. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-3-1\">3.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-3-2\">3.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-3-3\">3.3. Networking</a></li>\r\n<li><a href=\"#sec-3-4\">3.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-3-5\">3.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-3-6\">3.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-4\">4. Conclusion</a></li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\"><span class=\"section-number-2\">1</span> Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems.\r\nAround the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs.\r\nBuilding a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a>\r\nfrom Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b>\r\n</p>\r\n<ul class=\"org-ul\">\r\n<li><b>To AMD</b> for providing us with Firepro GPUs used in this build and more.\r\n</li>\r\n<li><b>To NVIDIA</b> for providing us with Tesla GPUs used in this build and more.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.\r\nA short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\"><span class=\"section-number-2\">3</span> Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\"><span class=\"section-number-3\">3.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\"><span class=\"section-number-3\">3.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\"><span class=\"section-number-3\">3.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\"><span class=\"section-number-3\">3.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,passowrd=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can query the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\"><span class=\"section-number-3\">3.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\"><span class=\"section-number-3\">3.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\"><span class=\"section-number-2\">4</span> Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations.\r\n</p>\r\n\r\n<p>\r\nWe plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 10:52:23','2016-06-23 14:52:23','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8944,17728,'2016-06-23 11:11:44','2016-06-23 15:11:44','<div id=\"text-table-of-contents\">\r\n<ul>\r\n<li><a href=\"#sec-1\">1. Introduction</a></li>\r\n<li><a href=\"#sec-2\">2. Setting up PCI passthrough</a>\r\n<ul>\r\n<li><a href=\"#sec-2-1\">2.1. Setup machine for virtualization</a></li>\r\n<li><a href=\"#sec-2-2\">2.2. Adding Devices for PCI passthrough</a></li>\r\n<li><a href=\"#sec-2-3\">2.3. Setting up QEMU + KVM + libvirt for PCI passthrough</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-3\">3. Setting up the virtual machines</a>\r\n<ul>\r\n<li><a href=\"#sec-3-1\">3.1. Creating a storage pool</a></li>\r\n<li><a href=\"#sec-3-2\">3.2. Creating a volume</a></li>\r\n<li><a href=\"#sec-3-3\">3.3. Networking</a></li>\r\n<li><a href=\"#sec-3-4\">3.4. Installing an OS</a></li>\r\n<li><a href=\"#sec-3-5\">3.5. Accessing the VMs</a></li>\r\n<li><a href=\"#sec-3-6\">3.6. Cloning the VMs</a></li>\r\n</ul>\r\n</li>\r\n<li><a href=\"#sec-4\">4. Conclusion</a></li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\"><span class=\"section-number-2\">1</span> Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<b>Special Thanks to AMD and NVIDIA</b> for providing us with the Firepro GPUs used in this build.\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.\r\nA short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\"><span class=\"section-number-2\">2</span> Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\"><span class=\"section-number-3\">2.1</span> Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\"><span class=\"section-number-3\">2.2</span> Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\"><span class=\"section-number-3\">2.3</span> Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\"><span class=\"section-number-2\">3</span> Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\"><span class=\"section-number-3\">3.1</span> Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\"><span class=\"section-number-3\">3.2</span> Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\"><span class=\"section-number-3\">3.3</span> Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\"><span class=\"section-number-3\">3.4</span> Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can query the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\"><span class=\"section-number-3\">3.5</span> Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\"><span class=\"section-number-3\">3.6</span> Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\"><span class=\"section-number-2\">4</span> Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system. We also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>\r\n','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 11:11:44','2016-06-23 15:11:44','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8945,17728,'2016-06-23 12:30:00','2016-06-23 16:30:00','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU.\r\nA short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend.\r\nThis can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking.\r\nIf you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface through <code>libvirt</code>\r\nwhich can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>\r\n','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 12:30:00','2016-06-23 16:30:00','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8946,17728,'2016-06-23 12:43:58','2016-06-23 16:43:58','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/9100\">AMD Firepro W9100</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 12:43:58','2016-06-23 16:43:58','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8949,17728,'2016-06-23 13:16:54','2016-06-23 17:16:54','','server_7_gpus','','inherit','open','closed','','server_7_gpus-2','','','2016-06-23 13:16:54','2016-06-23 17:16:54','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus-1.jpg',0,'attachment','image/jpeg',0),(8950,17728,'2016-06-23 13:17:26','2016-06-23 17:17:26','','server_7_gpus_top','','inherit','open','closed','','server_7_gpus_top','','','2016-06-23 13:17:26','2016-06-23 17:17:26','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top.jpg',0,'attachment','image/jpeg',0),(8951,17728,'2016-06-23 13:17:52','2016-06-23 17:17:52','','server_gpus','','inherit','open','closed','','server_gpus','','','2016-06-23 13:17:52','2016-06-23 17:17:52','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus.jpg',0,'attachment','image/jpeg',0),(8952,17728,'2016-06-23 13:18:10','2016-06-23 17:18:10','','server_gpus','','inherit','open','closed','','server_gpus-2','','','2016-06-23 13:18:10','2016-06-23 17:18:10','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-1.jpg',0,'attachment','image/jpeg',0),(8953,17728,'2016-06-23 13:18:36','2016-06-23 17:18:36','','server_gpus','','inherit','open','closed','','server_gpus-3','','','2016-06-23 13:18:36','2016-06-23 17:18:36','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-2.jpg',0,'attachment','image/jpeg',0),(8954,17728,'2016-06-23 13:19:24','2016-06-23 17:19:24','From left to right, K40, K10, K80, S9150, S9150, S9150, S7150','GPU Server','','inherit','open','closed','','server_gpus-4','','','2016-06-23 13:22:00','2016-06-23 17:22:00','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-3.jpg',0,'attachment','image/jpeg',0),(8955,17728,'2016-06-23 13:20:05','2016-06-23 17:20:05','','server_7_gpus_top','','inherit','open','closed','','server_7_gpus_top-2','','','2016-06-23 13:20:05','2016-06-23 17:20:05','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg',0,'attachment','image/jpeg',0),(8956,17728,'2016-06-23 13:23:18','2016-06-23 17:23:18','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-3.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-3-300x225.jpg\" alt=\"GPU Server\" width=\"300\" height=\"225\" class=\"alignnone size-medium wp-image-8954\" /></a>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:23:18','2016-06-23 17:23:18','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8957,17728,'2016-06-23 13:23:55','2016-06-23 17:23:55','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-3.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-3-300x225.jpg\" alt=\"GPU Server\" width=\"400\" height=\"350\" class=\"alignnone size-medium wp-image-8954\" /></a>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:23:55','2016-06-23 17:23:55','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8958,17728,'2016-06-23 13:24:43','2016-06-23 17:24:43','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-3.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_gpus-3-300x225.jpg\" alt=\"GPU Server\" width=\"300\" height=\"225\" class=\"alignnone size-medium wp-image-8954\" /></a>\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:24:43','2016-06-23 17:24:43','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8959,17728,'2016-06-23 13:25:40','2016-06-23 17:25:40','','server_front','','inherit','open','closed','','server_front','','','2016-06-23 13:26:22','2016-06-23 17:26:22','',8927,'http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg',0,'attachment','image/jpeg',0),(8960,17728,'2016-06-23 13:26:24','2016-06-23 17:26:24','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg\" alt=\"server_front\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8959\" /></a>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:26:24','2016-06-23 17:26:24','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8961,17728,'2016-06-23 13:28:36','2016-06-23 17:28:36','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<table>\r\n<tr>\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n</td>\r\n\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg\" alt=\"server_front\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8959\" /></a>\r\n</td>\r\n</tr>\r\n</table>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:28:36','2016-06-23 17:28:36','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8962,17728,'2016-06-23 13:29:25','2016-06-23 17:29:25','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<table border=\"1\">\r\n<tr>\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n</td>\r\n\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg\" alt=\"server_front\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8959\" /></a>\r\n</td>\r\n</tr>\r\n</table>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:29:25','2016-06-23 17:29:25','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8963,17728,'2016-06-23 13:30:06','2016-06-23 17:30:06','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<table border=\"0\">\r\n<tr>\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n</td>\r\n\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg\" alt=\"server_front\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8959\" /></a>\r\n</td>\r\n</tr>\r\n</table>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:30:06','2016-06-23 17:30:06','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8964,17728,'2016-06-23 13:30:49','2016-06-23 17:30:49','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<table border=\"0\">\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n</td>\r\n\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg\" alt=\"server_front\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8959\" /></a>\r\n</td>\r\n</table>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:30:49','2016-06-23 17:30:49','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),(8965,17728,'2016-06-23 13:31:09','2016-06-23 17:31:09','<div id=\"outline-container-sec-1\" class=\"outline-2\">\r\n<h4 id=\"sec-1\">Introduction</h4>\r\n<div class=\"outline-text-2\" id=\"text-1\">\r\n<p>\r\nA couple of months ago, I began investigating GPU passthrough on my workstation to test ArrayFire on different operating systems. Around the same time, we at ArrayFire found oursleves with a few surplus GPUs.\r\n</p>\r\n\r\n<p>\r\nHaving had great success with my virtualization efforts, we decided to build a Virtualized GPU Server to utilize these GPUs. Building a Virtualized GPU Server alleviated one of the pain points at our company: We no longer need to swap GPUs or Hard Disks to test a new environment.\r\n</p>\r\n\r\n<p>\r\nTo maximize the number of GPUs we can put in a machine, we ended up getting a <a href=\"http://exxactcorp.com/index.php/solution/solu_detail/207\">Quantum TXR430-0768R</a> from Exxact Computing which comes in a 4U form factor and supports upto 8x double width GPUs.\r\n</p>\r\n\r\n<table border=\"0\">\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_front-300x225.jpg\" alt=\"server_front\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8959\" /></a>\r\n</td>\r\n\r\n<td>\r\n<a href=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1.jpg\"><img src=\"http://sandbox.arrayfire.com/wp-content/uploads/2016/06/server_7_gpus_top-1-300x225.jpg\" alt=\"server_7_gpus_top\" width=\"300\" height=\"225\" class=\"aligncenter size-medium wp-image-8955\" /></a>\r\n</td>\r\n</table>\r\n\r\n<p>\r\nThe GPUs used for this build include:\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/workstation/firepro-remote-graphics/s9150\">AMD Firepro S9150</a> x3\r\n</li>\r\n<li><a href=\"http://www.amd.com/en-us/products/graphics/server/s7150\">AMD Firepro S7150</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-k80.html\">NVIDIA Tesla K80</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/object/tesla-workstations.html\">NVIDIA Tesla K40</a>\r\n</li>\r\n<li><a href=\"http://www.nvidia.com/content/tesla/pdf/tesla-kseries-overview-lr.pdf\">NVIDIA Tesla K10</a> (PDF)\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Special Thanks</b> to AMD and NVIDIA for providing us with the GPUs used in this build.\r\n</p>\r\n\r\n<p>\r\n<b>Note:</b> If you are planning to purchase hardware for PCI passthrough, ensure both your motherboard and the processor support IOMMU. A short (but by no means comprehensive) list <a href=\"https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware\">can be found here</a>.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2\" class=\"outline-2\">\r\n<h4 id=\"sec-2\">Setting up PCI passthrough</h4>\r\n<div class=\"outline-text-2\" id=\"text-2\">\r\n<p>\r\nThis section describes how to setup PCI passthrough on Ubuntu 16.04. This\r\nis based on <a href=\"https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF\">this arch wiki article</a> about the same topic.\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-1\" class=\"outline-3\">\r\n<h5 id=\"sec-2-1\">Setup machine for virtualization</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-1\">\r\n<ul class=\"org-ul\">\r\n<li>Add <code>intel_iommu=on</code> to the kernel parameter in grub\r\n</li>\r\n<li>Edit <code>/etc/default/grub</code> and add <code>intel_iommu=on</code> (or <code>amd_iommu=on</code>, if you have AMD) to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> line.\r\n</li>\r\n<li>Make sure you create the new <code>grub.cfg</code> file by running the following command\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo grub-mkconfig -o /boot/grub/grub.cfg\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Blacklist <code>nouveau</code>, <code>radeon</code>, <code>amdgpu</code>\r\n</li>\r\n<li>create a file called <code>/etc/modprobe.d/blacklist-gpu.conf</code> and add the following lines\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nblacklist nouveau\r\nblacklist radeon\r\nblacklist amdgpu\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>This step may break GUI login on your server machine. Make sure you have an SSH or VNC login to your server.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-2\" class=\"outline-3\">\r\n<h5 id=\"sec-2-2\">Adding Devices for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-2\">\r\n<ul class=\"org-ul\">\r\n<li>Run the following command to get the list of IOMMU groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo \"IOMMU group $(basename \"$iommu_group\")\"; for device in $(ls -1 \"$iommu_group\"/devices/); do echo -n $\'\\t\'; lspci -nns \"$device\"; done; done\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>The PCI IDs are of the form <code>[abcd:xyzw]</code>\r\n</li>\r\n\r\n<li>Add the PCI ids you want to pass through to <code>/etc/modprobe.d/vfio.conf</code>. Create the file if it does not exist.\r\n</li>\r\n\r\n<li>A sample configuration is given below.\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\noptions vfio-pci ids=10de:1024,10de:102d\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the following vfio modules to <code>/etc/initramfs-tools/modules</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvfio\r\nvfio_iommu_type1\r\nvfio_pci\r\nvfio_virqfd\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run <code>sudo update-initramfs -u</code> to generate the new initrd image.\r\n</li>\r\n\r\n<li>Reboot\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-2-3\" class=\"outline-3\">\r\n<h5 id=\"sec-2-3\">Setting up QEMU + KVM + libvirt for PCI passthrough</h5>\r\n<div class=\"outline-text-3\" id=\"text-2-3\">\r\n<ul class=\"org-ul\">\r\n<li>Install the required virtualization packages\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo apt-get install qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin virt-manager\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Add the user to the <code>kvm</code> and <code>libvirtd</code> groups\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">sudo usermod -a -G kvm userid\r\nsudo usermod -a -G libvirtd userid\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/libvirt/qemu.conf</code> and add (or uncomment) the following line if not already present\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nnvram = [    \"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd\"\r\n]\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>restart libvrit-bin service\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsudo systemctl restart libvirt-bin\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You are now ready to pass through the GPU!\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3\" class=\"outline-2\">\r\n<h4 id=\"sec-3\">Setting up the virtual machines</h4>\r\n<div class=\"outline-text-2\" id=\"text-3\">\r\n<p>\r\nThis section describes how to set up and maintain VMs using <code>libvirt</code> in the frontend and <code>qemu+kvm</code> in the backend. This can be achieved by either using the command line (<code>virsh</code>) and GUI (<code>virt-manager</code>).\r\n</p>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-1\" class=\"outline-3\">\r\n<h5 id=\"sec-3-1\">Creating a storage pool</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-1\">\r\n<p>\r\nIf you plan on using the default pool, skip this section.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a pool from command line using <code>virsh</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/ch13s03s03.html\">follow these instructions</a>.\r\n</li>\r\n<li>For creating a pool using <code>virt-manager</code>, <a href=\"https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Virtualization-Storage_Pools-Creating-Local_Directories.html#idp13442168\">follow these instructions</a>.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-2\" class=\"outline-3\">\r\n<h5 id=\"sec-3-2\">Creating a volume</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-2\">\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume from the command line using <code>virsh</code>, use the following commands\r\n</li>\r\n</ul>\r\n\r\n<div class=\"org-src-container\">\r\n\r\n<pre class=\"src src-sh\">virsh vol-create-as --pool my-storage-pool --name disk-01.qcow2 --capacity 80G --format qcow2\r\n</pre>\r\n</div>\r\n\r\n<ul class=\"org-ul\">\r\n<li>For creating a volume using <code>virt-manager</code>, double click on your connection,\r\n</li>\r\n</ul>\r\n<p>\r\nnavigate to storage, choose your pool, press <code>+</code> and follow the instructions.\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-3\" class=\"outline-3\">\r\n<h5 id=\"sec-3-3\">Networking</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-3\">\r\n<p>\r\n<code>libvirt</code> defaults to creating a NAT interface for networking. If you want to use networking through a NAT interface, you can skip this section.\r\n</p>\r\n\r\n<p>\r\nThe rest of the section deals with creating a bridge interface which can later be used by the VMs.\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Edit <code>/etc/network/interfaces</code>\r\n</li>\r\n<li>Add the following lines towards the end. Use the appropriate network interface instead of <code>eth1</code>\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nauto br0\r\niface br0 inet dhcp\r\n    bridge_ports eth1\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Restart the network\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nsystemctl restart networking network-manager\r\n</pre>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-4\" class=\"outline-3\">\r\n<h5 id=\"sec-3-4\">Installing an OS</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-4\">\r\n<p>\r\n<b>Installing an OS from command line</b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Run a command similar to the following\r\n</li>\r\n</ul>\r\n\r\n<pre class=\"example\">\r\nvirt-install --name test-vm-01 --memory 8192\\\r\n--vcpus 4 --disk /my/storage/location/disk-01.qcow2\\\r\n--graphics vnc,password=foobar,listen=0.0.0.0\\\r\n--cdrom /path/to/os-install.iso\\\r\n--network bridge=br0\\\r\n--host-device=pci_abcd_vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note the pci id passed to <code>host-device</code> is in a slightly different format than the output from <code>lspci</code>\r\n</li>\r\n<li>The <code>vw_xy_z</code> format is extracted from the output of <code>lspci</code> which returns the ids in <code>vw:xy.z</code> format.\r\n</li>\r\n<li>You can now run the following command to get the appropriate pci id for <code>virsh</code>\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh nodedev-list | grep vw_xy_z\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Note, <code>listen=0.0.0.0</code> makes the vm visible to everyone. Mask it to the required subnets appropriately.\r\n</li>\r\n<li>change the password to be something more appropriate.\r\n</li>\r\n\r\n<li>Once the command is run, you can queiry the <code>vncdisplay</code> using the following command:\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirsh vncdisplay test-vm-01\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>You can then use this information to connect to the server using a vnc client of choice\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<b>Installing an OS using <code>virt-manager</code></b>\r\n</p>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Click on create new virtual machine (top left icon)\r\n</li>\r\n<li><code>Local install media</code>. Browse and choose install media\r\n</li>\r\n<li>Optional: Set OS type and version appropriately\r\n</li>\r\n<li>Choose the desired amount of RAM and CPU cores\r\n</li>\r\n<li>Expand <code>Network selection</code> and make sure the Bridge <code>br0</code> is selected\r\n</li>\r\n<li><code>Select or create custom storage</code>. Browse to the storage disk created earlier\r\n</li>\r\n<li>Check the box that says <b>Customize configuration before install</b>\r\n</li>\r\n<li>Click on <code>Add Hardware</code>\r\n</li>\r\n<li>Choose <code>PCI Host Device</code> and select the device you want to pass through\r\n</li>\r\n<li>Click <code>Begin Installation</code> on top left corner.\r\n</li>\r\n</ul>\r\n\r\n<p>\r\n<i>If the GPU being pass through has audio support, you need to add it to the VM as well.</i>\r\n</p>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-5\" class=\"outline-3\">\r\n<h5 id=\"sec-3-5\">Accessing the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-5\">\r\n<ul class=\"org-ul\">\r\n<li>If you setup the VM to use bridge networking, it can be accessed directly via the ip within the LAN.\r\n</li>\r\n<li>If you setup the VM to use VNC, you can access the VM using any VNC client software.\r\n</li>\r\n<li>If you are on Linux, you can install <code>virt-manager</code> and connect to the remote server and all of it\'s VMs.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-3-6\" class=\"outline-3\">\r\n<h5 id=\"sec-3-6\">Cloning the VMs</h5>\r\n<div class=\"outline-text-3\" id=\"text-3-6\">\r\n<ul class=\"org-ul\">\r\n<li>From command line run the following command\r\n</li>\r\n</ul>\r\n<pre class=\"example\">\r\nvirt-clone --original original-vm-name --name new-vm-name --file /path/to/new/disk.qcow2\r\n</pre>\r\n\r\n<ul class=\"org-ul\">\r\n<li>Alternatively, from <code>virt-manager</code> simply right click on the vm and choose clone.\r\n</li>\r\n</ul>\r\n</div>\r\n</div>\r\n</div>\r\n\r\n<div id=\"outline-container-sec-4\" class=\"outline-2\">\r\n<h4 id=\"sec-4\">Conclusion</h4>\r\n<div class=\"outline-text-2\" id=\"text-4\">\r\n<p>\r\nAlthough time consuming, this build has been very fruitful for us. It allows us to seamlessly snapshot, clone, restore, and switch between various configurations. We plan on doing interesting things with this setup including integrating it with our <a href=\"http://ci.arrayfire.org/\">continuous integration</a> system.\r\n</p>\r\n\r\n<p>\r\nWe also have other interesting ideas brewing internally to leverage this infrastructure. Follow this blog for updates in this regard!\r\n</p>\r\n</div>\r\n</div>\r\n</div>','Using GPUs in KVM Virtual Machines','','inherit','closed','closed','','8927-revision-v1','','','2016-06-23 13:31:09','2016-06-23 17:31:09','',8927,'http://sandbox.arrayfire.com/8927-revision-v1/',0,'revision','',0),