Programmers and Data Scientists want to take advantage of fast and parallel computational devices. One of the best ways of doing this is to write vectorized code; however, this is often easier said than done. In today's continuation of the blog series "<a href="http://sandbox.arrayfire.com/tag/learning-arrayfire-from-scratch/">Learning ArrayFire from scratch</a>", we will discuss the various methods by which you can maximize the performance of your code by using ArrayFire's built-in vectorization features.

<!--more-->

ArrayFire provides many ways to vectorize a given code segment. In this tutorial, we present several methods to vectorize code using ArrayFire and discuss the benefits and drawbacks associated with each method.
<h4><a id="GenericDefault_vectorization_10"></a>Generic/Default vectorization</h4>
By its very nature, ArrayFire is a vectorized library. Most functions operate on arrays as a whole – on all elements in parallel. Wherever possible, existing vectorized functions should be used opposed to manually indexing into arrays. For example consider the following code:
<pre class="lang:default decode:true ">af::array a = af::range(10); // [0,  9]
for(int i = 0; i &lt; a.dims(0); ++i)
{
    a(i) = a(i) + 1;         // [1, 10]
}</pre>
Although completely valid, the code is very inefficient as it results in a kernel that operate on one datum. Instead, the developer should have used ArrayFire's overload of the + operator:
<pre><code class="language-{.cpp}">
af::array a = af::range(10);  // [0,  9]
a = a + 1;                    // [1, 10]
</code></pre>
This code will result in a single kernel that operates on all 10 elements of <code>a</code> in parallel.

Most ArrayFire functions are vectorized. A small subset of these include:

<strong>Arithmetic operations:</strong> +, -, *, / , %, &gt;&gt;, &lt;&lt;
<strong>Logical operations:</strong> &amp;&amp;, ||, &lt;, &gt;, ==, !=
<strong>Numeric functions:</strong> abs(), floor(), round(), min(), max()…
<strong>Complex operations:</strong> real(), imag(), conj()…
<strong>Exponential and logarithmic functions:</strong> exp(), log(), expm1(), log1p()…
<strong>Trigonometric functions:</strong> sin(), cos(), tan()…
<strong>Hyperbolic functions:</strong> sinh(), cosh(), tanh()…

In addition to element-wise operations, many other functions are also vectorized in ArrayFire.

Notice that even that perform some form of aggregation (e.g. <code>sum()</code> or <code>min()</code>), signal processing (like <code>convolve()</code>), and even image processing functions (i.e. <code>rotate()</code>) all support vectorization on different columns or images. For example, if we have <code>NUM</code> images of size <code>WIDTH</code> by <code>HEIGHT</code>, one could convolve each image in a vector fashion as follows:
<pre class="lang:default decode:true ">float g_coef[] = { 1, 2, 1,
                   2, 4, 2,
                   1, 2, 1 };
af::array filter = 1.f/16 * af::array(3, 3, f_coef);
af::array signal = randu(WIDTH, HEIGHT, NUM);
af::array conv = convolve2(signal, filter);</pre>
Similarly, one can rotate 100 images by 45 degrees in a single call using code like the following:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">// Construct an array of 100 WIDTH x HEIGHT images of random numbers
af::array imgs = randu(WIDTH, HEIGHT, 100);
// Rotate all of the images in a single command
af::array rot_imgs = rotate(imgs, 45);</pre>
</div>
</div>
Although <em>most</em> functions in ArrayFire do support vectorization, some do not. Most notably, all linear algebra functions. Even though they are not vectorized linear algebra operations still execute in parallel on your hardware.

Using the built in vectorized operations should be the first and preferred method of vectorizing any code written with ArrayFire.
<h4><a id="GFOR_Parallel_forloops_76"></a>GFOR: Parallel for-loops</h4>
Another novel method of vectorization present in ArrayFire is the GFOR loop replacement construct. GFOR allows launching all iterations of a loop in parallel on the GPU or device, as long as the iterations are independent. While the standard for-loop performs each iteration sequentially, ArrayFire's gfor-loop performs each iteration at the same time (in parallel). ArrayFire does this by tiling out the values of all loop iterations and then performing computation on those tiles in one pass. You can think of gfor as performing auto-vectorization of your code, e.g. you write a gfor-loop that increments every element of a vector but behind the scenes ArrayFire rewrites it to operate on the entire vector in parallel.

The original for-loop example at the beginning of this document could be rewritten using GFOR as follows:
<pre><code class="language-{.cpp}">af::array a = af::range(10);
gfor(seq i, n)
    a(i) = a(i) + 1;</code></pre>
In this case, each instance of the gfor loop is independent, thus ArrayFire will automatically tile out the <code>a</code> array in device memory and execute the increment kernels in parallel.

To see another example, you could run an <a class="el" title="C++ Interface exclusive sum (cumulative sum) of an array. " href="group__scan__func__accum.htm#gae450135c4ae1bdaa4489804a215f3214">accum()</a> on every slice of a matrix in a for-loop, or you could "vectorize" and simply do it all in one gfor-loop operation:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">// runs each accum() in sequence
for (int i = 0; i &lt; N; ++i)
    B(span,i) = accum(A(span,i));
 
// runs N accums in parallel
gfor (seq i, N)
    B(span,i) = accum(A(span,i));</pre>
</div>
</div>
However, returning to our previous vectorization technique, <a class="el" title="C++ Interface exclusive sum (cumulative sum) of an array. " href="group__scan__func__accum.htm#gae450135c4ae1bdaa4489804a215f3214">accum()</a> is already vectorized and the operation could be completely replaced with merely:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">B = accum(A);</pre>
</div>
</div>
It is best to vectorize computation as much as possible to avoid the overhead in both for-loops and gfor-loops. However, the gfor-loop construct is most effective in the narrow case of broadcast-style operations. Consider the case when we have a vector of constants that we wish to apply to a collection of variables, such as expressing the values of a linear combination for multiple vectors. The broadcast of one set of constants to many vectors works well with gfor-loops:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">const static int p=4, n=1000;
af::array consts = af::randu(p);
af::array var_terms = randn(p, n);
 
gfor(seq i, n)
    combination(span, i) = consts * var_terms(span, i);</pre>
</div>
</div>
Using GFOR requires following several rules and multiple guidelines for optimal performance. The details of this vectorization method can be found in the <a class="el" href="page_gfor.htm#gfor">GFOR documentation</a>.
<h4><a id="Batching_127"></a>Batching</h4>
The <a class="el" href="namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b">batchFunc()</a> function allows the broad application of existing ArrayFire functions to multiple sets of data. Effectively,<a class="el" href="namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b">batchFunc()</a> allows ArrayFire functions to execute in "batch processing" mode. In this mode, functions will find a dimension which contains "batches" of data to be processed and will parallelize the procedure.

Consider the following example. Here we create a filter which we would like to apply to each of the weight vectors. The naive solution would be using a for-loop as we have seen previously:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">// Create the filter and the weight vectors
af::array filter = randn(1, 5);
af::array weights = randu(5, 5);
 
// Apply the filter using a for-loop
af::array filtered_weights = constant(0, 5, 5);
for(int i=0; i&lt;weights.dims(1); ++i){
    filtered_weights.col(i) = filter * weights.col(i);
}</pre>
</div>
</div>
However, as we have discussed above, this solution will be very inefficient. One may be tempted to implement a vectorized solution as follows:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">// Create the filter and the weight vectors
af::array filter = randn(1, 5);
af::array weights = randu(5, 5);
 
af::array filtered_weights = filter * weights; // fails due to dimension mismatch</pre>
</div>
</div>
However, the dimensions of <code>filter</code> and <code>weights</code> do not match, thus ArrayFire will generate a runtime error.

<code>batchfunc()</code> was created to solve this specific problem. The signature of the function is as follows:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">array batchFunc(const array &amp;lhs, const array &amp;rhs, batchFunc_t func);</pre>
</div>
</div>
where <code>batchFunc_t</code> is a function pointer of the form:
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">typedef array (*batchFunc_t) (const array &amp;lhs, const array &amp;rhs);</pre>
</div>
</div>
So, to use <a class="el" href="namespaceaf.htm#aa0eb9e160f5be4b95234543e5c47934b">batchFunc()</a>, we need to provide the function we wish to apply as a batch operation. For illustration's sake, let's "implement" a multiplication function following the format.
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">af::array my_mult (const af::array &amp;lhs, const af::array &amp;rhs){
return lhs * rhs;
}</pre>
</div>
</div>
Our final batch call is not much more difficult than the ideal syntax we imagined.
<div class="fragment">
<div class="line">
<pre class="lang:default decode:true ">// Create the filter and the weight vectors
af::array filter = randn(1, 5);
af::array weights = randu(5, 5);
 
// Apply the batch function
af::array filtered_weights = batchFunc( filter, weights, my_mult );</pre>
</div>
</div>
The batch function will work with many previously mentioned vectorized ArrayFire functions. It can even work with a combination of those functions if they are wrapped inside a helper function matching the <code>__batchFunc_t__</code> signature. One limitation of <code>batchfunc()</code> is that it cannot be used from within a <code><a class="el" href="gfor_8h.htm#a6d6cd11d53daad217fd0b92f5a7ceec6">gfor()</a></code> loop at the present time.
<h4><a id="Advanced_Vectorization_181"></a>Advanced Vectorization</h4>
We have seen the different methods ArrayFire provides to vectorize our code. Tying them all together is a slightly more involved process that needs to consider data dimensionality and layout, memory usage, nesting order, etc. An excellent example and discussion of these factors can be found in another blog post:

<a href="http://sandbox.arrayfire.com/how-to-write-vectorized-code/">http://sandbox.arrayfire.com/how-to-write-vectorized-code/</a>

It’s worth noting that the whole implementation discussed in the blog has since been transformed into a convenient <strong>af::nearestNeighbour()</strong> function. Before writing something from scratch, check that ArrayFire doesn’t already have an implementation. The default vectorized nature of ArrayFire and an extensive collection of functions will speed things up in addition to replacing dozens of lines of code!