This is the fifth in a series of posts looking at our current ArrayFire <a href="http://www.accelereyes.com/arrayfire/c/examples.htm" target="_blank">examples</a>. The code can be compiled and run from <code>arrayfire/examples/</code> when you <a href="http://www.accelereyes.com/download_arrayfire">download</a> and install the ArrayFire library. Today we will discuss the examples found in the <code>machine_learning/</code> directory.

In these examples, my machine has the following configuration:
<pre>ArrayFire v1.9 (build XXXXXXX) by AccelerEyes (64-bit Mac OSX)
License: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
CUDA toolkit 5.0, driver 304.54
GPU0 GeForce GT 560M, 1024 MB, Compute 3.0 (single,double)
Display Device: GPU0 GeForce GT 650M
Memory Usage: 245 MB free (1024 MB total)...</pre>
&nbsp;

<strong> 1. K-Means Clustering - <a title="kmeans.cpp" href="http://www.accelereyes.com/arrayfire/c/examples_2machine_learning_2kmeans_8cpp-example.htm">kmeans.cpp</a></strong>
<p style="text-align: center;"><a href="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-12.25.57-PM.png"><img class="aligncenter  wp-image-4251" alt="Screen Shot 2013-05-31 at 12.25.57 PM" src="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-12.25.57-PM-1024x438.png" width="717" height="307" /></a></p>
<p style="text-align: center;">Figure 1</p>
This is an example of K-Means Clustering Algorithm. K-Means Clustering Algorithm is a data mining technique that partitions the given data into groups by their similarities. All you need to know from this example is that this algorithm will repeatedly run through many loops to stabilize the clusters. Again, ArrayFire has this beautiful tool called <a href="http://www.accelereyes.com/arrayfire/c/page_gfor.htm#gfor_intro">GFOR</a> loop that handles any matrix array arithmetic and transformation in parallel. This increases the speed of both the cluster stabilization and k-means computation in this example.
<pre class="lang:default decode:true" title="gfor loops"> 35        gfor(array i, num_uniq) {    // current classification
 36             array diffs = abs(hist_idx(i) - means_vec);
 37             array val, idx;
 38             min(val, idx, diffs);
 39             hist_counts(hist_idx(i)) = idx;
 40         }  
 ..
 57     gfor(array i, k) { // pixel differences from computed means
 58         vol(span, span, i) = abs(data - means_vec(i));
 59     }</pre>
Just to verify the speed improvement in computation with ArrayFire again, I made a simple change in the code to find the elapsed times in FOR loop (left below) and GFOR loop (right below) as following.
<pre class="lang:default decode:true"> FOR Loop                                               GFOR Loop
 36  clock_t begin = clock();                           47  clock_t begin = clock();  
 37  for(int i=0; i&lt;num_uniq; i++){                     48  gfor(array i, num_uniq) {    // current classification
 38    array diffs = abs(hist_idx(i) - means_vec);      49    array diffs = abs(hist_idx(i) - means_vec);
 39    array val, idx;                                  50    array val, idx;
 40    min(val, idx, diffs);                            51    min(val, idx, diffs);
 41    hist_counts(hist_idx(i)) = idx;                  52    hist_counts(hist_idx(i)) = idx;
 42  }                                                  53  }   
 43                                                     54
 44  clock_t end = clock();                             55  clock_t end = clock();
 45  double t =  double(end - begin) / CLOCKS_PER_SEC;  56  double t =  double(end - begin) / CLOCKS_PER_SEC;
 46  printf("Elapsed Time: %f secn", t);               57  printf("Elapsed Time: %f secn", t);</pre>
In result, each GFOR loop was approximately 200x faster than each FOR loop as shown below. Since there were a total of twelve loops in this example, ArrayFire has performed almost 2000x faster in time. If there were more loops performed, you can imagine how much time we would save with ArrayFire.
<pre class="lang:r decode:true">FOR Loop                                    GFOR Loop
Elapsed Time: 0.016808 sec                  Elapsed Time: 0.002231 sec
Elapsed Time: 0.014868 sec                  Elapsed Time: 0.000676 sec
Elapsed Time: 0.053122 sec                  Elapsed Time: 0.000088 sec
Elapsed Time: 0.048089 sec                  Elapsed Time: 0.000074 sec
Elapsed Time: 0.048284 sec                  Elapsed Time: 0.000074 sec
Elapsed Time: 0.033541 sec                  Elapsed Time: 0.000075 sec
Elapsed Time: 0.013796 sec                  Elapsed Time: 0.000068 sec      
Elapsed Time: 0.029183 sec                  Elapsed Time: 0.000068 sec
Elapsed Time: 0.011643 sec                  Elapsed Time: 0.000068 sec
Elapsed Time: 0.011273 sec                  Elapsed Time: 0.000067 sec
Elapsed Time: 0.026709 sec                  Elapsed Time: 0.000068 sec
Elapsed Time: 0.010750 sec                  Elapsed Time: 0.000068 sec</pre>
The number of loops in this example was actually arbitrary because stabling the cluster could take longer than expected depending on the situation; thus it was a great example to evaluate the performance of ArrayFire. In the result of my experiment, ArrayFire was again proven to provide big performance benefits within loops.

&nbsp;

<strong>2. Neural Network - <a href="http://www.accelereyes.com/arrayfire/c/examples_2machine_learning_2neuralnetwork_8cpp-example.htm">neuralnetwork.cpp</a>
</strong>
<p style="text-align: center;"><img class="aligncenter  wp-image-4234" alt="Screen Shot 2013-05-31 at 10.58.20 AM" src="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-10.58.20-AM-1024x584.png" width="614" height="350" /></p>
<p style="text-align: center;">Figure 2</p>
This is an example of Neural Network algorithm. Neural Network algorithm (also known as Artificial Neural Network) is another data mining technique that is inspired by biological neural network system. I won't talk about the details of this algorithm, but let's first take a look at the three following images.
<p style="text-align: center;"><img class="aligncenter  wp-image-4230" alt="fireice" src="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-11.30.17-AM-1024x196.png" width="737" height="141" /></p>
These three layers are used as an input for our algorithm. The layers first get joined in a single matrix array by <a href="http://www.accelereyes.com/arrayfire/c/group__manip.htm#ga4971f0f3af2fd52c2db2967b721c5547">JOIN</a> function after flattened into a column vector by <a href="http://www.accelereyes.com/arrayfire/c/group__manip.htm#ga6667548d963698db28ff83cea4380154">FLAT</a> function. The algorithm contains a "train" procedure with a given number of iteration time (200 iterations in this example). During this iteration process, the algorithm determines errors for each layer and corrects them to train (or learn) the most optimized weight for each pixel using gradient. As shown Figure 2, the example loads another image to compare with the optimized pixel values after the train procedure has completed. It finally prints out the result of numbers that indicates the difference between the image (in Figure 2) and trained weights.
<pre class="lang:default decode:true"> 80     array samples = join(0,
 81                          flat(loadimage("images/fire-1.jpg")).T(),
 82                          flat(loadimage("images/ice-1.jpg")).T(),
 83                          flat(loadimage("images/fire-2.jpg")).T());
 84     samples = join(0,
 85                    samples,
 86                    flat(loadimage("images/ice-2.jpg")).T(),
 87                    flat(loadimage("images/fire-3.jpg")).T(),
 88                    flat(loadimage("images/ice-3.jpg")).T());
 89</pre>
Normally, this algorithm has to be done with the massive number of iteration for each pixel in all three layers. ArrayFire is, again, convenient in this algorithm; instead of having three layers independently, ArrayFire uses <a href="http://www.accelereyes.com/arrayfire/c/group__manip.htm#ga4971f0f3af2fd52c2db2967b721c5547">JOIN</a> function to combine all the layers into one single matrix array. This way, ArrayFire reduces all the matrix iterations and maximizes the efficiency of parallel matrix arithmetics (such as <a href="http://www.accelereyes.com/arrayfire/c/group__matmul__mat.htm#ga7491b0767aab0b0f7fe1a67274bdf586">MATMUL</a>, <a href="http://www.accelereyes.com/arrayfire/c/group__arith.htm#gacfc8b9d8b36c97330f23ef72e1bfaa31">POW</a>, and <a href="http://www.accelereyes.com/arrayfire/c/group__manip.htm#gaf7c3f5aad54e191bb3badd69e8ad4014">TILE</a>). Basically, this example ended up not requiring any matrix iteration by using ArrayFire (but still requires only the iterations for the number of train, which must be done without any trick).

&nbsp;

<strong>3. Principal Component Analysis - <a href="http://www.accelereyes.com/arrayfire/c/examples_2machine_learning_2pca_8cpp-example.htm">pca.cpp</a></strong>
<p style="text-align: center;"><a href="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-12.27.40-PM.png"><img class="aligncenter  wp-image-4252" alt="Screen Shot 2013-05-31 at 12.27.40 PM" src="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-12.27.40-PM-1024x556.png" width="717" height="389" /></a></p>
<p style="text-align: center;">Figure 3</p>
<p style="text-align: left;">This is an example of Principal Component Analysis. Principal Component Analysis is yet another data mining technique for finding unique patterns in high-dimensional data space. This technique is often found in applications for face recognition and image compression. The basic idea is to reduce the high-dimensional data space to a lower dimension in order to have a simpler representation of data set; so, it is easier to handle the data set as well as to plot them in a human readable dimensional space (three-dimension in our example).</p>

<pre class="lang:default decode:true"> 20     // data normalization
 21     array means = mean(data);
 22     array stdvs = stdev(data);
 23     array sdata = (data - tile(means, n, 1)) / tile(stdvs, n, 1);
 24 
 25     // decomposition
 26     array V, D, E;
 27     eigen(V, D, cov(sdata));
 28     E = diag(D);
 29     array val, idx;
 30     sort(val, idx, -1*E);
 31     array comps = matmul(sdata, V); // principal components
 ..
 68         fig("sub",2,1,1); fig("title","Input");
 69         plot3(data(span,1), data(span,2), data(span,3));
 70 
 71         fig("sub",2,1,2); fig("title","14D-&gt;3D PCA");
 72         plot3(proj(span,1), proj(span,2), proj(span,3));
 73 
 74         fig("draw");
 75     }</pre>
For this purpose, this algorithm consists of various matrix transformations such as Covariance (<a href="http://www.accelereyes.com/arrayfire/c/group__stats__mat.htm#gafd568a5dba252d6706e76a5bbd390af5">COV</a>), Standard Deviation (<a href="http://www.accelereyes.com/arrayfire/c/group__stats__mat.htm#ga4489167f96d0a302899341ca2af4fa91">STDEV</a>), Eigenvalues (<a href="http://www.accelereyes.com/arrayfire/c/group__factor__mat.htm#gaf97c09bd475b8e0b415de3e73e8a5589">EIGEN</a>), Means (<a href="http://www.accelereyes.com/arrayfire/c/group__stats__mat.htm#ga2173742174ff6ff7c336db645fd59828">MEAN</a>), Multiplication (<a href="http://www.accelereyes.com/arrayfire/c/group__matmul__mat.htm#ga7491b0767aab0b0f7fe1a67274bdf586">MATMUL</a>), and Sort (<a href="http://www.accelereyes.com/arrayfire/c/group__sort__mat.htm#gabafd9194924cfe3ebe4544fdf4f2863e">SORT</a>). The ArrayFire library already has these functions available for user's usage. ArrayFire certainly accelerates the speed of these matrix transformations as we have shown in previous examples. Finally, this example uses <a href="http://www.accelereyes.com/arrayfire/c/group__gfx__mat.htm#ga3ead87d1cd240d67389eb843ecda3784">PLOT3</a> function to plot the final data set in three-dimensional space for visualization as shown in Figure 3.

&nbsp;

<strong>4. Genetic Algorithm - <a href="http://www.accelereyes.com/arrayfire/c/examples_2machine_learning_2geneticalgorithm_8cpp-example.htm">geneticalgorithm.cpp</a></strong>
<p style="text-align: center;"><a href="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-12.28.59-PM.png"><img class="aligncenter  wp-image-4254" alt="Screen Shot 2013-05-31 at 12.28.59 PM" src="http://blog.accelereyes.com/blog/wp-content/uploads/2013/05/Screen-Shot-2013-05-31-at-12.28.59-PM-1024x574.png" width="717" height="402" /></a></p>
<p style="text-align: center;">Figure 4</p>
<p style="text-align: left;">This is an example of Genetic Algorithm. Genetic Algorithm is the last data mining technique in this post. This algorithm is inspired by the concept of neurological development as in learning through repetition over time. Therefore, a good model to describe this algorithm is the brain, which habitually tries and learns to achieve the optimal answer. The algorithm is a powerful tool for many applications, but the steps are simple enough.</p>

<pre class="lang:default decode:true"> 33 array selectFittest() {
 34     //pick top 50% fittest
 35     array sorted = sort(samplez);
 36     array indices = where(samplez &gt;= sorted.row(nsamples / 2));
 37 
 38     if (indices.elements() &gt; nsamples / 2) {
 39         indices = indices.rows(indices.elements() - nsamples / 2, indices.elements() - 1);
 40     }
 41 
 42     return indices;
 43 }</pre>
First, the algorithm selects only the top 50% fittest data elements (potential candidates close to the answer) from the initial data set. This may require to create a new data structure with unknown size and to check each element under the given condition in order to collect only the wanted data. Normally, the performance can be really slow, because it has to resize the data structure and check with the condition. To accomplish this procedure more efficient and faster, ArrayFire has <a href="http://www.accelereyes.com/arrayfire/c/group__sp__help.htm#ga767a83c029df2889ed4dc43afecf4f1e">WHERE</a> function that collects the elements that satisfy the specific requirements, of course, in a parallel manner, and forms a new matrix array with the collected elements.
<pre class="lang:default decode:true"> 52     //Divide selection in two
 53     array parentsx1 = parentsx.rows(0, parentsx.elements() / 2 - 1);
 54     array parentsx2 = parentsx.rows(parentsx.elements() / 2, parentsx.elements() - 1);
 55     array parentsy1 = parentsy.rows(0, parentsy.elements() / 2 - 1);
 56     array parentsy2 = parentsy.rows(parentsy.elements() / 2, parentsy.elements() - 1);    
 .. 
 63     //Create children as the cross between two parents
 64     array children_x1 = (parentsx1 &amp; uppermask) + (parentsx2 &amp; lowermask);
 65     array children_y1 = (parentsy1 &amp; uppermask) + (parentsy2 &amp; lowermask);
 66     
 67     array children_x2 = (parentsx2 &amp; uppermask) + (parentsx1 &amp; lowermask);
 68     array children_y2 = (parentsy2 &amp; uppermask) + (parentsy1 &amp; lowermask);</pre>
After the new data set has been created with the fittest elements, secondly, the algorithm divides the elements in two parent groups and uses the crossover points to create children (You don't need to know how to calculate the crossover points in this example; all you need to know is that the algorithm uses the crossover points to generate children for the next generation). For this procedure, the algorithm uses many element-wise arithmetic operations. Without ArrayFire library, this must take a long annoying waiting time to finish these same computations repeatedly. But, since the same equations will be applied on every element, we can simply use <a href="http://www.accelereyes.com/arrayfire/c/group__arith.htm">Element-wise arithmetics</a>, already built in ArrayFire library, to apply the maths on all the elements in a parallel manner.
<pre class="lang:default decode:true"> 70     //Join two new sets
 71     samplex = join(0, children_x1, children_x2);
 72     sampley = join(0, children_y1, children_y2);
 ..
 83     samplex = join(0, samplex, mutantx);
 84     sampley = join(0, sampley, mutanty);</pre>
Thirdly, the algorithm performs <a href="http://www.accelereyes.com/arrayfire/c/group__manip.htm#ga4971f0f3af2fd52c2db2967b721c5547">JOIN</a> to combine all the children in one single matrix array indicating itself as the current generation. In the meantime, it generates mutant elements by purposely masking random bits; mutants are formed to possibly arrive closer to the answer by random chance (Again, you don't need to know much detail about the Mutant either if you are only here for learning ArrayFire). Lastly, it updates current data set with the new generation data set and plot into three-dimensional space for display. With our ArrayFire library, the algorithm is now capable of executing these heavy procedures very quickly in real time.

&nbsp;

Machine Learning is really a technique that requires lots of digging, collecting, organizing, and optimizing the massive data sets. If you ever felt it slow, <a href="http://www.accelereyes.com/download_arrayfire">download</a> the ArrayFire library and accelerate your performance today!

&nbsp;

---

<strong>Posts in this series:</strong>
<ol>
	<li><a href="http://blog.accelereyes.com/blog/2013/03/04/arrayfire-examples-part-1-of-8-getting-started/">ArrayFire Examples (Part 1 of 8) – Getting Started</a></li>
	<li><a href="http://blog.accelereyes.com/blog/2013/03/06/arrayfire-examples-part-2-of-8-benchmarks/">ArrayFire Examples (Part 2 of 8) – Benchmarks</a></li>
	<li><a href="http://blog.accelereyes.com/blog/2013/03/25/arrayfire-examples-part-3-of-8-financial/">ArrayFire Examples (Part 3 of 8) – Financial</a></li>
	<li><a href="http://blog.accelereyes.com/blog/2013/04/16/arrayfire-examples-part-4-of-8-image-processing">ArrayFire Examples (Part 4 of 8) - Image Processing</a></li>
	<li><a href="http://blog.accelereyes.com/blog/2013/06/05/arrayfire-examples-part-5-of-8-machine-learning/">ArrayFire Examples (Part 5 of 8) - Machine Learning</a></li>
	<li><a href="http://blog.accelereyes.com/blog/2013/06/12/arrayfire-examples-part-6-of-8-multiple-gpus/">ArrayFire Examples (Part 6 of 8) - Multiple GPU</a></li>
	<li><a href="http://blog.accelereyes.com/blog/2013/07/03/arrayfire-examples-part-7-of-8-pde/">ArrayFire Examples (Part 7 of 8) - PDE</a></li>
</ol>
&nbsp;