(8032,17728,'2015-03-10 17:38:04','2015-03-10 17:38:04','<!DOCTYPE html><html><head><meta charset=\"utf-8\"><title>Linear_Classifiers.md</title><style></style></head><body>\r\n<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire&#39;s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier</li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a linear number of operations represented below.</p>\r\n<pre><code>P(Ck | x) = (P(Ck) / P(x)) * P(Ck | x0) * P(Ck | x1) * P(Ck | x2) ... P(Ck | xn)\r\n</code></pre><p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled</li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>    // Iterate through each class\r\n    for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n        // train_feats is of size [num_feats, num_train]\r\n        // train_classes is of size [1, num_train]\r\n\r\n        // Get indices where train_classes matches current class\r\n        array idx = where(train_classes == ii);\r\n\r\n        // Get train_feats from the current class only\r\n        array train_feats_ii = train_feats(span, idx);\r\n\r\n        // Get the distribution parameters\r\n        mu(span, ii)  = mean(train_feats_ii, 1);\r\n        sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n        // Calculate priors\r\n        priors[ii] = (float)idx.elements() / (float)num_samples;\r\n    }\r\n</code></pre><p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>    for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n        // test_feats is of size [num_feats, num_test]\r\n        // mu, sig2 are of size  [num_feats, num_classes]\r\n        // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n        // Tile the mean and variance of current class to match test_feats\r\n        array Mu  = tile(mu (span, ii), 1, num_test);\r\n        array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n        array Df = test_feats - Mu;\r\n\r\n        // CDF of normal distribution is as follows\r\n        // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n        // Multiply the conditional probabilities and the prior of the class\r\n        // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n        // Calculate log_P instead for numerical stability\r\n        array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n        // Accumulate the log of conditional probabilities and the prior of the class\r\n        log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n    }\r\n\r\n    array val; // ignored\r\n    array idx; // Gets the class id\r\n\r\n    // Get the class ID with maximum log_probability\r\n    max(val, idx, log_probs, 1);\r\n</code></pre><h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>    // Activation function\r\n    array sigmoid(const array &amp;Xw)\r\n    {\r\n        // Transforms Xw to a value between [0, 1]\r\n        return 1 / (1 + exp(-Xw));\r\n    }\r\n\r\n    // Predict based on given parameters\r\n    array predict(const array &amp;X, const array &amp;Weights)\r\n    {\r\n        // Weighted inputs\r\n        array Xw = matmul(X, Weights);\r\n        // Transform Weighted inputs to output\r\n        return sigmoid(Xw);\r\n    }\r\n</code></pre><p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights</li>\r\n<li>Predict the output based on training data</li>\r\n<li>Update the weights based on the following formula\r\n//Enter formula here</li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre><h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr>\r\n<th>Algorithm</th>\r\n<th>Hardware</th>\r\n<th>Accuracy</th>\r\n<th>Train Time</th>\r\n<th>Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr>\r\n<td>Naive Bayes</td>\r\n<td>NVIDIA GTX 680</td>\r\n<td>80.69%</td>\r\n<td>0.003s</td>\r\n<td>0.011s</td>\r\n</tr>\r\n<tr>\r\n<td>Naive Bayes</td>\r\n<td>AMD Radeon 7970</td>\r\n<td>81.19%</td>\r\n<td>0.007s</td>\r\n<td>0.007s</td>\r\n</tr>\r\n<tr>\r\n<td>Perceptron (1000 iters)</td>\r\n<td>NVIDIA GTX 680</td>\r\n<td>83.64%</td>\r\n<td>2.005s</td>\r\n<td>0.0003s</td>\r\n</tr>\r\n<tr>\r\n<td>Perceptron (1000 iters)</td>\r\n<td>AMD Radeon 7970</td>\r\n<td>85.29%</td>\r\n<td>4.935s</td>\r\n<td>0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA</li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>\r\n\r\n</body></html>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 17:38:04','2015-03-10 17:38:04','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8033,17728,'2015-03-10 17:49:26','2015-03-10 17:49:26','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a linear number of operations represented below.</p>\r\n<pre><code>P(Ck | x) = (P(Ck) / P(x)) * P(Ck | x0) * P(Ck | x1) * P(Ck | x2) ... P(Ck | xn)</code></pre>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 17:49:26','2015-03-10 17:49:26','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8035,17728,'2015-03-10 18:00:13','2015-03-10 18:00:13','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a linear number of operations represented below.</p>\r\n<p>So the Baye\'s rule simplifies in the following manner:</p>\r\n<pre><code>\\[ P(C_k | x) = \\frac {P(C_k) * P(x | C_k)} {P(x)} \\]</code></pre>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:00:13','2015-03-10 18:00:13','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8036,17728,'2015-03-10 18:01:06','2015-03-10 18:01:06','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a linear number of operations represented below.</p>\r\n<p>So the Baye\'s rule simplifies in the following manner:</p>\r\n<p><br /><span class=\"math\">$$ P(C_k | x) = \\frac {P(C_k) * P(x | C_k)} {P(x)} $$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:01:06','2015-03-10 18:01:06','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8037,17728,'2015-03-10 18:20:49','2015-03-10 18:20:49','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>) = <em>f</em><em>r</em><em>a</em><em>c</em><em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em>(<em>x</em>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> is class k<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:20:49','2015-03-10 18:20:49','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8038,17728,'2015-03-10 18:21:52','2015-03-10 18:21:52','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> is class k<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:21:52','2015-03-10 18:21:52','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8039,17728,'2015-03-10 18:22:59','2015-03-10 18:22:59','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k $is class k$ $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:22:59','2015-03-10 18:22:59','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8040,17728,'2015-03-10 18:24:06','2015-03-10 18:24:06','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li>$<br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br />$ is class k<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:24:06','2015-03-10 18:24:06','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8041,17728,'2015-03-10 18:25:18','2015-03-10 18:25:18','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub>$<em>i</em><em>s</em><em>c</em><em>l</em><em>a</em><em>s</em><em>s</em><em>k</em>$</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:25:18','2015-03-10 18:25:18','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8042,17728,'2015-03-10 18:26:20','2015-03-10 18:26:20','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li>$C_k is from class k$<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:26:20','2015-03-10 18:26:20','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8043,17728,'2015-03-10 18:28:07','2015-03-10 18:28:07','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<!-- - \\[ C_k \\] is from class k -->\r\n\r\n<!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] -->\r\n\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:28:07','2015-03-10 18:28:07','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8044,17728,'2015-03-10 18:29:13','2015-03-10 18:29:13','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<!-- - \\[ C_k \\] is from class k -->\r\n\r\n<!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] -->\r\n\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:29:13','2015-03-10 18:29:13','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8045,17728,'2015-03-10 18:30:58','2015-03-10 18:30:58','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p><br /><span class=\"math\">$$ $C_k$ is class k $$</span><br /><br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:30:58','2015-03-10 18:30:58','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8046,17728,'2015-03-10 18:32:23','2015-03-10 18:32:23','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p><br /><span class=\"math\">$$ $$C_k$$ is class k $$</span><br /><br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:32:23','2015-03-10 18:32:23','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8047,17728,'2015-03-10 18:33:02','2015-03-10 18:33:02','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<!-- Where: -->\r\n\r\n\r\n<!-- \\[ $$C_k$$ is class k \\] -->\r\n\r\n<!-- - \\[ C_k \\] is from class k -->\r\n\r\n<!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] -->\r\n\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:33:02','2015-03-10 18:33:02','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8048,17728,'2015-03-10 18:34:00','2015-03-10 18:34:00','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub><em>i</em><em>s</em><em>c</em><em>l</em><em>a</em><em>s</em><em>s</em><em>k</em></span><br /><br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:34:00','2015-03-10 18:34:00','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8049,17728,'2015-03-10 18:38:38','2015-03-10 18:38:38','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\r\n\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}\r\ny = argmax_i(p(C_k) \\prod_i p(x_i | C_k)\r\n$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:38:38','2015-03-10 18:38:38','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8050,17728,'2015-03-10 18:39:22','2015-03-10 18:39:22','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod_i p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:39:22','2015-03-10 18:39:22','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8051,17728,'2015-03-10 18:41:20','2015-03-10 18:41:20','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_{i=1} p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod_{i=1} p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:41:20','2015-03-10 18:41:20','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8052,17728,'2015-03-10 18:42:26','2015-03-10 18:42:26','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:42:26','2015-03-10 18:42:26','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8053,17728,'2015-03-10 18:44:07','2015-03-10 18:44:07','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub>is class k</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em>is a feature vector containing the features{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /></li>\r\n</ul>\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:44:07','2015-03-10 18:44:07','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8054,17728,'2015-03-10 18:44:48','2015-03-10 18:44:48','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub> is class k</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em> is a feature vector containing the features{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /></li>\r\n</ul>\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:44:48','2015-03-10 18:44:48','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8055,17728,'2015-03-10 18:46:14','2015-03-10 18:46:14','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /></li>\r\n</ul>\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:46:14','2015-03-10 18:46:14','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8056,17728,'2015-03-10 18:48:46','2015-03-10 18:48:46','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:48:46','2015-03-10 18:48:46','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8057,17728,'2015-03-10 18:49:28','2015-03-10 18:49:28','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:49:28','2015-03-10 18:49:28','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8058,17728,'2015-03-10 18:53:30','2015-03-10 18:53:30','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p>[\\ y_k = f(W_k_i * x_i) \\]</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:53:30','2015-03-10 18:53:30','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8059,17728,'2015-03-10 18:54:40','2015-03-10 18:54:40','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>y</em><sub><em>k</em></sub> = <em>f</em>(<em>W</em><sub><em>k</em><sub><em>i</em></sub></sub> * <em>x</em><sub><em>i</em></sub>)</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:54:40','2015-03-10 18:54:40','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8060,17728,'2015-03-10 18:57:09','2015-03-10 18:57:09','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em><sub><em>i</em></sub></sub> * <em>x</em><sub><em>i</em></sub></span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:57:09','2015-03-10 18:57:09','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8061,17728,'2015-03-10 18:57:49','2015-03-10 18:57:49','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:57:49','2015-03-10 18:57:49','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8062,17728,'2015-03-10 18:58:27','2015-03-10 18:58:27','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\"><em>Y</em><sub><em>k</em></sub> = <em>f</em>(<em>Z</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:58:27','2015-03-10 18:58:27','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8063,17728,'2015-03-10 18:59:50','2015-03-10 18:59:50','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\">$$ Y_k = \\function f(Z_k) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:59:50','2015-03-10 18:59:50','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8064,17728,'2015-03-10 19:00:24','2015-03-10 19:00:24','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\">$$ Y_k = f\\l(Z_k\\l) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:00:24','2015-03-10 19:00:24','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0);
INSERT INTO `wp_posts` VALUES (8065,17728,'2015-03-10 19:00:48','2015-03-10 19:00:48','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\">$$ Y_k = f\\small(Z_k\\small) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:00:48','2015-03-10 19:00:48','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8066,17728,'2015-03-10 19:02:02','2015-03-10 19:02:02','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:02:02','2015-03-10 19:02:02','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8067,17728,'2015-03-10 19:05:36','2015-03-10 19:05:36','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 + \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:05:36','2015-03-10 19:05:36','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8068,17728,'2015-03-10 19:06:27','2015-03-10 19:06:27','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:06:27','2015-03-10 19:06:27','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8069,17728,'2015-03-10 19:12:26','2015-03-10 19:12:26','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space \\small(output_k - target_k\\small) * input_i $$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:12:26','2015-03-10 19:12:26','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8070,17728,'2015-03-10 19:13:36','2015-03-10 19:13:36','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space outer\\small(small(output_k - target_k\\small),  input_i\\small) $$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:13:36','2015-03-10 19:13:36','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8071,17728,'2015-03-10 19:14:41','2015-03-10 19:14:41','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space small(output_k - target_k\\small) \\outer  input_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:14:41','2015-03-10 19:14:41','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8072,17728,'2015-03-10 19:16:03','2015-03-10 19:16:03','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space small(output_k - target_k\\small)^T  input_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:16:03','2015-03-10 19:16:03','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8073,17728,'2015-03-10 19:16:34','2015-03-10 19:16:34','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space \\small(output_k - target_k\\small)^T * input_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:16:34','2015-03-10 19:16:34','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8074,17728,'2015-03-10 19:17:48','2015-03-10 19:17:48','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:17:48','2015-03-10 19:17:48','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8075,17728,'2015-03-10 19:20:20','2015-03-10 19:20:20','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:20:20','2015-03-10 19:20:20','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8076,17728,'2015-03-10 19:21:52','2015-03-10 19:21:52','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p><img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\"></p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:21:52','2015-03-10 19:21:52','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8077,17728,'2015-03-10 19:23:09','2015-03-10 19:23:09','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p><img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\"></p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:23:09','2015-03-10 19:23:09','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8078,17728,'2015-03-10 19:26:58','2015-03-10 19:26:58','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:26:58','2015-03-10 19:26:58','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8080,17728,'2015-03-10 19:34:30','2015-03-10 19:34:30','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:34:30','2015-03-10 19:34:30','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8081,17728,'2015-03-10 19:35:15','2015-03-10 19:35:15','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:35:15','2015-03-10 19:35:15','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8082,17728,'2015-03-10 19:38:30','2015-03-10 19:38:30','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n\r\n<ul>\r\n<li><a href=\"#naive-bayesian-classifier\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"#perceptron\">Perceptron</a></li>\r\n</ul>\r\n\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:38:30','2015-03-10 19:38:30','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8084,17728,'2015-03-10 19:52:37','2015-03-10 19:52:37','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayes Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n\r\n<ul>\r\n<li><a href=\"#naive-bayes-classifier\">Naive Bayes Classifier</a><br /></li>\r\n<li><a href=\"#perceptron\">Perceptron</a></li>\r\n</ul>\r\n\r\n<h3 id=\"naive-bayes-classifier\">Naive Bayes Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayes classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayes rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayes method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayes Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:52:37','2015-03-10 19:52:37','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),