(8037,17728,'2015-03-10 18:20:49','2015-03-10 18:20:49','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>) = <em>f</em><em>r</em><em>a</em><em>c</em><em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em>(<em>x</em>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> is class k<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:20:49','2015-03-10 18:20:49','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8038,17728,'2015-03-10 18:21:52','2015-03-10 18:21:52','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> is class k<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:21:52','2015-03-10 18:21:52','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8039,17728,'2015-03-10 18:22:59','2015-03-10 18:22:59','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k $is class k$ $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:22:59','2015-03-10 18:22:59','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8040,17728,'2015-03-10 18:24:06','2015-03-10 18:24:06','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li>$<br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br />$ is class k<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:24:06','2015-03-10 18:24:06','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8041,17728,'2015-03-10 18:25:18','2015-03-10 18:25:18','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub>$<em>i</em><em>s</em><em>c</em><em>l</em><em>a</em><em>s</em><em>s</em><em>k</em>$</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:25:18','2015-03-10 18:25:18','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8042,17728,'2015-03-10 18:26:20','2015-03-10 18:26:20','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li>$C_k is from class k$<br /></li>\r\n<li><br /><span class=\"math\"><em>x</em></span><br /> is a feature vector containing the features <br /><span class=\"math\">{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>∣<em>x</em>)</span><br /> is the posterior probability of class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /> given feature vector <br /><span class=\"math\"><em>x</em></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>C</em><sub><em>k</em></sub>)</span><br /> is the apriori probability distribution of the class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>p</em>(<em>x</em>∣<em>C</em><sub><em>k</em></sub>)</span><br /> is the probability of likelyhood <br /><span class=\"math\"><em>x</em></span><br /> belongs to class <br /><span class=\"math\"><em>C</em><sub><em>k</em></sub></span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:26:20','2015-03-10 18:26:20','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8043,17728,'2015-03-10 18:28:07','2015-03-10 18:28:07','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<!-- - \\[ C_k \\] is from class k -->\r\n\r\n<!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] -->\r\n\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em><sub><em>k</em></sub>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)<em>p</em><em>r</em><em>o</em><em>d</em><sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:28:07','2015-03-10 18:28:07','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8044,17728,'2015-03-10 18:29:13','2015-03-10 18:29:13','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<!-- - \\[ C_k \\] is from class k -->\r\n\r\n<!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] -->\r\n\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:29:13','2015-03-10 18:29:13','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8045,17728,'2015-03-10 18:30:58','2015-03-10 18:30:58','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p><br /><span class=\"math\">$$ $C_k$ is class k $$</span><br /><br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:30:58','2015-03-10 18:30:58','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8046,17728,'2015-03-10 18:32:23','2015-03-10 18:32:23','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p><br /><span class=\"math\">$$ $$C_k$$ is class k $$</span><br /><br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:32:23','2015-03-10 18:32:23','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8047,17728,'2015-03-10 18:33:02','2015-03-10 18:33:02','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<!-- Where: -->\r\n\r\n\r\n<!-- \\[ $$C_k$$ is class k \\] -->\r\n\r\n<!-- - \\[ C_k \\] is from class k -->\r\n\r\n<!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] -->\r\n\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:33:02','2015-03-10 18:33:02','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8048,17728,'2015-03-10 18:34:00','2015-03-10 18:34:00','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub><em>i</em><em>s</em><em>c</em><em>l</em><em>a</em><em>s</em><em>s</em><em>k</em></span><br /><br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\"><em>y</em> = <em>a</em><em>r</em><em>g</em><em>m</em><em>a</em><em>x</em>(<em>p</em>(<em>C</em><sub><em>k</em></sub>)∏<sub><em>i</em></sub><em>p</em>(<em>x</em><sub><em>i</em></sub>∣<em>C</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:34:00','2015-03-10 18:34:00','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8049,17728,'2015-03-10 18:38:38','2015-03-10 18:38:38','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\r\n\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}\r\ny = argmax_i(p(C_k) \\prod_i p(x_i | C_k)\r\n$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:38:38','2015-03-10 18:38:38','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8050,17728,'2015-03-10 18:39:22','2015-03-10 18:39:22','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_i p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod_i p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:39:22','2015-03-10 18:39:22','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8051,17728,'2015-03-10 18:41:20','2015-03-10 18:41:20','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod_{i=1} p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod_{i=1} p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:41:20','2015-03-10 18:41:20','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8052,17728,'2015-03-10 18:42:26','2015-03-10 18:42:26','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<p>$C_k$ is class k<br /><!-- - \\[ C_k \\] is from class k --><br /><!-- - \\[ x \\] is a feature vector containing the features \\[ \\{ x_0, x_1, x_2, ... x_n \\} \\] --><br /><!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] --><br /><!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] --><br /><!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] --></p>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:42:26','2015-03-10 18:42:26','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8053,17728,'2015-03-10 18:44:07','2015-03-10 18:44:07','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub>is class k</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em>is a feature vector containing the features{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /></li>\r\n</ul>\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:44:07','2015-03-10 18:44:07','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8054,17728,'2015-03-10 18:44:48','2015-03-10 18:44:48','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\"><em>C</em><sub><em>k</em></sub> is class k</span><br /><br /></li>\r\n<li><br /><span class=\"math\"><em>x</em> is a feature vector containing the features{<em>x</em><sub>0</sub>, <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, ...<em>x</em><sub><em>n</em></sub>}</span><br /></li>\r\n</ul>\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:44:48','2015-03-10 18:44:48','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8055,17728,'2015-03-10 18:46:14','2015-03-10 18:46:14','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /></li>\r\n</ul>\r\n<!-- - \\[ p(C_k | x) \\] is the posterior probability of class \\[ C_k \\] given feature vector \\[ x \\] -->\r\n\r\n<!-- - \\[ p(C_k) \\] is the apriori probability distribution of the class \\[ C_k \\] -->\r\n\r\n<!-- - \\[ p(x | C_k) \\] is the probability of likelyhood \\[ x \\] belongs to class \\[ C_k \\] -->\r\n\r\n\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k)$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:46:14','2015-03-10 18:46:14','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8056,17728,'2015-03-10 18:48:46','2015-03-10 18:48:46','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <br /><span class=\"math\"><em>p</em>(<em>x</em>)</span><br /> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:48:46','2015-03-10 18:48:46','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8057,17728,'2015-03-10 18:49:28','2015-03-10 18:49:28','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requres us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into an output in the range [0, 1].</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:49:28','2015-03-10 18:49:28','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8058,17728,'2015-03-10 18:53:30','2015-03-10 18:53:30','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p>[\\ y_k = f(W_k_i * x_i) \\]</p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:53:30','2015-03-10 18:53:30','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8059,17728,'2015-03-10 18:54:40','2015-03-10 18:54:40','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>y</em><sub><em>k</em></sub> = <em>f</em>(<em>W</em><sub><em>k</em><sub><em>i</em></sub></sub> * <em>x</em><sub><em>i</em></sub>)</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:54:40','2015-03-10 18:54:40','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8060,17728,'2015-03-10 18:57:09','2015-03-10 18:57:09','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em><sub><em>i</em></sub></sub> * <em>x</em><sub><em>i</em></sub></span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:57:09','2015-03-10 18:57:09','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8061,17728,'2015-03-10 18:57:49','2015-03-10 18:57:49','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:57:49','2015-03-10 18:57:49','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8062,17728,'2015-03-10 18:58:27','2015-03-10 18:58:27','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\"><em>Y</em><sub><em>k</em></sub> = <em>f</em>(<em>Z</em><sub><em>k</em></sub>)</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:58:27','2015-03-10 18:58:27','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8063,17728,'2015-03-10 18:59:50','2015-03-10 18:59:50','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\">$$ Y_k = \\function f(Z_k) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 18:59:50','2015-03-10 18:59:50','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8064,17728,'2015-03-10 19:00:24','2015-03-10 19:00:24','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\">$$ Y_k = f\\l(Z_k\\l) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:00:24','2015-03-10 19:00:24','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0);
INSERT INTO `wp_posts` VALUES (8065,17728,'2015-03-10 19:00:48','2015-03-10 19:00:48','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\"><em>Z</em><sub><em>k</em></sub> = <em>W</em><sub><em>k</em>, <em>i</em></sub> * <em>X</em><sub><em>i</em></sub></span><br /><br /><br /><span class=\"math\">$$ Y_k = f\\small(Z_k\\small) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:00:48','2015-03-10 19:00:48','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8066,17728,'2015-03-10 19:02:02','2015-03-10 19:02:02','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>// Enter the formulas for the transformation here</p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Xw)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Xw));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:02:02','2015-03-10 19:02:02','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8067,17728,'2015-03-10 19:05:36','2015-03-10 19:05:36','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 + \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:05:36','2015-03-10 19:05:36','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8068,17728,'2015-03-10 19:06:27','2015-03-10 19:06:27','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights based on the following formula<br />//Enter formula here<br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:06:27','2015-03-10 19:06:27','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8069,17728,'2015-03-10 19:12:26','2015-03-10 19:12:26','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space \\small(output_k - target_k\\small) * input_i $$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:12:26','2015-03-10 19:12:26','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8070,17728,'2015-03-10 19:13:36','2015-03-10 19:13:36','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space outer\\small(small(output_k - target_k\\small),  input_i\\small) $$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:13:36','2015-03-10 19:13:36','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8071,17728,'2015-03-10 19:14:41','2015-03-10 19:14:41','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space small(output_k - target_k\\small) \\outer  input_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:14:41','2015-03-10 19:14:41','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8072,17728,'2015-03-10 19:16:03','2015-03-10 19:16:03','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space small(output_k - target_k\\small)^T  input_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:16:03','2015-03-10 19:16:03','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8073,17728,'2015-03-10 19:16:34','2015-03-10 19:16:34','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ weights_{k,i} = weights_{k, i} \\space + \\space \\small(output_k - target_k\\small)^T * input_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:16:34','2015-03-10 19:16:34','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8074,17728,'2015-03-10 19:17:48','2015-03-10 19:17:48','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:17:48','2015-03-10 19:17:48','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8075,17728,'2015-03-10 19:20:20','2015-03-10 19:20:20','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:20:20','2015-03-10 19:20:20','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8076,17728,'2015-03-10 19:21:52','2015-03-10 19:21:52','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii &lt; num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p><img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\"></p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i &lt; maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean&lt;float&gt;(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err  &lt; maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:21:52','2015-03-10 19:21:52','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8077,17728,'2015-03-10 19:23:09','2015-03-10 19:23:09','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1]. To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation.</p>\r\n<p><img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\"></p>\r\n<p>For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:23:09','2015-03-10 19:23:09','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8078,17728,'2015-03-10 19:26:58','2015-03-10 19:26:58','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features to a normal distribution. This requires us to calculate the mean and the variance in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:26:58','2015-03-10 19:26:58','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8080,17728,'2015-03-10 19:34:30','2015-03-10 19:34:30','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iters)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:34:30','2015-03-10 19:34:30','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8081,17728,'2015-03-10 19:35:15','2015-03-10 19:35:15','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n<ul>\r\n<li>Naive Bayesian Classifier<br /></li>\r\n<li>Perceptron</li>\r\n</ul>\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:35:15','2015-03-10 19:35:15','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8082,17728,'2015-03-10 19:38:30','2015-03-10 19:38:30','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayesian Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n\r\n<ul>\r\n<li><a href=\"#naive-bayesian-classifier\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"#perceptron\">Perceptron</a></li>\r\n</ul>\r\n\r\n<h3 id=\"naive-bayesian-classifier\">Naive Bayesian Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayesian classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayesian rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayesian method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayesian Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:38:30','2015-03-10 19:38:30','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8084,17728,'2015-03-10 19:52:37','2015-03-10 19:52:37','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayes Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n\r\n<ul>\r\n<li><a href=\"#naive-bayes-classifier\">Naive Bayes Classifier</a><br /></li>\r\n<li><a href=\"#perceptron\">Perceptron</a></li>\r\n</ul>\r\n\r\n<h3 id=\"naive-bayes-classifier\">Naive Bayes Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayes classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayes rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayes method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &amp;Z)\r\n{\r\n    // Transforms Xw to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &amp;X, const array &amp;Weights)\r\n{\r\n    // Weighted inputs\r\n    array Xw = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Xw);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayes Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-10 19:52:37','2015-03-10 19:52:37','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8085,17728,'2015-03-11 01:43:33','2015-03-11 01:43:33','<p>Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayes Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.</p>\r\n<p>ArrayFire\'s easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.</p>\r\n\r\n<ul>\r\n<li><a href=\"#naive-bayes-classifier\">Naive Bayes Classifier</a><br /></li>\r\n<li><a href=\"#perceptron\">Perceptron</a></li>\r\n</ul>\r\n\r\n<h3 id=\"naive-bayes-classifier\">Naive Bayes Classifier</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Naive_Bayes_classifier\">Naive bayes classifier</a> is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayes rule to a simple multiplication of probabilities as show below.</p>\r\n<p>First we start with the simple Baye\'s rule.</p>\r\n<p><br /><span class=\"math\">$$ p(C_k | x) = \\frac{p(C_k)}{p(x)} \\prod\\limits_{i=1}^n p(x_i | C_k) $$</span><br /></p>\r\n<p>Where:</p>\r\n<ul>\r\n<li><br /><span class=\"math\">$$ C_k \\space \\text{is class k} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ x \\space \\text{is a feature vector containing the features} \\space \\{ x_0, x_1, x_2, ... x_n \\} $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k | x) \\space \\text {is the posterior probability of class} \\space C_k \\space \\text {given feature vector} \\space x $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(C_k) \\space \\text {is the apriori probability distribution of the class} \\space C_k $$</span><br /><br /></li>\r\n<li><br /><span class=\"math\">$$ p(x | C_k) \\space \\text {is the probability of likelyhood} \\space x \\space belongs to class \\space C_k $$</span><br /></li>\r\n</ul>\r\n<p>Since <code>p(x)</code> is the same for all classes, it can be safely ignored. The classification is then done using the following formula</p>\r\n<p><br /><span class=\"math\">$$\\newcommand{\\argmax}{\\operatornamewithlimits{argmax}}y = argmax_i(p(C_k) \\prod\\limits_{i=1}^n p(x_i | C_k))$$</span><br /></p>\r\n<p>Classification based on Naive Bayes method uses the following phases:</p>\r\n<ul>\r\n<li>Training Phase: The probability ditribution of each feature variable is modeled<br /></li>\r\n<li>Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.</li>\r\n</ul>\r\n<p>In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, <code>mu</code> and the variance <code>Sig2</code> in the training phase.</p>\r\n<p>The necessary ArrayFire code for the training phase can be seen below:</p>\r\n<pre><code>// Iterate through each class\r\nfor (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // train_feats is of size [num_feats, num_train]\r\n    // train_classes is of size [1, num_train]\r\n\r\n    // Get indices where train_classes matches current class\r\n    array idx = where(train_classes == ii);\r\n\r\n    // Get train_feats from the current class only\r\n    array train_feats_ii = train_feats(span, idx);\r\n\r\n    // Get the distribution parameters\r\n    mu(span, ii)  = mean(train_feats_ii, 1);\r\n    sig2(span,ii)  = var(train_feats_ii, 0, 1);\r\n\r\n    // Calculate priors\r\n    priors[ii] = (float)idx.elements() / (float)num_samples;\r\n}</code></pre>\r\n<p>With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.</p>\r\n<p>However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using <code>log</code> and <code>sum</code> instead of <code>exp</code> and <code>mul</code>.</p>\r\n<p>The prediciton phase can be seen below.</p>\r\n<pre><code>for (int ii = 0; ii < num_classes; ii++) {\r\n\r\n    // test_feats is of size [num_feats, num_test]\r\n    // mu, sig2 are of size  [num_feats, num_classes]\r\n    // mu(span, ii), sig2(span, ii) will get mean and variance of current class\r\n\r\n    // Tile the mean and variance of current class to match test_feats\r\n    array Mu  = tile(mu (span, ii), 1, num_test);\r\n    array Sig2 = tile(sig2(span, ii), 1, num_test) + 0.01; // to avoid dividing by zero\r\n\r\n    array Df = test_feats - Mu;\r\n\r\n    // CDF of normal distribution is as follows\r\n    // array P = exp(-(Df * Df) / 2 * Sig2) / sqrt(2 * af::Pi * Sig2);\r\n    // Multiply the conditional probabilities and the prior of the class\r\n    // probs(span, ii) = priors[ii] * product(P).T();\r\n\r\n    // Calculate log_P instead for numerical stability\r\n    array log_P =  (-(Df * Df) / (2 * Sig2))  - log(sqrt(2 * af::Pi * Sig2));\r\n    // Accumulate the log of conditional probabilities and the prior of the class\r\n    log_probs(span, ii) = log(priors[ii]) + sum(log_P).T();\r\n}\r\n\r\narray val; // ignored\r\narray idx; // Gets the class id\r\n\r\n// Get the class ID with maximum log_probability\r\nmax(val, idx, log_probs, 1);</code></pre>\r\n<h3 id=\"perceptron\">Perceptron</h3>\r\n<p><a href=\"http://en.wikipedia.org/wiki/Perceptron\">Perceptron</a> is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range <code>[0, 1]</code>.</p>\r\n<figure>\r\n<img src=\"https://docs.google.com/drawings/d/1sAZyAb3Q_YlGbI8ytwjmZg91NLV2_CCH7--yxt5Z7lA/pub?w=480&amp;h=360\">\r\n<figcaption> A visual representation of a perceptron </figcaption>\r\n</figure>\r\n<p>To perform classification on <code>K</code> classes, we use <code>K</code> perceptrons and choose the class with the maximum activation. For the perceptron <code>k</code>, the activation function can be seen below:</p>\r\n<p><br /><span class=\"math\">$$ Y_k = f\\small(W_{k,i} * X_i\\small) $$</span><br /></p>\r\n<p>To get an activation function that is continuous we use sigmoid function show below:</p>\r\n<p><br /><span class=\"math\">$$ sigmoid\\small(Z\\small) = \\frac{1}{1 \\space + \\space \\exp\\small(-Z\\small)} $$</span><br /></p>\r\n\r\n<p>The equivalent ArrayFire code can be seen here:</p>\r\n<pre><code>// Activation function\r\narray sigmoid(const array &Z)\r\n{\r\n    // Transforms Z to a value between [0, 1]\r\n    return 1 / (1 + exp(-Z));\r\n}\r\n\r\n// Predict based on given parameters\r\narray predict(const array &X, const array &Weights)\r\n{\r\n    // Weighted inputs\r\n    array Z = matmul(X, Weights);\r\n    // Transform Weighted inputs to output\r\n    return sigmoid(Z);\r\n}</code></pre>\r\n<p>The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:</p>\r\n<ol>\r\n<li>Initialize the weights<br /></li>\r\n<li>Predict the output based on training data<br /></li>\r\n<li>Update the weights in proportion to the folloing<br /><br /><span class=\"math\">$$ W_{k,i} = W_{k, i} \\space + \\space \\alpha * \\space \\small(Y_k - T_k\\small)^T \\space * \\space X_i$$</span><br /><br /></li>\r\n<li>if error greater than treshold, go to step 2.</li>\r\n</ol>\r\n<p>The training process of the perceptron can be implemented in ArrayFire using the following code.</p>\r\n<pre><code>    for (int i = 0; i < maxiter; i++) {\r\n\r\n        // Make a prediction based on current weights\r\n        array predicted = predict(input, Weights);\r\n\r\n        // G\r\n        array err = target - predicted;\r\n\r\n        // Get the average error\r\n        float mean_abs_err = mean<float>(abs(err));\r\n        // Stop training if the error is less than specified bounds\r\n        if (mean_abs_err < maxerr) break;\r\n\r\n        // Update the weights based on the input and error\r\n        Weights = Weights + alpha * matmul(transpose(input), err);\r\n    }\r\n</code></pre>\r\n<h3 id=\"results\">Results</h3>\r\n<p>The complete source code for testing and benchmarking these algorithms can be found here:</p>\r\n<ul>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/naive_bayes.cpp\">Naive Bayes Classifier</a><br /></li>\r\n<li><a href=\"https://github.com/arrayfire/arrayfire/blob/devel/examples/machine_learning/perceptron.cpp\">Peceptron</a></li>\r\n</ul>\r\n<p>We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.</p>\r\n<p>The performance and accuracy results can be seen below:</p>\r\n<table>\r\n<thead>\r\n<tr class=\"header\">\r\n<th align=\"left\">Algorithm</th>\r\n<th align=\"left\">Hardware</th>\r\n<th align=\"left\">Accuracy</th>\r\n<th align=\"left\">Train Time</th>\r\n<th align=\"left\">Predict Time</th>\r\n</tr>\r\n</thead>\r\n<tbody>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">80.69%</td>\r\n<td align=\"left\">0.003s</td>\r\n<td align=\"left\">0.011s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Naive Bayes</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">81.19%</td>\r\n<td align=\"left\">0.007s</td>\r\n<td align=\"left\">0.007s</td>\r\n</tr>\r\n<tr class=\"odd\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">NVIDIA GTX 680</td>\r\n<td align=\"left\">83.64%</td>\r\n<td align=\"left\">2.005s</td>\r\n<td align=\"left\">0.0003s</td>\r\n</tr>\r\n<tr class=\"even\">\r\n<td align=\"left\">Perceptron (1000 iter)</td>\r\n<td align=\"left\">AMD Radeon 7970</td>\r\n<td align=\"left\">85.29%</td>\r\n<td align=\"left\">4.935s</td>\r\n<td align=\"left\">0.0005s</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<h3 id=\"conclusion\">Conclusion</h3>\r\n<p>We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!</p>\r\n<p>If are going to GTC 2015, you may also be interested in the following tutorials.</p>\r\n<ul>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5803\">S5803</a>: Simplified Machine Learning for CUDA<br /></li>\r\n<li><a href=\"https://registration.gputechconf.com/form/session-listing&amp;doSearch=true&amp;queryInput=S5722\">S5722</a>: Deep Belief Networks using ArrayFire</li>\r\n</ul>','Machine Learning with ArrayFire: Linear Classifiers','','inherit','open','open','','8031-revision-v1','','','2015-03-11 01:43:33','2015-03-11 01:43:33','',8031,'http://sandbox.arrayfire.com/8031-revision-v1/',0,'revision','',0),(8097,17728,'2015-03-16 06:19:42','2015-03-16 06:19:42','[content_band inner_container=\"true\" no_margin=\"true\" border=\"none\" padding_top=\"10px\" padding_bottom=\"40px\"] [custom_headline type=\"center\" level=\"h2\"]This is the part where you download ArrayFire.[/custom_headline][gap size=\"3%\"]\r\n<p style=\"text-align: center;\">[button  type=\"flat\"  shape=\"rounded\" href=\"https://github.com/arrayfire/arrayfire\" size=\"x-large\" ]Get the source (GitHub repository)[/button]</p>\r\n<hr />\r\n<p style=\"text-align: center;\">[button  type=\"flat\"  shape=\"rounded\" href=\"https://sandbox.arrayfire.com/download-installers\" size=\"x-large\" ]Get the binary installers[/button]</p>\r\n[/content_band]','Download','','inherit','open','open','','5931-revision-v1','','','2015-03-16 06:19:42','2015-03-16 06:19:42','',5931,'http://sandbox.arrayfire.com/5931-revision-v1/',0,'revision','',0),(8098,17728,'2015-03-16 06:20:42','2015-03-16 06:20:42','[content_band inner_container=\"true\" no_margin=\"true\" border=\"none\" padding_top=\"10px\" padding_bottom=\"40px\"] [custom_headline type=\"center\" level=\"h2\"]This is the part where you download ArrayFire.[/custom_headline][gap size=\"3%\"]\r\n<p style=\"text-align: center;\">[button  type=\"flat\"  shape=\"rounded\" href=\"https://github.com/arrayfire/arrayfire\" size=\"x-large\" ]Get the source (GitHub repository)[/button]</p>\r\n<p style=\"text-align: center;\">[button  type=\"flat\"  shape=\"rounded\" href=\"https://sandbox.arrayfire.com/download-installers\" size=\"x-large\" ]Get the binary installers[/button]</p>\r\n[/content_band]','Download','','inherit','open','open','','5931-revision-v1','','','2015-03-16 06:20:42','2015-03-16 06:20:42','',5931,'http://sandbox.arrayfire.com/5931-revision-v1/',0,'revision','',0),(8101,7,'2015-03-17 19:07:51','2015-03-17 19:07:51','[content_band inner_container=\"true\" no_margin=\"true\" border=\"none\" padding_top=\"10px\" padding_bottom=\"40px\"] [custom_headline type=\"center\" level=\"h2\"]This is the part where you download ArrayFire.[/custom_headline][gap size=\"8%\"]\r\n<p style=\"text-align: center;\">[button  type=\"flat\"  shape=\"rounded\" href=\"https://github.com/arrayfire/arrayfire\" size=\"x-large\" ]Get the source (GitHub repository)[/button]</p>\r\n<p style=\"text-align: center;\">[button  type=\"flat\"  shape=\"rounded\" href=\"https://sandbox.arrayfire.com/download-installers\" size=\"x-large\" ]          Get the binary installers          [/button][gap size=\"8%\"]</p>\r\n[/content_band]','Download','','inherit','open','open','','5931-revision-v1','','','2015-03-17 19:07:51','2015-03-17 19:07:51','',5931,'http://sandbox.arrayfire.com/5931-revision-v1/',0,'revision','',0),(8124,18528,'2015-05-05 16:57:05','2015-05-05 16:57:05','<h3>Introduction</h3>\r\nWith Intel CPUs making up nearly <a href=\"https://www.cpubenchmark.net/market_share.html\">80% of the CPU market</a> and <a href=\"http://jonpeddie.com/press-releases/details/gpu-market-upintel-and-nvidia-graphics-winners-in-q4-amd-down/\">66% of computers using integrated graphics</a> one can easily argue that integrated graphics devices represent one of the greatest markets for GPU-accelerated computing. Here at ArrayFire, we have long recognized the potential of these devices and offer built-in support for Intel CPUs, GPUs, and AMD APUs in the OpenCL backend of our <a href=\"http://www.arrayfire.com\">ArrayFire</a> GPU computing library. Yet one common theme for debate in the office has been how the hardware performs on different operating systems with different drivers across hardware revisions.\r\n\r\nTo answer these questions (and, perhaps, to win some intra-office geek cred) I decided to write a series of blog posts about Intel\'s GPU OpenCL performance. In this first installment I will compare the performance of OpenCL on a 3rd generation Intel processor on Linux and Windows.\r\n<h3>Monte Carlo Pi estimation</h3>\r\nTo evaluate the performance of Intel\'s hardware I took a textbook problem of computing Pi using a Monte Carlo method. This is a classic example of a memory bound application where the time to read/write data from/to memory dominates most of the computation time. Thus, throwing N-more compute cores at the problem may not always achieve a linear acceleration unless careful attention to memory access patterns and cache usage is performed.\r\n\r\nThe Monte Carlo process for computing pi is fairly simple, consisting of three primary steps:\r\n<ol>\r\n	<li>Generate a series of random (x,y) pairs from within the <a href=\"http://en.wikipedia.org/wiki/Unit_square\">unit square</a></li>\r\n	<li>Determine which of these samples reside within the <a href=\"http://en.wikipedia.org/wiki/Unit_circle\">unit circle</a></li>\r\n	<li>Compute the ratio of the samples that reside within the square to those which reside within the circle and then multiply by four (to account for this being only one quadrant of a circle).</li>\r\n</ol>\r\nThere is a great visualization of this process and a demonstration of how the <a href=\"http://commons.wikimedia.org/wiki/File:Pi_30K.gif\">number of samples impacts the Pi estimate on wikipedia</a>.\r\n\r\nIf you were to implement this in C/C++, your code might look like this:\r\n<pre class=\"lang:default decode:true\" title=\"Sample CPU code for Monte Carlo Pi estimation\">...\r\n\r\nint count = 0;\r\nfloat x, y;\r\nvector&lt;float&gt; randomNums(2*samples);\r\n\r\n// init random numbers within the unit square.\r\n// Pack as (x,y) pairs\r\nfor(int i = 0; i &lt; 2*samples; i++)	{\r\n	randomNums[i] = float(rand()) / RAND_MAX;\r\n}\r\n\r\n// count how many samples reside in the circle\r\nfor(int i = 0; i &lt; samples; i++)\r\n{\r\n	x = randomNums[2*i];\r\n	y = randomNums[2*i + 1];\r\n\r\n	if(x*x + y*y &lt; 1.0)\r\n		count++;\r\n}\r\n\r\n// calculate the estimate and print the result\r\nfloat pi_estimate = 4 * float(count) / samples;\r\n\r\n...</pre>\r\n<h3>Benchmark methods</h3>\r\nWe implemented four different versions of the Pi problem with small changes in the implementation which test various aspects of the hardware and software. These five implementations are:\r\n<ol>\r\n	<li>Single core CPU - A direct implementation of the code listing above</li>\r\n	<li>A naive port of the CPU code to the GPU with no optimizations. Region testing is performed on the GPU and the result is summed on the CPU</li>\r\n	<li>An effort to reduce the workload of the CPU by partially summing the region testing results on the GPU</li>\r\n	<li>A classic GPU optimization using coalesced memory load/stores</li>\r\n	<li>And a final integrated-device specific optimization that uses <a href=\"http://sandbox.arrayfire.com/zero-copy-on-integrated-gpus/\">zero-copy memory buffers</a></li>\r\n</ol>\r\nFor these tests we used a few-year old ASUS N56V laptop we had in the office which features an Intel i7-3610QM CPU with a HD Graphics 4000 GT2 GPU. This particular GPU has 16 execution units which can host up to 128 concurrent threads. We installed Window 7 (Intel OpenCL GPU driver 10.18.10.395; Intel OpenCL CPU drivers 5.1.0.25) and Ubuntu 14.10 (GPU driver Beignet 1.1 from git acfc2a2 and CPU driver \"Build 92\"). In future blog posts we will cover 4th and 5th generation hardware (including Iris pro).\r\n<h3>Results and discussion</h3>\r\nOn Linux a single core CPU took <span data-sheets-value=\"[null,3,null,31583]\" data-sheets-userformat=\"[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]\">31.583 ms to execute on a sample size of 20 million sample points. Interestingly, the same code on Windows too over 66 ms to execute on the same sample size (in release mode, with /02 and high power settings selected). This curious result isn\'t particularly important as the OpenCL performance on the CPU does not appear to be degraded between operating systems.</span>\r\n\r\nBelow we plot the speed up of our OpenCL kernels relative to the performance of the Linux single core CPU result above for both the i7-3610QM CPU and HD Graphics 4000 GPU.\r\n\r\n<iframe width=\"694\" height=\"371\" seamless=\"\" frameborder=\"0\" scrolling=\"no\" src=\"https://docs.google.com/spreadsheets/d/1vMeMPs4d78wsbURINCKmAFDFBwIFzpAl438Q-geoeNs/pubchart?oid=2103791532&amp;format=interactive\"></iframe>\r\n\r\nWe see that the naive port of the CPU code directly to OpenCL is slower than the pure CPU implementation by a factor of four. This result is not terribly surprising once you consider the poorly conceived implementation encounters a full O(N) cost for data transfer and summing region test result for the 20 million sample points.\r\n\r\nThe OCL reduction kernel offloads some of the CPU\'s work by partially summing the region detection results on the GPU. This tweak increased the performance of the kernel between 4x (for the OpenCL on the GPU) to 10x (for OpenCL on the CPU). Curiously, the CPU kernels on Ubuntu and Windows perform very well, achieving speed-ups of about 3x the CPU. This is still below the theoretical 4x speed improvement one would expect on this quad-core CPU namely due to thread management and poor memory access patterns.\r\n\r\nLastly we will discuss the coalesced memory and zero copy kernels. The optimization here involves changing the memory access pattern from using strieded memory access:\r\n<pre class=\"lang:default decode:true\" title=\"Sample CPU code for Monte Carlo Pi estimation\">x = randomNums[2*i];\r\ny = randomNums[2*i + 1];</pre>\r\nto using OpenCL\'s float2 data type to access sequential elements, thus improving utilization of the memory bus. This technique improved throughput by 2.2x on Ubuntu and 3.25x on Windows. This makes the GPU about twice as fast as our initial CPU implementation. We see little to no acceleration in using zero copy buffers for this problem because the problem size for the final CPU sum is quite small (on the order of 1000 elements).\r\n<h3>Conclusion and a preview of the next post</h3>\r\nFor the Monte Carlo Pi problem, it appears Ubuntu and Beignet achieve slightly better performance than the current Windows Intel HD Graphics OpenCL drivers; however, given the limited number of trials performed (and lack of statistical error bars) we may conservatively state that the two OpenCL implementations perform similarly on the same hardware.\r\n\r\nWe found that using OpenCL on the GPU achieves better acceleration than OpenCL on the CPU in general. In one of our kernels, OpenCL on the i7-3610QM CPU was slightly faster than on the GPU, but this is probably due to automatic vectorization in the CPU OpenCL backend for this particular kernel.\r\n\r\nIn our next post we will compare the performance of OpenCL on 3rd, 4th, and 5th generation Intel processors using both memory and compute-bound kernels.','Intel OpenCL performance: 3rd generation hardware','','publish','open','open','','intel-opencl-performance-3rd-generation-hardware','','','2015-05-05 16:57:05','2015-05-05 16:57:05','',0,'http://sandbox.arrayfire.com/?p=8124',0,'post','',1),(8125,18528,'2015-04-09 16:02:35','2015-04-09 16:02:35','One of the most powerful aspects of parallel program on integrated GPUs is taking advantage of shared memory and caches. The best example of this is sharing common data between the CPU and GPU via. zero-copy buffers. This technique permits your program to avoid the O(N) cost of copying data to/from the GPU. This feature is particularly useful for applications that deal with real-time data streams, like video processing.\r\n\r\n<!--more-->\r\n\r\nCreating zero copy buffers is not that difficult, but there is one caveat when it comes to using the buffers. First, lets consider how to make zero-copy buffers:\r\n<h4 id=\"method-1-let-opencl-allocate-memory-and-use-map-unmap-buffers\">Method 1: OpenCL allocation of zero-copy buffers</h4>\r\nThe first method of allocating zero-copy buffers is quite simple: let OpenCL do it for you. To create a buffer in this fashion simply specify the <code>CL_MEM_ALLOC_HOST_PTR</code> argument to the <code>cl::Buffer</code> constructor (or<code>clCreateBuffer</code> if you are using the C API):\r\n<pre><code class=\"lang-c++\">// Instruct OpenCL to allocate host memory\r\n::size_t size = N * sizeof(float)\r\nBuffer d_results(context, CL_MEM_ALLOC_HOST_PTR, size);\r\n\r\n... launch kernels, etc\r\n</code></pre>\r\n<h4 id=\"method-2-allocate-aligned-memory-and-use-map-unmap-buffers\">Method 2: Programmer-allocated aligned memory</h4>\r\nIn situations where you are streaming data or doing DMA between devices, it may be necessary to use aligned memory. In this situation, Intel OpenCL devices require that data be allocated with 4k alignment and be a multiple of 64 bytes (a cache line). At present AMD APUs follow this same alignment convention. Aligned allocation requires calls to OS-specific functions, after the memory is allocated, you can use create a buffer using CL_MEM_USE_HOST_PTR function:\r\n<pre><code class=\"lang-c++\">::size_t size = N * sizeof(float); // size must be a multiple of 64\r\n::size_t alignment = 4096;\r\n// Allocate aligned memory on the host. The particular function is OS-dependent.\r\n//  memalign(size_t alignment, size_t size) on Linux, release memory with free(...)\r\n//  _aligned_malloc(size_t size, size_t alignment) on Windows, release with _aligned_free(...)\r\n//  posix_memalign(void ** ptr, size_t alignment, size_t size) on OSX, release with free(...)\r\nfloat * h_results = (float*) memalign(alignment, size);\r\n\r\n// Create an OpenCL buffer using the host pointer\r\nBuffer d_results = cl::Buffer(context, CL_MEM_USE_HOST_PTR, size, t_results);\r\n</code></pre>\r\n<h4 id=\"accessing-memory\">Accessing zero-copy memory buffers</h4>\r\nAfter your buffers are created, how do we access the data contained in them? Given method 2, one might think your code could directly access the elements of \"h_results\", but this probably won\'t work. Perhaps the most misunderstood aspect of zero-copy buffers in OpenCL relates to the state of the buffer after being used in a kernel invocation. It is often thought that the memory in the buffers will be updated immediately after the kernel completes; however, the OpenCL specification <strong>does not guarantee this behavior</strong>. Instead, it is up to the programmer to ensure that the buffer is correctly updated prior to accessing the data. To do so, simply use the OpenCL map/unmap buffer commands:\r\n<pre><code class=\"lang-c++\">// Get exclusive access to the buffer\r\nfloat * temp = (float*) queue.enqueueMapBuffer(d_results, CL_TRUE, CL_MAP_READ, 0, size);\r\n\r\n... use the mapped buffer\r\n\r\n// release the buffer back to OpenCL’s control\r\nqueue.enqueueUnmapMemObject(d_results, temp);</code></pre>\r\nNote that you must unmap the buffer after use to inform OpenCL that it has been given control over the memory location.\r\n<h4 id=\"further-reading\">Further reading</h4>\r\n<a href=\"https://software.intel.com/en-us/articles/getting-the-most-from-opencl-12-how-to-increase-performance-by-minimizing-buffer-copies-on-intel-processor-graphics\">Getting the Most from OpenCL™ 1.2: How to Increase Performance by Minimizing Buffer Copies on Intel® Processor Graphics</a>\r\n\r\n<a href=\"http://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/opencl-optimization-guide/#50401315_pgfId-503784\">AMD OpenCL Optimization Guide: Mapping and zero-copy buffers</a>\r\n\r\n&nbsp;','Using zero-copy buffers on integrated GPUs','','publish','open','open','','zero-copy-on-integrated-gpus','','\nhttp://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/opencl-optimization-guide/#50401315_pgfId-503784','2015-04-09 16:06:03','2015-04-09 16:06:03','',0,'http://sandbox.arrayfire.com/?p=8125',0,'post','',1),(8126,18528,'2015-04-07 19:42:45','2015-04-07 19:42:45','One of the most powerful aspects of parallel programming on integrated GPUs is that you may avoid the O(N) copy operation associated with moving data from host to device memory. This feature can be particularly useful for applications that deal with real-time data streams, like video processing.\r\n<h2 id=\"method-1-let-opencl-allocate-memory-and-use-map-unmap-buffers\">Method 1: Let OpenCL allocate memory and use map/unmap buffers</h2>\r\nThe first method of allocating zero-copy buffers is quite simple: let OpenCL do it for you. To create a buffer in this fashion simply specify the<code>CL_MEM_ALLOC_HOST_PTR</code> argument to the <code>cl::Buffer</code> constructor (or<code>clCreateBuffer</code> if you are using the C API):\r\n<pre><code class=\"lang-c++\">// Instruct OpenCL to allocate host memory\r\n::size_t size = N * sizeof(float)\r\nBuffer d_results(context, CL_MEM_ALLOC_HOST_PTR, size);\r\n\r\n<span class=\"hljs-keyword\">...</span> launch kernels, etc\r\n</code></pre>\r\n<h2 id=\"method-2-allocate-aligned-memory-and-use-map-unmap-buffers\">Method 2: Allocate aligned memory and use map/unmap buffers</h2>\r\nIn situations where you are streaming data or doing DMA between devices, it may be necessary to use aligned memory. In this situation, Intel OpenCL devices require that data be allocated with 4k alignment and be a multiple of 64 bytes (a cache line) as follows:\r\n<pre><code class=\"lang-c++\">::size_t size = N * sizeof(float); // size must be a multiple of <span class=\"hljs-number\">64</span>\r\n::size_t alignment = <span class=\"hljs-number\">4096</span>;\r\n// Allocate aligned memory on the host. The particular <span class=\"hljs-keyword\">function</span> is OS-dependent.\r\n//  memalign(size_t alignment, size_t size) on Linux, release memory with free(<span class=\"hljs-keyword\">...</span>)\r\n//  _aligned_malloc(size_t size, size_t alignment) on Windows, release with _aligned_free(<span class=\"hljs-keyword\">...</span>)\r\n//  posix_memalign(void ** ptr, size_t alignment, size_t size) on OSX, release with free(<span class=\"hljs-keyword\">...</span>)\r\nfloat * h_results = (float*) memalign(alignment, size);\r\n\r\n// Create an OpenCL buffer using the host pointer\r\nBuffer d_results = cl::Buffer(context, CL_MEM_USE_HOST_PTR, size, t_results);\r\n</code></pre>\r\n<h2 id=\"accessing-memory\">Accessing memory</h2>\r\nPerhaps the most misunderstood aspect of zero-copy buffers in OpenCL relates to the state of the buffer after being used in a kernel invocation. It is often thought that the memory in the buffers will be updated immediately after the kernel completes; however, the OpenCL specification does not guarantee this behavior. To ensure that a zero-copy buffer is properly updated prior to use, a developer must map (and unmap!) a buffer prior to use in host code. This is actually a quite simple process:\r\n<pre><code class=\"lang-c++\">// Get exclusive access to the buffer\r\nfloat * temp = (float*) queue.enqueueMapBuffer(d_results, CL_TRUE, CL_MAP_READ, <span class=\"hljs-number\">0</span>, size);\r\n\r\n<span class=\"hljs-keyword\">...</span> use the mapped buffer\r\n\r\n// release the buffer back to OpenCL’s control\r\nqueue.enqueueUnmapMemObject(d_results, temp);\r\n</code></pre>\r\n<h2 id=\"further-reading\">Further reading</h2>\r\n<a href=\"https://software.intel.com/en-us/articles/getting-the-most-from-opencl-12-how-to-increase-performance-by-minimizing-buffer-copies-on-intel-processor-graphics\">Getting the Most from OpenCL™ 1.2: How to Increase Performance by Minimizing Buffer Copies on Intel® Processor Graphics</a>','Improving OpenCL performance using zero-copy buffers on Intel GPUs','','inherit','open','open','','8125-revision-v1','','','2015-04-07 19:42:45','2015-04-07 19:42:45','',8125,'http://sandbox.arrayfire.com/8125-revision-v1/',0,'revision','',0),(8128,18528,'2015-04-07 19:51:25','2015-04-07 19:51:25','One of the most powerful aspects of parallel programming on integrated GPUs is that you may avoid the O(N) copy operation associated with moving data from host to device memory. This feature can be particularly useful for applications that deal with real-time data streams, like video processing.\r\n<h2 id=\"method-1-let-opencl-allocate-memory-and-use-map-unmap-buffers\">Method 1: Let OpenCL allocate memory and use map/unmap buffers</h2>\r\nThe first method of allocating zero-copy buffers is quite simple: let OpenCL do it for you. To create a buffer in this fashion simply specify the<code>CL_MEM_ALLOC_HOST_PTR</code> argument to the <code>cl::Buffer</code> constructor (or<code>clCreateBuffer</code> if you are using the C API):\r\n<pre><code class=\"lang-c++\">// Instruct OpenCL to allocate host memory\r\n::size_t size = N * sizeof(float)\r\nBuffer d_results(context, CL_MEM_ALLOC_HOST_PTR, size);\r\n\r\n... launch kernels, etc\r\n</code></pre>\r\n<h2 id=\"method-2-allocate-aligned-memory-and-use-map-unmap-buffers\">Method 2: Allocate aligned memory and use map/unmap buffers</h2>\r\nIn situations where you are streaming data or doing DMA between devices, it may be necessary to use aligned memory. In this situation, Intel OpenCL devices require that data be allocated with 4k alignment and be a multiple of 64 bytes (a cache line) as follows:\r\n<pre><code class=\"lang-c++\">::size_t size = N * sizeof(float); // size must be a multiple of <span class=\"hljs-number\">64</span>\r\n::size_t alignment = <span class=\"hljs-number\">4096</span>;\r\n// Allocate aligned memory on the host. The particular <span class=\"hljs-keyword\">function</span> is OS-dependent.\r\n//  memalign(size_t alignment, size_t size) on Linux, release memory with free(...)\r\n//  _aligned_malloc(size_t size, size_t alignment) on Windows, release with _aligned_free(...)\r\n//  posix_memalign(void ** ptr, size_t alignment, size_t size) on OSX, release with free(...)\r\nfloat * h_results = (float*) memalign(alignment, size);\r\n\r\n// Create an OpenCL buffer using the host pointer\r\nBuffer d_results = cl::Buffer(context, CL_MEM_USE_HOST_PTR, size, t_results);\r\n</code></pre>\r\n<h2 id=\"accessing-memory\">Accessing memory</h2>\r\nPerhaps the most misunderstood aspect of zero-copy buffers in OpenCL relates to the state of the buffer after being used in a kernel invocation. It is often thought that the memory in the buffers will be updated immediately after the kernel completes; however, the OpenCL specification does not guarantee this behavior. To ensure that a zero-copy buffer is properly updated prior to use, a developer must map (and unmap!) a buffer prior to use in host code. This is actually a quite simple process:\r\n<pre><code class=\"lang-c++\">// Get exclusive access to the buffer\r\nfloat * temp = (float*) queue.enqueueMapBuffer(d_results, CL_TRUE, CL_MAP_READ, <span class=\"hljs-number\">0</span>, size);\r\n\r\n... use the mapped buffer\r\n\r\n// release the buffer back to OpenCL’s control\r\nqueue.enqueueUnmapMemObject(d_results, temp);\r\n</code></pre>\r\n<h2 id=\"further-reading\">Further reading</h2>\r\n<a href=\"https://software.intel.com/en-us/articles/getting-the-most-from-opencl-12-how-to-increase-performance-by-minimizing-buffer-copies-on-intel-processor-graphics\">Getting the Most from OpenCL™ 1.2: How to Increase Performance by Minimizing Buffer Copies on Intel® Processor Graphics</a>','Improving OpenCL performance using zero-copy buffers on Intel GPUs','','inherit','open','open','','8125-revision-v1','','','2015-04-07 19:51:25','2015-04-07 19:51:25','',8125,'http://sandbox.arrayfire.com/8125-revision-v1/',0,'revision','',0),(8129,18528,'2015-04-07 19:52:23','2015-04-07 19:52:23','One of the most powerful aspects of parallel programming on integrated GPUs is that you may avoid the O(N) copy operation associated with moving data from host to device memory. This feature can be particularly useful for applications that deal with real-time data streams, like video processing.\r\n<h2 id=\"method-1-let-opencl-allocate-memory-and-use-map-unmap-buffers\">Method 1: Let OpenCL allocate memory and use map/unmap buffers</h2>\r\nThe first method of allocating zero-copy buffers is quite simple: let OpenCL do it for you. To create a buffer in this fashion simply specify the<code>CL_MEM_ALLOC_HOST_PTR</code> argument to the <code>cl::Buffer</code> constructor (or<code>clCreateBuffer</code> if you are using the C API):\r\n<pre><code class=\"lang-c++\">// Instruct OpenCL to allocate host memory\r\n::size_t size = N * sizeof(float)\r\nBuffer d_results(context, CL_MEM_ALLOC_HOST_PTR, size);\r\n\r\n... launch kernels, etc\r\n</code></pre>\r\n<h2 id=\"method-2-allocate-aligned-memory-and-use-map-unmap-buffers\">Method 2: Allocate aligned memory and use map/unmap buffers</h2>\r\nIn situations where you are streaming data or doing DMA between devices, it may be necessary to use aligned memory. In this situation, Intel OpenCL devices require that data be allocated with 4k alignment and be a multiple of 64 bytes (a cache line) as follows:\r\n<pre><code class=\"lang-c++\">::size_t size = N * sizeof(float); // size must be a multiple of 64\r\n::size_t alignment = 4096;\r\n// Allocate aligned memory on the host. The particular <span class=\"hljs-keyword\">function is OS-dependent.\r\n//  memalign(size_t alignment, size_t size) on Linux, release memory with free(...)\r\n//  _aligned_malloc(size_t size, size_t alignment) on Windows, release with _aligned_free(...)\r\n//  posix_memalign(void ** ptr, size_t alignment, size_t size) on OSX, release with free(...)\r\nfloat * h_results = (float*) memalign(alignment, size);\r\n\r\n// Create an OpenCL buffer using the host pointer\r\nBuffer d_results = cl::Buffer(context, CL_MEM_USE_HOST_PTR, size, t_results);\r\n</code></pre>\r\n<h2 id=\"accessing-memory\">Accessing memory</h2>\r\nPerhaps the most misunderstood aspect of zero-copy buffers in OpenCL relates to the state of the buffer after being used in a kernel invocation. It is often thought that the memory in the buffers will be updated immediately after the kernel completes; however, the OpenCL specification does not guarantee this behavior. To ensure that a zero-copy buffer is properly updated prior to use, a developer must map (and unmap!) a buffer prior to use in host code. This is actually a quite simple process:\r\n<pre><code class=\"lang-c++\">// Get exclusive access to the buffer\r\nfloat * temp = (float*) queue.enqueueMapBuffer(d_results, CL_TRUE, CL_MAP_READ, 0, size);\r\n\r\n... use the mapped buffer\r\n\r\n// release the buffer back to OpenCL’s control\r\nqueue.enqueueUnmapMemObject(d_results, temp);\r\n</code></pre>\r\n<h2 id=\"further-reading\">Further reading</h2>\r\n<a href=\"https://software.intel.com/en-us/articles/getting-the-most-from-opencl-12-how-to-increase-performance-by-minimizing-buffer-copies-on-intel-processor-graphics\">Getting the Most from OpenCL™ 1.2: How to Increase Performance by Minimizing Buffer Copies on Intel® Processor Graphics</a>','Improving OpenCL performance using zero-copy buffers on Intel GPUs','','inherit','open','open','','8125-revision-v1','','','2015-04-07 19:52:23','2015-04-07 19:52:23','',8125,'http://sandbox.arrayfire.com/8125-revision-v1/',0,'revision','',0),(8131,7,'2015-04-08 19:02:52','2015-04-08 19:02:52','[content_band inner_container=\"true\" no_margin=\"true\" border=\"top\" padding_top=\"40px\" padding_bottom=\"40px\"][column type=\"one-fifth\"].[/column][column type=\"three-fifths\"][custom_headline type=\"left\" level=\"h3\"]Reveal Imaging, an SAIC Company[/custom_headline][/column][column type=\"one-fifth\" last=\"true\"].[/column][gap size=\"3%\"]\r\n\r\n[column type=\"one-fifth\"].[/column][column type=\"three-fifths\"]<img class=\"alignright size-full wp-image-6896\" src=\"http://sandbox.arrayfire.com/wp-content/uploads/2014/10/reveal-logo-promo-01.png\" alt=\"reveal logo, promo-01\" width=\"360\" height=\"120\" />Iterative reconstruction from projections is typically a function reserved for the medical industry. This approach allows medical professionals to reconstruct computed tomography (CT) images from acquired data; the reconstructed images display virtual “slices” of a scanned object, such as a heart or brain.\r\n\r\nIn reconstructing an image from the data, algorithms are used to approximate the solution. The closer the solution, the better the reconstruction. Multiple iterations produce more accurate solutions, and, thus, higher quality images. This technique, however, consumes computation time; speeding the process through the GPU is one solution to this problem.\r\n<h4>Mission</h4>\r\nReveal Imaging, an SAIC company, uses this CT technology in developing threat-detection products for transportation systems—such as cargo and baggage screening in airports—and in securing facilities and public events. Speed is critical in their technology, and they approached us seeking acceleration.\r\n\r\nReveal Imaging was running its cone-beam reconstructor via CUDA code previously implemented by a different contractor, but the performance was sub-par. They tasked ArrayFire with improving that performance on their GTX690s.\r\n<h4>Action</h4>\r\nOur engineers approached this by following standard CUDA practices. The project consisted mostly of cleaning up the existing code to allow it to run faster.\r\n\r\nFirst we removed the dependence of the permitted slice thickness on the number of GPU cards in the system. Then we allowed for one of the GPUs in the system to be excluded from those used by the reconstruction code and dedicated to driving a display. Next, we assured that the code met specifications with the latest CUDA compilers through version 5.0. Finally, we implemented performance enhancements, bottleneck reductions, and bug fixes previously identified.\r\n<h4>Results</h4>\r\nThe conclusion of the project was followed by a three week testing period, during which time Reveal Imaging determined reliability and overall satisfaction of the code’s performance. The code’s original speed was measured at around 200 slices per second—through our optimization efforts we were able to increase that number to 320 slices per second. Upon completion of the testing period Reveal Imaging was very pleased with the results and end performance of their code.\r\n<blockquote>The changes that you made sped up the system more than expected! How often does anybody get to say that things work better than expected?  —Reveal Imaging</blockquote>\r\nIn the end, Reveal Imaging was able to focus on other development aspects of their project while we focused on what we do best—working relentlessly to achieve the best possible results in parallel computing.\r\n<h6>About SAIC</h6>\r\nScience Applications International Corporation (SAIC) is a leading technology integrator that provides full lifecycle services and solutions in the technical, engineering, and enterprise IT markets. SAIC has approximately 13,000 employees worldwide.[/column][column type=\"one-fifth\" last=\"true\"].[/column][/content_band]','Case Study: Reveal Imaging','','inherit','open','open','','6923-revision-v1','','','2015-04-08 19:02:52','2015-04-08 19:02:52','',6923,'http://sandbox.arrayfire.com/6923-revision-v1/',0,'revision','',0),