In the upcoming months we will be adding a lot of new Computer Vision functionality to ArrayFire. This means dozens of new functions specifically targeted for the most commonly used applications in this field, such as feature tracking, object classification, scene segmentation, optical flow, stereo-vision, among others, will be added to take full advantage of <abbr title="Graphics Processing Unit">GPU</abbr> devices.
<div>[In the upcoming months we'll be adding a lot of new Computer Vision functionality to ArrayFire, <span style="color: #333333;">specifically targeting the most commonly used applications in this field. New functions include</span> feature tracking, object classification, scene segmentation, optical flow, and stereo-vision.]</div>
Feature tracking consists of three basic steps:
<ol>
	<li>Detecting good or unique features<del>,</del>; normally they are corners or blobs of an object.</li>
	<li>Extracting a descriptor for each feature<del>,</del>--understanding the texture of a small patch around each feature.</li>
	<li>Descriptor matching<del>,</del>--finding out the best match for each pair of descriptors (one from the object being tracked, another from a scene that potentially contains that object), if any.</li>
</ol>
<h3>Harris corner detector</h3>
In this article we will be using ArrayFire to dive deeper into the first step of feature tracking: detecting good features. Through a step-by-step guide<del>,</del> we will develop a simple corner detector, known as the Harris corner detector [1].

The Harris corner detector is very simple to implement, especially when you have so many good tools and libraries out there to do the core matrix manipulation. What we specifically need for Harris are some convolutions, together with a few element-wise matrix addition and multiplication operations<del>,</del>; all of these operations are present in most scientific tools and libraries that work with matrix manipulation. I will try to avoid going into the mathematical details in this post and go directly to the implementation.
<h5>Loading an image</h5>
Before <del>starting</del> writing the code for the corner detector, we need to load an image. The image can be either color or <del>gray-scale</del> grayscale, but it [what is this pronoun referring to--the Harris corner detector?] will only work on the gray-scale version<del>, so</del>; if <del>it is</del> we upload a color image, we have to convert it before proceeding.
<pre>// Load a color image
array img_color = loadimage("lena.jpg", true);
// Convert the image from RGB to gray-scale
array img_gray = colorspace(img_color, af_gray, af_rgb);
// For visualization in ArrayFire, color images must be in the [0.0f-1.0f] interval
img_color /= 255.f;
</pre>
<h5>Compute first-order derivatives</h5>
The first step in the Harris corner detector is to determine first order derivatives of an image, in both horizontal and vertical directions. This can be achieved by one convolution for each direction, with the kernel (-1, 0, 1)  for horizontal derivative and its transpose for vertical derivative. ArrayFire contains a function called <a href="http://www.accelereyes.com/arrayfire/c/group__math__func__cpp__grad.htm">grad</a>, <del>it</del> which calculates gradients in both directions and will perform faster than the two convolutions. <span style="color: rgb(0, 128, 0);">The only difference [between what?] is [that] <a href="http://www.accelereyes.com/arrayfire/c/group__math__func__cpp__grad.htm"><span style="color: rgb(0, 128, 0);">grad</span></a> divides the result by 2, thus, the response factor will change, but this can be compensated for by using a smaller response, while computation performance is improved, so <a href="http://www.accelereyes.com/arrayfire/c/group__math__func__cpp__grad.htm"><span style="color: rgb(0, 128, 0);">grad</span></a> [can?] be used instead.</span> &lt;--This sentence is confusing and long. Split it up to improve clarity.
<pre>// Calculate image gradients
array ix, iy;
grad(ix, iy, img_gray);
</pre>
<h5>Compute second-order derivatives</h5>
At this point<del>,</del> we have both horizontal and vertical first order derivatives of our image, but Harris corner detector is processed with second-order derivatives, which can be easily computed in ArrayFire with simple element-wise multiplications of the first-order derivatives.
<pre>// Compute second-order derivatives
array ixx = ix * ix;
array ixy = ix * iy;
array iyy = iy * iy;
</pre>
<h5>Compute an isotropic window</h5>
To check the cornerness of each pixel of an image, it's not enough to check the pixel value itself. We must also check a small window around that pixel. To reduce noise sensitivity and have an isotropic response, Harris claims that a smooth circular window should be used<del>, and it</del>. This can be achieved by filtering of all three second-order derivatives with a Gaussian kernel.
<pre>// Compute a Gaussian kernel with standard deviation of 1.0 and length of 5 pixels
// These values can be changed to use a smaller or larger window
array gauss_filt = gaussiankernel(5, 5, 1.0, 1.0);

// Filter second-order derivatives with Gaussian kernel computed previously
ixx = convolve(ixx, gauss_filt);
ixy = convolve(ixy, gauss_filt);
iyy = convolve(iyy, gauss_filt);
</pre>
<h5>Compute trace and determinant</h5>
From a second-order Hessian matrix, we then compute the values of its trace and determinant. These are the values we will need later to calculate the Harris response for each pixel in the image.
<pre>// Calculate trace
array tr = ixx + iyy;
// Calculate determinant
array det = ixx * iyy - ixy * ixy;
</pre>
<h5>Compute Harris response</h5>
Next, we calculate the Harris response for each pixel. For this calculation, Harris uses an empirically defined constant <code>k</code>. This constant is usually very small, normally ranging from <code>0.04</code> to <code>0.06</code>. In our case, we will use <code>k = 0.04</code>, but we encourage you to play with this value and see how the result is affected.
<pre>// Calculate Harris response
array response = det - 0.04f * (tr * tr);
</pre>
<h5>Selection of good corners</h5>
Now we already have our Harris responses for each pixel in the image. This value is used to determine whether a pixel is a corner or not. The standard way is to perform a non-maximal suppression, meaning that for each pixel it can only be a corner if it is a maximum compared to its 8 adjacent pixels. At the moment we do not have a direct way to compute a non-maximal suppression with ArrayFire, we have to write our own kernel for that. Non-maximal suppression is <del>only</del> just one of the cool things we will be able to do with our new computer vision toolset for ArrayFire.

An alternative is to do the non-maximal suppression in a naive way. To do this, we will use ArrayFire's <a href="http://www.accelereyes.com/arrayfire/c/group__image__func__cpp__maxfilt.htm">maxfilt</a> function, <del>that</del> which filters the image with a mask of <code>NxM</code> pixel and returns the maximum value of the neighborhood to the pixel centered within the mask. In this case we will use a <code>3x3</code> pixel mask to return the maximum of the 8-neighborhood, including the central pixel itself.

The <a href="http://www.accelereyes.com/arrayfire/c/group__image__func__cpp__maxfilt.htm">maxfilt</a> function alone will not perform a non-maximal suppression, but by combining the function with a few other steps<del>,</del> we can get the proper responses.

The nature of a pixel can be determined by the Harris responses<del>,</del>; a negative response means that a pixel corresponds to the segment of an edge. If the response is small but positive, it corresponds to a flat region. Only a large response corresponds to a corner. By thresholding the response array with the minimum desired Harris response, <del>that</del> which, for our example, will be 10<sup>5</sup>, we can calculate a binary response array for which only pixels with a value greater than this threshold will be true. We then multiply it to the original response array to get a new array containing only responses greater that 10<sup>5</sup>.

Here<del>,</del> we will use a threshold of 10<sup>5</sup>. For the implementation presented in this article, we found that corners seem to be better for responses larger than 10<sup>4</sup>, but you can and should play with these values to see how it will affect the image you are working on.

The last step for this is testing whether each pixel is equal to the maximum Harris response of the neighborhood. If it is equal, then we keep it, else, the response is not a local maximum and we discard it.
<pre>// Gets maximum response for each 3x3 neighborhood
array max_resp = maxfilt(response, 3, 3);

// Discard responses that are not greater than threshold
array corners = response &gt; 1e5f;
corners = corners * response;

// Discard responses that are not equal to maximum neighborhood response,
// scale them to original response value
corners = (corners == max_resp) * corners;
</pre>
If you still think you want a non-maximal suppression kernel for yourself, you can write it instead of using the last piece of code above. If you have never written your own kernel to work with an ArrayFire array, check out the <a href="http://sandbox.arrayfire.com/custom-kernels-with-arrayfire/">Custom Kernels with ArrayFire</a> post. Writing the non-maximal suppression kernel itself will not take more than 20 or 30 lines of code.
<h5>Plotting corners</h5>
This last step is done on the <abbr title="Central Processing Unit">CPU</abbr> side<del>,</del>; we only do this to show the corners that are detected. This will most likely be unnecessary <del>Most likely, you will not need to do this</del> unless you want to visualize, but, for the sake of clarity, we will include this part to show the result.
<pre>// Gets host pointer to response data
float* h_corners = corners.host();

// Draw draw_len x draw_len crosshairs where the corners are
const int draw_len = 3;
for (int y = draw_len; y &lt; img_color.dims(0) - draw_len; y++) {
    for (int x = draw_len; x &lt; img_color.dims(1) - draw_len; x++) {
        // Only draws crosshair if is a corner
        if (h_corners[x * corners.dims(0) + y] &gt; 1e5f) {
            // Draw horizontal line of (draw_len * 2 + 1) pixels centered on the corner
            // Set only the first channel to 1 (green lines)
            img_color(y, seq(x-draw_len, x+draw_len), 0) = 0.f;
            img_color(y, seq(x-draw_len, x+draw_len), 1) = 1.f;
            img_color(y, seq(x-draw_len, x+draw_len), 2) = 0.f;

            // Draw vertical line of (draw_len * 2 + 1) pixels centered on  the corner
            // Set only the first channel to 1 (green lines)
            img_color(seq(y-draw_len, y+draw_len), x, 0) = 0.f;
            img_color(seq(y-draw_len, y+draw_len), x, 1) = 1.f;
            img_color(seq(y-draw_len, y+draw_len), x, 2) = 0.f;
        }
    }
}</pre>
<h5>Visualize results</h5>
Finally, the image can be visualized.
<pre>// Previews color image with green crosshairs
image(img_color);
</pre>
Below you can see Lena's picture after the Harris corner detection and plotting. The green crosshairs indicate where corners are.

<a href="http://sandbox.arrayfire.com/wp-content/uploads/2014/12/harris_lena.png"><img class="alignnone size-full wp-image-5710" src="http://sandbox.arrayfire.com/wp-content/uploads/2014/12/harris_lena.png" alt="Harris Corners - Lena" width="1024" height="600" /></a>

To test for correctness of the algorithm, nothing better than a picture with well-defined corners. Below you can see the picture of a white square on a black background and the corners marked by green crosshairs.

<a href="http://sandbox.arrayfire.com/wp-content/uploads/2014/12/harris_square.png"><img class="alignnone size-full wp-image-5711" src="http://sandbox.arrayfire.com/wp-content/uploads/2014/12/harris_square.png" alt="Harris Corners - Square" width="1024" height="600" /></a>

All sources to our examples can be found in our <a href="https://github.com/arrayfire/arrayfire_examples">ArrayFire examples project</a> on GitHub, and the complete code for this article can be viewed via this direct <a href="https://github.com/arrayfire/arrayfire_examples/blob/master/image_processing/harris_demo.cpp">link</a>.

We would like to hear from you, what you are working on the computer vision field and which features you would like to see in ArrayFire. Please share your thoughts and requests below or contact us at <a href="mailto:sales@arrayfire.com">sales@arrayfire.com</a>.
<h5>References</h5>
[1] Harris, Chris, and Mike Stephens. "A combined corner and edge detector." <em>Alvey vision conference</em>. Vol. 15. 1988.