During the next months, we will be working on a lot of new Computer Vision functionality for ArrayFire. This means dozens of new functions specifically targeted for the most common and used applications in this field, like feature tracking, object classification, scene segmentation, optical flow, stereo-vision, among others, will be added to take full advantage of GPU devices.

Feature tracking consists of three basic steps:
<ol>
	<li>Detecting good or unique features, normally they are corners or blobs of an image;</li>
	<li>Extracting a descriptor for each feature, that consists of understanding the texture of a small patch around each feature;</li>
	<li>Descriptor matching, finding out the best match for each pair of descriptors (one from the object being tracked, another from a scene that potentially contains that object), if any.</li>
</ol>
To get started on what is coming to ArrayFire, in this article we dive deeper in the first step of feature tracking, detecting good features. Through a step-by-step guide, we will develop a simple corner detector, known as the Harris corner detector.

The Harris corner detector is very simple to implement, especially when you have so many good tools and libraries out there to do the core matrix manipulation. What we specifically need for Harris are some convolutions, together with a few element-wise matrix addition and multiplication operations, all of these operations are present in most scientific tools and libraries that work with matrix manipulation. I will try to avoid going into the mathematical details in this post and go directly to the implementation.

Before starting writing the code for the corner detector, we need to load an image. The image can be either color or gray-scale, but it will only work on the gray-scale version, so if it is a color image, we have to convert it before proceeding.
<pre>// Load a color image
array img_color = loadimage("lena.jpg", true);
// Convert the image from RGB to gray-scale
array img_gray = colorspace(img_color, af_gray, af_rgb);
// For visualization in ArrayFire, color images must be in the [0.0f-1.0f] interval
img /= 255.f;
</pre>
The first step in the Harris corner detector is to determine first order derivatives of an image, in both horizontal and vertical directions. This can be achieved by one convolution for each direction, with the special filter (-1, 0, 1)  for horizontal derivative and its transpose for vertical derivative.
<pre>// Load a first-order derivative filter to an array
const unsigned deriv_len = 3;
const float h_deriv[deriv_len] = {-1.0f, 0.0f, 1.0f};
array deriv(1, deriv_len, h_deriv);

// Compute first-order derivatives
array ix = convolve(img_gray, deriv);
array iy = convolve(img_gray, deriv.T());
</pre>
At this point, we have both horizontal and vertical first order derivatives of our image, but Harris corner detector is processed with second-order derivatives, which can be easily computed in ArrayFire with simple element-wise multiplications of the first-order derivatives.
<pre>// Compute second-order derivatives
array ixx = ix * ix;
array ixy = ix * iy;
array iyy = iy * iy;
</pre>
To check the cornerness of each pixel of an image, it's not enough to check the pixel value itself, but a small window around that pixel. To reduce noise sensitivity and have an isotropic response, Harris claims that a smooth circular window should be used, and it can be achieved by filtering of all three second-order derivatives with a Gaussian kernel.
<pre>// Compute a Gaussian kernel with standard deviation of 1.0 and length of 5 pixels
// These values can be changed to use a smaller or larger window
array gauss_filt = gaussiankernel(5, 5, 1.0, 1.0);

// Filter second-order derivatives with Gaussian kernel computed previously
ixx = convolve(ixx, gauss_filt);
ixy = convolve(ixy, gauss_filt);
iyy = convolve(iyy, gauss_filt);
</pre>
From a second-order Hessian matrix, we then compute the values of its trace and determinant. These are the values we will need later to calculate the Harris response for each pixel in the image.
<pre>// Calculate trace
array tr = ixx + iyy;
// Calculate determinant
array det = ixx * iyy - ixy * ixy;
</pre>
Finally, we then calculate the Harris response for each pixel. For this calculation, Harris uses an empirically defined constant <code>k</code>. This constant is usually very small, and usually it ranges from <code>0.04</code> to <code>0.06</code>. In our case, we will use <code>k = 0.04</code>, but we encourage you to play with this value and see how the result changes.
<pre>// Calculate Harris response
array response = det - 0.04f * (tr * tr);
</pre>
Now we already have our Harris responses for each pixel in the image. This value is used to determine whether a pixel is a corner or not. The standard way is to perform a non-maximal suppression, meaning that for each pixel it can only be a corner if it's a maximum compared to its 8 adjacent pixels. At the moment we don't have an easy way to compute a non-maximal suppression with ArrayFire, we have to write our own kernel for that, and that is only one of the cool things we will be able to do with our new computer vision toolset for ArrayFire.

To avoid taking it much longer and going into much more detail, we will avoid the non-maximal suppression in this post, but we encourage you to write your own kernel for non-maximal suppression, we promise you it will not take more than 20 or 30 lines of code.

The last computation to determine whether a pixel is a corner or not, is establish a threshold. The nature of a pixel can be determined by the Harris responses, a negative response means that a pixel corresponds to the segment of an edge. If the response is small but positive, it corresponds to a flat region. Only a large response corresponds to a corner.

Here, we will use a threshold of 10<sup>6</sup>. For the implementation presented in this article, we found that corners seem to be better for responses larger than 10<sup>5</sup>, but you can and should play with these values to see how it will affect the image you are working on.

This last step is done on the CPU side, we only do this to show the corners that are detected. Most likely, you will not need to do this unless you want to visualize, but for the sake of clarity, we include this part to show the result.
<pre class="wp-code-highlight prettyprint">// Gets host pointer to response data
float* h_response = response.host();

// Draw draw_len x draw_len crosses where the corners are
const int draw_len = 3;
for (int y = draw_len; y &lt; response.dims(0) - draw_len; y++) {
    for (int x = draw_len; x &lt; response.dims(1) - draw_len; x++) {
        // Only draws corner if response is &gt; 10^6
        if (h_response[x * response.dims(0) + y] &gt; 1e6f) {
            // Draw horizontal line of (draw_len * 2 + 1) pixels over center of corner
            // Set only the first channel to 1 (green lines)
            img_color(y, seq(x-draw_len, x+draw_len), 0) = 0.f;
            img_color(y, seq(x-draw_len, x+draw_len), 1) = 1.f;
            img_color(y, seq(x-draw_len, x+draw_len), 2) = 0.f;

            // Draw vertical line of (draw_len * 2 + 1) pixels over center of corner
            // Set only the first channel to 1 (green lines)
            img_color(seq(y-draw_len, y+draw_len), x, 0) = 0.f;
            img_color(seq(y-draw_len, y+draw_len), x, 1) = 1.f;
            img_color(seq(y-draw_len, y+draw_len), x, 2) = 0.f;
        }
    }
}
</pre>
Finally, the image can be visualized.
<pre>// Previews color image with green crosses
image(img_color);
</pre>
Below you can see Lena's picture after the Harris corner detection and plotting. Note the green crosses, many of them look like green circles, and this is because we omitted the non-maximal suppression. When we perform the non-maximum suppression, there will never be two adjacent corners.

<a href="http://sandbox.arrayfire.com/wp-content/uploads/2014/12/harris_lena1.png"><img class="alignnone size-full wp-image-5670" src="http://sandbox.arrayfire.com/wp-content/uploads/2014/12/harris_lena1.png" alt="Lena with Harris corners" width="1024" height="600" /></a>

We would like to hear from you what you are working on the computer vision field and which features you would like to see in ArrayFire. Please share your thoughts and requests below.

&nbsp;