A few weeks ago we added some computer vision functionality to our open source <a href="http://sandbox.arrayfire.com/download">ArrayFire GPU computing</a> library. Specifically, we implemented the <a href="http://www.springerlink.com/index/10.1007/11744023">FAST</a> feature extractor, <a href="http://www.springerlink.com/index/10.1007/978-3-642-15561-1">BRIEF</a> feature point descriptor, <a href="http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6126544">ORB</a> multi-resolution scale invariant feature extractor, and a <a href="http://en.wikipedia.org/wiki/Hamming_distance">Hamming distance</a> function. When combined, these functions enable you to find features in videos (or images) and track them between successive frames.

Here is a quick video showing these functions executing on a few frames from the <a href="http://www.blender.org/">Blender Foundation</a>'s <a href="http://www.bigbuckbunny.org">Big Buck Bunny</a> video:

<iframe width="854" height="510" src="https://www.youtube.com/embed/sHHvORwgFQ4" frameborder="0" allowfullscreen="allowfullscreen"></iframe>

Now that you have seen the functionality in action, how do you use these in your code? Assuming that you have a suitable function that can load your video frames in to ArrayFire "array" objects, it's pretty simple:

<code>// Run FAST on a frame and get the features:
af::features fast_features = af::fast(frame, 20, 9, 1, 0.05f);
</code>

The `af::features` object is a special structure which contains the `(x,y)` location of the features, their scores, and the number of features. To get access to this information simply use the `getX()`, `getY()`, `getScore()`, and `getNumFeatures()` functions:

<code>af::array x_pos = fast_features.getX();
af::array y_pos = fast_features.getY();
af::array scores = fast_features.getScore();
int N = fast_features.getNumFeatures();
</code>

ORB has a slightly different syntax:

<code>// Run ORB on a frame and get the features and descriptors
af::features orb_features;
af::array orb_descriptors;
af::orb(orb_features, orb_descriptors, frame, 20, 1000, 1.2, 1, false);
</code>

The features from `ORB` may be accessed in the same fashion as `FAST` described above. The `orb_descriptors` variable is a Nx8 array containing extracted descriptors (read the <a href="http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6126544">ORB paper</a> for further details on this structure).

The Hamming matcher I mention in the video is almost ready to be committed to our upstream development branch, so it is also worth mentioning how it can be used. Assuming you have a list of features from two sources called `a_features` and `b_features` respectively, you can compute their Hamming distance using ArrayFire as follows:

<code>af::array idx;
af::array distances;
af::array a_features;
af::array b_features;
af::hamming_matcher(idx, distances, a_features, b_features, 0, 1);
</code>

Then you may get the indicies of features whose distance is less than, say 50, using ArrayFire's `where` operator. After you have these inidices, you can pick out these near-matching features pretty easily:

<code>// select features with distances less than 50:
af::array near_matches = where(dist &lt; 50);
// and now extract information about those features:
array a_feat_x = a_features.getX()(near_matches)
array a_feat_y = a_features.getY()(near_matches)
...
</code>

Because these are ArrayFire array objects, they exist in your GPU's (or other accelerator device's) memory. To get access to them within your host code, you need to get a copy of the data and place it somewhere in RAM:

<code>float * features_x = a_feat_x.host<float>();
float * features_y = a_feat_y.host<float>();
</code>