GesturePak API

The GesturePak API lets you match gestures created with the GesturePak Recorder in real-time in any .NET 4.0 app that uses the Microsoft Kinect For Windows SDK.

Requirements and Setup

See the Requirements and Setup section of the GesturePak Recorder documentation.

References

Any app that you build with GesturePak will need to have a reference to the following:

Assembly Location
Microsoft.Kinect.dll
\Program Files\Microsoft SDKs\Kinect\vX.X\Assemblies
GesturePakObjects.dll
\Program Files (x86)\Franklins.Net\GesturePak X.X\assembly

Classes in the GesturePak namespace:

Loading Gestures

Before you can match Gestures you must load the gesture XML files into Gesture objects. 

Create a new Gesture object by passing in the gesture filename to the constructor. The GesturePak Recorder (GPR) saves gesture files in the \Documents\GesturePak folder. You can alternatively use the LoadFile method:

C#

    string gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\\GesturePak\\wave.xml";
    var g = new GesturePak.Gesture(gesturefile);

    // or

    var g = new GesturePak.Gesture();
    g.LoadFile(gesturefile);

VB

    Dim gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\GesturePak\wave.xml"
    Dim g = new GesturePak.Gesture(gesturefile)

    '  or

    Dim g = new GesturePak.Gesture
    g.LoadFile(gesturefile)

Using the GestureMatcher API

You can load up multiple gestures into a GestureMatcher object by passing a List<Gesture> into the constructor.
There are two ways that you can use the GestureMatcher: Event Mode and Polling Mode

Event Mode

Pass in the list of Gesture objects AND a KinectSensor object. GestureMatcher will make sure the KinectSensor is running and then will commence to track it's skeleton. 

In Event Mode you must call the StartRecognizing event to start the process, and StopRecognizing when you are done. Keep in mind that even though you have called StopRecognizing, the SkeletonFrameReady event will contine to be handled. It will just return immediately.

Use Event Mode if there is no other code handling the KinectSensor's SkeletonFrameReady event. The goal is to only have one event handler listening for performance reasons.

Example:
VB

 

Imports GesturePak
Imports Microsoft.Kinect
Class MainWindow
    '-- Matcher
    Private WithEvents matcher As GestureMatcher
    '-- Path to gesture. Point this to any GesturePak created gesture
    Private gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) & "\gesturepak\flap.xml"

    Private Sub MainWindow_Loaded(sender As Object, e As System.Windows.RoutedEventArgs) Handles Me.Loaded
        '-- Create your gesture objects (however many you want to recognize)
        Dim g1 = New Gesture(gesturefile)
        '-- Add it to a gestures collection
        Dim gestures As New List(Of Gesture)
        gestures.Add(g1)
        '-- Create a new matcher from the Kinect sensor and the gestures
        matcher = New GestureMatcher(KinectSensor.KinectSensors(0), gestures)
        '-- Start recognizing your gestures!
        matcher.StartRecognizing()
    End Sub

    Private Sub matcher_GestureMatch(gesture As Gesture) Handles matcher.GestureMatch
        '-- We got a match!
        Label1.Content = gesture.Name
    End Sub
End Class
C#
using GesturePak;
using Microsoft.Kinect;
partial class MainWindow
{
    // Matcher
    private GestureMatcher matcher;
    // Path to gesture. Point this to any GesturePak created gesture
    private string gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\\GesturePak\\flap.xml";
    public MainWindow()
    {
        Loaded += MainWindow_Loaded;
    }

    private void MainWindow_Loaded(object sender, System.Windows.RoutedEventArgs e)
    {
        // Create your gesture objects (however many you want to test)
        Gesture g1 = new Gesture(gesturefile);
        // Add it to a gestures collection
        List<Gesture> gestures = new List<Gesture>();
        gestures.Add(g1);
        // Create a new matcher from the Kinect sensor and the gestures
        matcher = new GestureMatcher(KinectSensor.KinectSensors[0], gestures);
        // hook up events
        matcher.GestureMatch += matcher_GestureMatch;
        // Start recognizing your gestures!
        matcher.StartRecognizing();
    }

    private void matcher_GestureMatch(Gesture gesture)
    {
        // We got a match!
        Label1.Content = gesture.Name;
    }
}

Polling Mode

In Polling Mode, you do not pass in a KinectSensor. Rather, you pass data to it in real-time and then poll to determine if a gesture was matched.

In Polling Mode you do not call StartRecognizing or StopRecognizing, as your code drives the process. There is no KinectSensor involved so there is no handling of the SkeletonFrameReady event internally. 

Use Polling Mode when you are integrating geture matching with your existing application, which is already handling the SkeletonFrameReady event.

GestureMatcher offers two methods for handling Skeleton data in real-time. You can call ProcessRealTimeSkeletonData passing in a Microsoft.Kinect.Skeleton object.

Alternatively you could call HandleSkeletonFrameReady from your KinectSensor.SkeletonFrameReady event handler, passing in the two event handler arguments: (System.Object sender, Microsoft.Kinect.SkeletonFrameReadyEventArgs e).

Which you use is up to you. HandleSkeletonFrameReady checks to see that there is a Frame, whether the KinectSensor is tracking, and whether the frame has data. It then copies the data out and passes it to ProcessRealTimeSkeletonData, which calls on the gesture matching code internally.

So, if you already have the Skeleton frame data, you can pass it to ProcessRealTimeSkeletonData, bypassing all that other goo.
Once control returns to your code, you can test to see which Poses and Gestures were matched. Check each Gesture in the Gestures collection's Matched boolean property to see if that gesture was matched.

To check the poses, you can go one step further and Check each Pose in each Gesture's Poses collection's Matched property

Example:
VB
Imports GesturePak
Imports Microsoft.Kinect

Class MainWindow

    '-- Matcher
    Private WithEvents matcher As GestureMatcher
    '-- Grab the first Kinect Sensor
    Private WithEvents kinect As KinectSensor = KinectSensor.KinectSensors(0)
    '-- Path to gesture. Point this to any GesturePak created gesture
    Private gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) & "\gesturepak\wave.xml"

    Private Sub MainWindow_Loaded(sender As Object, e As System.Windows.RoutedEventArgs) Handles Me.Loaded
        '-- Create your gesture objects (however many you want to recognize)
        Dim g1 = New Gesture(gesturefile)
        '-- Add it to a gestures collection
        Dim gestures As New List(Of Gesture)
        gestures.Add(g1)
        '-- Create a new matcher from the gestures
        matcher = New GestureMatcher(gestures)
        '-- Start the Kinect
        kinect.SkeletonStream.Enable(New Microsoft.Kinect.TransformSmoothParameters)
        kinect.Start()
    End Sub

    Private Sub kinect_SkeletonFrameReady(ByVal sender As Object, _
        ByVal e As Microsoft.Kinect.SkeletonFrameReadyEventArgs) Handles kinect.SkeletonFrameReady

        '-- This is YOUR KinectSensor's SkeletonFrameReady event handler.
        'You do your thing here. When you want to test for a gesture match do this:

        '-- Get the frame
        Dim frame = e.OpenSkeletonFrame
        '-- Bail if it's nothing
        If frame Is Nothing Then Return
        Using frame
            '-- Bail if no data is returned.
            If frame.SkeletonArrayLength = 0 Then Return
            '-- Get the data from the frame into an array
            Dim data(frame.SkeletonArrayLength - 1) As Skeleton
            frame.CopySkeletonDataTo(data)
            '-- Is there really no data?
            If data.Length = 0 Then Return
            matcher.ProcessRealTimeSkeletonData(data(0))
        End Using

        '-- Alternatively, you could replace all of the above code with this one line:
        '   matcher.HandleSkeletonFrameReady(sender, e)

        '-- Test for a match!
        For Each gesture In matcher.Gestures
            If gesture.Matched Then
                '-- We got a match!
                Label1.Content = gesture.Name
                Exit For
            End If
        Next

    End Sub
End Class
C#
using System;
using System.Collections.Generic;
using System.Windows;
using GesturePak;
using Microsoft.Kinect;
partial class MainWindow
{
    // Matcher
    private GestureMatcher matcher;
    // Grab the first Kinect Sensor
    private KinectSensor kinect = KinectSensor.KinectSensors[0];
    // Path to gesture. Point this to any GesturePak created gesture
    private string gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\\GesturePak\\wave.xml";

    public MainWindow()
    {
        Loaded += MainWindow_Loaded;
    }

    private void MainWindow_Loaded(object sender, System.Windows.RoutedEventArgs e)
    {

// Create your gesture objects (however many you want to test)

        Gesture g1 = new Gesture(gesturefile);
        // Add it to a gestures collection
        List<Gesture> gestures = new List<Gesture>();
        gestures.Add(g1);
        // Create a new matcher from the gestures
        matcher = new GestureMatcher(gestures);
        // hook up events
        kinect.SkeletonFrameReady += kinect_SkeletonFrameReady;
        // Start the Kinect
        kinect.SkeletonStream.Enable(new Microsoft.Kinect.TransformSmoothParameters());
        kinect.Start();
    }

    private void kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
    { 
        // This is YOUR KinectSensor's SkeletonFrameReady event handler.
        // You do your thing here. When you want to test for a gesture match do this:

        // Get the frame
        SkeletonFrame frame = e.OpenSkeletonFrame();
        // Bail if it's nothing
        if (frame == null) return;

        using (frame)
        {
            // Bail if no data is returned.
            if (frame.SkeletonArrayLength == 0) return;
            // Get the data from the frame into an array
            var data = new Skeleton[frame.SkeletonArrayLength];
            frame.CopySkeletonDataTo(data);
            // Is there really no data?
            if (data.Length == 0) return;
            matcher.ProcessRealTimeSkeletonData(data[0]);
        }

        // Alternatively, you could replace all of the above code with this one line:
        //   matcher.HandleSkeletonFrameReady(sender, e);

        // Test for a match!
        foreach (Gesture gesture in matcher.Gestures)
        {
            if (gesture.Matched)
            {
                Label1.Content = gesture.Name;
                return;
            }
        }
    }
}