Classes in the GesturePak namespace:
- Gesture defines a Gesture.
- Pose defines a Pose within a Gesture.
- GestureRecorder is used to record Poses into a Gesture.
- GestureMatcher is used to match Gestures.
- MatchingPose pairs a real-time pose with a Gesture, used by GestureMatcher.
- GestureMouse aids in using the Kinect to control the mouse.
Loading Gestures
Before you can match Gestures you must load the gesture XML files into Gesture objects.
Create a new Gesture object by passing in the gesture filename to the constructor. The GesturePak Recorder (GPR) saves gesture files in the \Documents\GesturePak folder. You can alternatively use the LoadFile method:
C#
string gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\\GesturePak\\wave.xml";
var g = new GesturePak.Gesture(gesturefile);
// or
var g = new GesturePak.Gesture();
g.LoadFile(gesturefile);
VB
Dim gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\GesturePak\wave.xml"
Dim g = new GesturePak.Gesture(gesturefile)
' or
Dim g = new GesturePak.Gesture
g.LoadFile(gesturefile)
Using the GestureMatcher API
You can load up multiple gestures into a GestureMatcher object by passing a List<Gesture> into the constructor.
There are two ways that you can use the GestureMatcher: Event Mode and
Polling Mode.
Event Mode
Pass in the list of Gesture objects AND a KinectSensor object. GestureMatcher will make sure the KinectSensor is running and then will commence to track it's skeleton.
In Event Mode you must call the StartRecognizing event to start the process, and StopRecognizing when you are done. Keep in mind that even though you have called StopRecognizing, the SkeletonFrameReady event will contine to be handled. It will just return immediately.
Use Event Mode if there is no other code handling the KinectSensor's SkeletonFrameReady event. The goal is to only have one event handler listening for performance reasons.
Example:
VB
Imports GesturePak
Imports Microsoft.Kinect
Class MainWindow
'-- Matcher
Private WithEvents matcher As GestureMatcher
'-- Path to gesture. Point this to any GesturePak created gesture
Private gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) & "\gesturepak\flap.xml"
Private Sub MainWindow_Loaded(sender As Object, e As System.Windows.RoutedEventArgs) Handles Me.Loaded
'-- Create your gesture objects (however many you want to recognize)
Dim g1 = New Gesture(gesturefile)
'-- Add it to a gestures collection
Dim gestures As New List(Of Gesture)
gestures.Add(g1)
'-- Create a new matcher from the Kinect sensor and the gestures
matcher = New GestureMatcher(KinectSensor.KinectSensors(0), gestures)
'-- Start recognizing your gestures!
matcher.StartRecognizing()
End Sub
Private Sub matcher_GestureMatch(gesture As Gesture) Handles matcher.GestureMatch
'-- We got a match!
Label1.Content = gesture.Name
End Sub
End Class
C#
using GesturePak;
using Microsoft.Kinect;
partial class MainWindow
{
// Matcher
private GestureMatcher matcher;
// Path to gesture. Point this to any GesturePak created gesture
private string gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\\GesturePak
\\flap.xml";
public MainWindow()
{
Loaded += MainWindow_Loaded;
}
private void MainWindow_Loaded(object sender, System.Windows.RoutedEventArgs e)
{
// Create your gesture objects (however many you want to test)
Gesture g1 = new Gesture(gesturefile);
// Add it to a gestures collection
List<Gesture> gestures = new List<Gesture>();
gestures.Add(g1);
// Create a new matcher from the Kinect sensor and the gestures
matcher = new GestureMatcher(KinectSensor.KinectSensors[0], gestures);
// hook up events
matcher.GestureMatch += matcher_GestureMatch;
// Start recognizing your gestures!
matcher.StartRecognizing();
}
private void matcher_GestureMatch(Gesture gesture)
{
// We got a match!
Label1.Content = gesture.Name;
}
}
Polling Mode
In Polling Mode, you do not pass in a KinectSensor. Rather, you pass data to it in real-time and then poll to determine if a gesture was matched.
In Polling Mode you do not call StartRecognizing or StopRecognizing, as your code drives the process. There is no KinectSensor involved so there is no handling of the SkeletonFrameReady event internally.
Use Polling Mode when you are integrating geture matching with your existing application, which is already handling the SkeletonFrameReady event.
GestureMatcher offers two methods for handling Skeleton data in real-time. You can call ProcessRealTimeSkeletonData passing in a Microsoft.Kinect.Skeleton object.
Alternatively you could call HandleSkeletonFrameReady from your KinectSensor.SkeletonFrameReady event handler, passing in the two event handler arguments: (System.Object sender, Microsoft.Kinect.SkeletonFrameReadyEventArgs e).
Which you use is up to you. HandleSkeletonFrameReady checks to see that there is a Frame, whether the KinectSensor is tracking, and whether the frame has data. It then copies the data out and passes it to ProcessRealTimeSkeletonData, which calls on the gesture matching code internally.
So, if you already have the Skeleton frame data, you can pass it to ProcessRealTimeSkeletonData, bypassing all that other goo.
Once control returns to your code, you can test to see which Poses and Gestures were matched. Check each Gesture in the Gestures collection's Matched boolean property to see if that gesture was matched.
To check the poses, you can go one step further and Check each Pose in each Gesture's Poses collection's Matched property
Example:
VB
Imports GesturePak
Imports Microsoft.Kinect
Class MainWindow
'-- Matcher
Private WithEvents matcher As GestureMatcher
'-- Grab the first Kinect Sensor
Private WithEvents kinect As KinectSensor = KinectSensor.KinectSensors(0)
'-- Path to gesture. Point this to any GesturePak created gesture
Private gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) & "\gesturepak\wave.xml"
Private Sub MainWindow_Loaded(sender As Object, e As System.Windows.RoutedEventArgs) Handles Me.Loaded
'-- Create your gesture objects (however many you want to recognize)
Dim g1 = New Gesture(gesturefile)
'-- Add it to a gestures collection
Dim gestures As New List(Of Gesture)
gestures.Add(g1)
'-- Create a new matcher from the gestures
matcher = New GestureMatcher(gestures)
'-- Start the Kinect
kinect.SkeletonStream.Enable(New Microsoft.Kinect.TransformSmoothParameters)
kinect.Start()
End Sub
Private Sub kinect_SkeletonFrameReady(ByVal sender As Object, _
ByVal e As Microsoft.Kinect.SkeletonFrameReadyEventArgs) Handles kinect.SkeletonFrameReady
'-- This is YOUR KinectSensor's SkeletonFrameReady event handler.
'You do your thing here. When you want to test for a gesture match do this:
'-- Get the frame
Dim frame = e.OpenSkeletonFrame
'-- Bail if it's nothing
If frame Is Nothing Then Return
Using frame
'-- Bail if no data is returned.
If frame.SkeletonArrayLength = 0 Then Return
'-- Get the data from the frame into an array
Dim data(frame.SkeletonArrayLength - 1) As Skeleton
frame.CopySkeletonDataTo(data)
'-- Is there really no data?
If data.Length = 0 Then Return
matcher.ProcessRealTimeSkeletonData(data(0))
End Using
'-- Alternatively, you could replace all of the above code with this one line:
' matcher.HandleSkeletonFrameReady(sender, e)
'-- Test for a match!
For Each gesture In matcher.Gestures
If gesture.Matched Then
'-- We got a match!
Label1.Content = gesture.Name
Exit For
End If
Next
End Sub
End Class
C#
using System;
using System.Collections.Generic;
using System.Windows;
using GesturePak;
using Microsoft.Kinect;
partial class MainWindow
{
// Matcher
private GestureMatcher matcher;
// Grab the first Kinect Sensor
private KinectSensor kinect = KinectSensor.KinectSensors[0];
// Path to gesture. Point this to any GesturePak created gesture
private string gesturefile = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) + "\\GesturePak\\wave.xml";
public MainWindow()
{
Loaded += MainWindow_Loaded;
}
private void MainWindow_Loaded(object sender, System.Windows.RoutedEventArgs e)
{
// Create your gesture objects (however many you want to test)
Gesture g1 = new Gesture(gesturefile);
// Add it to a gestures collection
List<Gesture> gestures = new List<Gesture>();
gestures.Add(g1);
// Create a new matcher from the gestures
matcher = new GestureMatcher(gestures);
// hook up events
kinect.SkeletonFrameReady += kinect_SkeletonFrameReady;
// Start the Kinect
kinect.SkeletonStream.Enable(new Microsoft.Kinect.TransformSmoothParameters());
kinect.Start();
}
private void kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
// This is YOUR KinectSensor's SkeletonFrameReady event handler.
// You do your thing here. When you want to test for a gesture match do this:
// Get the frame
SkeletonFrame frame = e.OpenSkeletonFrame();
// Bail if it's nothing
if (frame == null) return;
using (frame)
{
// Bail if no data is returned.
if (frame.SkeletonArrayLength == 0) return;
// Get the data from the frame into an array
var data = new Skeleton[frame.SkeletonArrayLength];
frame.CopySkeletonDataTo(data);
// Is there really no data?
if (data.Length == 0) return;
matcher.ProcessRealTimeSkeletonData(data[0]);
}
// Alternatively, you could replace all of the above code with this one line:
// matcher.HandleSkeletonFrameReady(sender, e);
// Test for a match!
foreach (Gesture gesture in matcher.Gestures)
{
if (gesture.Matched)
{
Label1.Content = gesture.Name;
return;
}
}
}
}