GesturePak Recorder

GesturePak Recorder (GPR) is a Windows application for recording, editing and testing gestures which can be recognized with the GesturePak API in any .NET 4.x application.

The GPR uses the Microsoft Kinect to record your body positions into gestures that are saved as .XML files in your Documents\GesturePak folder.

Requirements and Setup

a) You need a Kinect for Windows v2 Sensor or an XBox One Kinect Sensor with the optional Kinect Adapter for Windows
   
b) You must install the Kinect For Windows SDK. Download it from kinectforwindows.com

Place the kinect about 1 meter off the floor and stand back about 2 meters

Place the Kinect

The Kinect for Windows device does a much better job of tracking you when the device is about 1 meter off the floor.

Stand about 2 meters from the Kinect for best results.

Your results may vary, but our tests have shown that you get the best results no closer than 2 meters, or about 7 feet, from the Kinect.


Using the GesturePak Recorder


Gestures

A gesture is a series of frames that are matched in sequence. You might have a "Raise Right Hand" gesture comprised of two frames. In the first frame your right hand might be at chest level, and in the second frame your right hand might be raised over your head. A gesture is matched when the user's joints match the position of each frame in order within a defined amount of time. By default this interval is .5 seconds. A "fudge factor" or margin of error allows for more or less accuracy when matching frames.

Main Screen

The "Main" screen lets you load a gesture for editing and testing, record a new gesture, or just interact to get a feel for how the Kinect tracks your body.

Main Screen

Click on the "Record" button to create a new gesture.

New Gesture

The "New Gesture" screen lets you define your gesture's properties before recording.

New Gesture

Let's look at these properties.

Name

Name your gesture. For a demo, let's call the first one "Wave". A gesture file will be created in your documents\gesturepak folder called wave.xml.

Time Required

How many seconds will you need to completely perform your gesture? The default is 5 seconds. Let's leave it at that.

Track Right Hand State

You can pick which joints you want to track on another screen. The hand state tracking feature allows you to track whether the hand is open, closed, or pointing (lasso). If you want the state of your right hand to be used in your gesture, select this option. In our demo, we will leave it un-checked.

Track Left Hand State

If you want the state of the left hand to be used in your gesture, select this option. In our demo, we will leave it un-checked.

Axis Tracking (X, Y, and Z)

You can restrict tracking in two ways: by axis and by joint. Joint selection happens on another screen. Joints can be tracked across, the X (horizontal), Y (vertical), and Z (front to back) axis. You can opt out of any of these to increase accuracy. By default, Z axis tracking is off. That lets you perform gestures with your hands (or other joints) at any distance from your body. In our demo, we will track only X and Y.

Once you are done and ready to record, press the "Start Recording" button.

CountDown

Yout get 10 seconds to get into position and get ready.

CountDown

After the countdown you will see "GO" on the screen:

Go

Now you get 5 seconds to perform your gesture. Bring your right hand straight up at eye level, hold it there for a second, and then swipe it to the left so it is just under your head.

Frame 1

Frame 2

After you have completed your gesture, and the 5 seconds are up, GesturePak Recorder advances to the "Edit" screen.

Edit screen

GesturePak Recorder has captured all the frames from the 5 seconds that you performed your gesture.

Use the mouse wheel to move from frame to frame. Press the "Animate" button to see an animation of your body's stick figure performing the gesture.

Trim Unwanted Frames

Keeping all the frames in your gesture file is a good idea. You can use this animation to instruct your users in how to perform the gesture correctly. However, for the purposes of being succinct, you can trim off any frames that don't add anything to the animation.

Trim Start

Use the mouse wheel to advance to the frame just before you started the gesture, and press the "Trim Start" button. Now the gesture starts with that frame. Be careful! All of the frames that came before are now gone and removed from your Gesture file.

Trim Start

Trim End

Use the mouse wheel to advance to the frame just after you completed the gesture and press the "Trim End" button. All remaining frames are removed. Again, double-check before you press the button.

Trim End

Fudge Factor

This is a value you can use to increase or decrease accuracy required in order to match frames. If you increase the value, you make it easier to match the frames, but allow for more false positives. Decreasing the value means your user has to be more accurate in positioning their joints close to the recorded positions, but will make it harder to match. We've found that a value of .2 is a good starting place.

Joint Tracking

GesturePak needs to track at least one joint in order to match your gesture. In this case, we're only interested in the right hand. Hover the mouse over the hand until you see "Right Hand" on the top of the screen, and then click on the "Right Hand" joint to select it for tracking. Notice that the green circle at the hand turns white to indicate it is being tracked.

Select Right Hand

Selecting Frames to Match

The final step in editing your gesture is to select which frames GesturePak will match against. It would be overkill (and introduce complexity) if GesturePak were to match every frame in your gesture. So, consider which frames are critical for defining this gesture, the fewer the better. Use the mouse wheel to select the first frame you want to match and then click the "Match" button. In my case, I selected frame #7.

Click the Match button to select the first frame of the gesture.

Now move to to the next frame in the gesture and again click the "Match" button. In my case, I selected frame #14.

Click the Match button to select the next frame of the gesture.

Continue this pattern (in your own gestures) until each frame you want to match has been selected.

Animate Matched Frames

If you want to animate only the matched frames, check the "Matched Frames Only" checkbox and press the "Animate" button.

Testing

You can test your gesture right here in GesturePak recorder by pressing the "Test" button. There are no time limits. Just stand back and perform the gesture. Swipe your hand from right to left just as you did when you recorded the gesture. When a match is identified, you'll see the word "Match" superimposed in yellow across the screen.

Successfully testing your gesture

Gesture Files

Gestures are saved in \Documents\GesturePak as XML files named with the same name as your gesture. For example, a gesture named "Right Wave" will be saved as "\Documents\GesturePak\Right Wave.xml"

You may use this .xml file with the GesturePak API to recognize it in your own applications. More on that later.

Sometimes you may want to selectively change allowed time for individual frames. There is no way to do this through the GPR, however you can edit the XML file in NotePad (don't use a word processor).

Each frame in the XML file starts like this:

  <Frame Name="Frame X" Match="False" xml:lang="en">
    <DurationMax>5000000</DurationMax>
    <DurationMin>0</DurationMin>
    <LeftHandState>0</LeftHandState>
    <RightHandState>2</RightHandState>
    <Tag />
    ...

Change the values for each frame and save the .XML file.

DurationMax is the Allowed Time, and can be set for each frame. The value is in tens of thousands of milliseconds. 5000000 is 500 milliseconds. Take the number of milliseconds and add 4 zeros to it. Yeah, I know. :)

Now You're Ready!

Now you are ready for the next step, loading the gesture into your application and matching it in real-time!

Click here for the GesturePak API Documentation