GesturePak Recorder

GesturePak Recorder (GPR) is a Windows application for recording, editing and testing gestures which can be recognized with the GesturePak API in any .NET 4.0 application.

The GPR uses the Microsoft Kinect to record your body positions into gestures that are saved as .XML files in your Documents\GesturePak folder.

Requirements and Setup

a) You need a Kinect for Windows Device. Although people say GesturePak works with the Kinect for Xbox, we don't support it. The Kinect for Xbox's skeletal tracking is not as good as the Kinect for Windows.
   
b) You must install the Kinect For Windows Runtime. Download it from kinectforwindows.com

Using Speech Recognition

Kinect Microphone Setup

Make sure the Kinect is your default audio input device

  • Go to Control Panel / Sound Applet / Recording Tab


  • Right-Click on the device labeled:

               "Microphone Array - Microsoft Kinect USB Audio"


  • Select "Set as Default Device" if it's not already the default device

Train Your Computer's Voice Recognition

Train your PC to recognize your voice better (10 minutes)

  • Go to Control Panel / Speech Recognition


  • Set up Microphone as a Desktop microphone


  • Train your computer to better understand you.


  • Go through two passes if you have time.

Place the kinect at eye-level

Place the Kinect at eye level.

The Kinect for Windows device does a much better job of tracking you when the device is at eye level tilted down slightly.

Stand about 2 meters from the Kinect for best results.

Your results may vary, but our tests have shown that you get the best results no closer than 2 meters, or about 6 feet, from the Kinect.


Using the GesturePak Recorder


Using GestureMouse

GestureMouse lets you use your hand to move the mouse cursor and "click" the mouse.

To use the GestureMouse, wave your hand in front of you until you see a red sphere surrounding the mouse cursor. The cursor moves with your hand.

To click on an item, hold your hand still for three seconds so the sphere hovers over it. The sphere will blink three times. After the third blink, GestureMouse will click on the item.

If the sphere moves erratically or you have trouble keeping it in position, move your hand further away from your body so the Kinect can track you more easily.
GestureMouse

Turning Speech On

When speech is off, the GPR's title bar is red and the text reads "Turn Speech On." To turn it on, use the GestureMouse to click on the title bar. 

Turning Speech Off

When Speech is on, the GPR's title bar is green and the text reads "Turn Speech Off." To turn if off, say "Turn Speech Off" or use the GestureMouse to click on the title bar.

Navigating the GPR

Each screen's commands, or menu options, are listed on the right. If speech is on, you can say any command to select it. You can also use the GestureMouse to select any command.

You can also select "Previous" to go to the previous screen, "Next" to move to the Next screen, and "Close" or "Exit Program" to exit the GPR.

You can also say "Grow Font" to increase the font size or "Shrink Font" to decrease the font size.

Some of the screens have red and green dots on areas that you can click on to toggle on or off. When the dots are red, clicking on them turns them green and enables the option. Clicking on green dots turns them red and disables the option.

Gestures

A gesture is a series of poses that are matched in sequence. You might have a "Raise Right Hand" gesture comprised of two poses. In the first pose your right hand might be at chest level, and in the second pose your right hand might be raised over your head. 

You can restrict tracking in two ways: by axis and by joint.

Axis Tracking

Gestures can be tracked across, the X (horizontal), Y (vertical), and Z (front to back) axis. You can opt out of any of these to increase accuracy. By default, Z axis tracking is off. That lets you perform gestures with your hands (or other joints) at any distance from your body.

Joint Tracking

GesturePak tracks one or more joints. By default, the right hand is the only joint being tracked, but you can turn tracking on for any joint. The Microsoft Kinect defines 20 joints: Head, Shoulder Center, Left Shoulder, Right Shoulder, Left Elbow, Right Elbow, Left Wrist, Right Wrist, Left Hand, Right Hand, Spine, Hip Center, Left Hip, Right Hip, Left Knee, Right Knee, Left Ankle, Right Ankle, Left Foot, and Right Foot.



Gesture Library

Gesture Library

GesturePak comes with a handful of gestures that you can test and modify.

You can switch back and forth between the Gesture Library and your gestures by selecting "Gesture Library" and then "My Gestures."

The Gesture Library gestures are located in \Program Files (x86)\Franklins.Net\GesturePak 1.0\library.

Backup copies of these gestures are located in \Program Files (x86)\Franklins.Net\GesturePak 1.0\library Backup.


Creating a New Gesture

From the main screen select "Create New" and the Create/Edit screen comes up. After a second or two (or sometimes longer) your skeleton will appear and it will move as you move. 

Tips

Creating a gesture starts with determining what the poses will be. Here are some tips:
a) Limit the number of poses to the least significant amount. 
If you want to create a "flap your arms" gesture, you do not need to record more than two poses. The first one has your arms outstretched and your hands up at shoulder level, and the second pose has your arms still outstretched and your hands at hip level.
b) Limit the number of axes to the least significant amount.
In my previous example, the "flap your arms" gesture, you need to track the X and Y axis, but not the Z axis. That allows your flapper to flap with their hands in front of them or in back of them. You may even opt to track only the Y axis. That would allow your flapper to simply raise and lower their hands no matter where they were located otherwise.
c) Limit the number of joints to the least significant amount.
Sticking with the "flap your arms" example, there is no reason to track anything except the left and right hands. If the left hands are in position, of course the elbows are as well. Tracking your elbows or other joints would just decrease the accuracy.
Create a New Gesture

Step 1: Record your Poses

The "snapshot" command is used to take a pose, but I realize that you probably can't click on "snapshot" with the GestureMouse without destroying your pose.

Because you have to hold position while the GPR "takes a snapshot," there are two "hands-off" ways to tell the GPR to take a snapshot for a pose: using speech commands and using a custom gesture.

To take a snapshot with a speech command, you simply say the word "snapshot." Speech commands can be turned on and off. 

To take a snapshot with a custom gesture, simply flip your head to the left and then to the right. This custom gesture can be changed, but the head flip works well, because the head is usually not involved in a gesture. If it is involved, you can always use the speech command.
GestureMouse
Changing the Custom Snapshot Gesture

You can change the custom gesture by replacing the following file:

    \Program Files (x86)\Franklins.Net\GesturePak 1.0\library Backup\snapshot.xml

with a gesture file of your choice.

Selecting Poses

You can cycle through the poses by selecting "Next Pose" and "Previous Pose." You can also select "Animate" to see all the poses displayed one second at a time.

Inserting and Deleting Poses

Whether you are editing an existing gesture or creating a new one, you can insert a pose by selcting the pose before which you wish to insert a new one, and then creating a new pose by saying "snapshot" or performing the custom snapshot gesture. The new pose is automatically inserted before the current pose, and all of the pose numbers are adjusted accordingly.

You can delete a pose by selecting "Delete Pose"
GestureMouse
Tip

Be careful with the GestureMouse. If your pose lands your right hand on a command, you may find yourself back at the home screen or deleting poses. Future revisions of the GPR will allow you to toggle the GestureMouse on and off.
When you are done recording your poses, select "Next" to move to the next step: Name your Gesture.

Step 2: Name your Gesture

Name your gesture by using the GestureMouse to type the name using the on-screen keyboard. Be brief, as the gesture names have to fit in a selectable list box on the screen and still be readable.

Remember: If you have trouble holding the GestureMouse over a key, move your hand further away from your body to increase tracking accuracy.

Select "Next" when you are done naming your gesture.
GestureMouse

GestureMouse

Step 3: Select the Axes to Track

Select "X Axis", "Y Axis", or "Z Axis" to toggle them on or off. Select "Next" when you are done.

GestureMouse

Step 4: Select Joints to Track

Select any of the Joints to toggle tracking on or off. Select "Next" when you are done.


Where are the Gestures Saved?

Your gesture is saved in \Documents\GesturePak as an XML file named with the same name as your gesture. For example, a gesture named "Right Wave" will be saved as \Documents\GesturePak\Right Wave.xml

You may use this .xml file with the GesturePak API to recognize it in your own applications, although you should definitely test it before you put it into production. See Testing Gestures for details.

Editing a Gesture

To edit a gesture, first select the gesture by name, then select "Edit Gesture"

The Create/Edit screen appears where you can select, insert, and delete poses (see Creating a Gesture for specifics).

When you are done, you must select "Next" through the rest of the steps in creating a gesture.

Testing Gestures

From the main screen select "Test Gestures" to bring up the Gesture Testing screen.

Selecting Gestures for Testing

All of the gestures are listed on the left. To select a gesture for testing, select it from the list (or say it's name if speech is on). The gesture name will jump from the list of all gestures to the list of selected gestures.

You may notice that if only one gesture is selected, there are some other options available. We will talk about them in a minute, but note that you can test more than one gesture at a time by selecting them from the list on the left.
GestureMouse

GestureMouse

Testing a Single Gesture

If you haven't tested your gesture by itself, you should do that first so you can adjust some of its parameters.

Perform The Test

Select "Begin Test." Your "live" skeleton appears in the black box. Go ahead and perform the gesture.

If the thest is successful you will see the name of the geture highlighted in yellow at the top of the screen. If you selected your gesture with a speech command, you'll also hear yourself speaking the name of the gesture.

If it doesn't work you may need to adjust the gesture's parameters. There are two settings: Acuracy, and Allowed Time.
GestureMouse

Adjusting Accuracy

Accuracy is a term that is inversely related to the margin of error around a pose. With more accuracy, the user has to get closer to the recorded X/Y/Z position of the joints being tracked. With less accuracy, The user does not have to be as close to the recorded X/Y/Z position of the joints being tracked.

You may find that more or less accuracy is required in order for the gesture to be recognized. Allow several people to test it before deciding where to set Accuracy. Out of the box, the accuracy is pretty good.

Select "More Accuracy" or "Less Accuracy" to adjust accuracy. If these options are not available, then you have more than one gesture selected for testing. Doh!
GestureMouse

Adjusting Allowed Time

Allowed Time is the maximum amount of time allowed between poses. By default it is 500 milliseconds. So by default, you have half a second to get from pose to pose. If you take longer than that, you'll have to start over.

You may find that more or less time allowed makes a significant difference in how well your gesture is recognized. If it's too short, the user will have to rush through the poses in order to match. If it's too long, you'll get more false positives.
GestureMouse

Adjusting Axis Tracking, Joint Tracking, and Allowed Time of Individual Poses

Sometimes you may want to selectively change these parameters for individual poses. There is no way to do this through the GPR, however you can edit the XML file in NotePad (don't use a word processor).

Each pose in the XML file starts like this:

  <pose name="Pose X">
    <TrackXAxis>false</TrackXAxis>
    <TrackYAxis>true</TrackYAxis>
    <TrackZAxis>false</TrackZAxis>
    <DurationMax>5000000</DurationMax>
    <DurationMin>0</DurationMin>
    <TrackHead>false</TrackHead>
    <TrackHipCenter>false</TrackHipCenter>
    <TrackLeftAnkle>false</TrackLeftAnkle>
    <TrackLeftElbow>false</TrackLeftElbow>
    ...

Change the values for each pose and save the .XML file.

DurationMax is the Allowed Time, and can be set for each pose. The value is in tens of thousands of milliseconds. 5000000 is 500 milliseconds. Take the number of milliseconds and add 5 zeros to it. Yeah, I know. :)

Now You're Ready!

Now you are ready for the next step, loading the gesture into your application and matching it in real-time!

Click here for the GesturePak API Documentation