GesturePak Recorder (GPR) is a Windows application for recording, editing and testing gestures which can be recognized with the GesturePak API
in any .NET 4.x application.
The GPR uses the Microsoft Kinect to record your body positions into gestures that are saved as .XML files in your Documents\GesturePak folder.
Requirements and Setup
Place the Kinect
The Kinect for Windows device does a much better job of tracking you
when the device is about 1 meter off the floor.
Stand about 2 meters from the Kinect for best results.
Your results may vary, but our tests have shown that you get the best
results no closer than 2 meters, or about 7 feet, from the Kinect.
Using the GesturePak Recorder
A gesture is a series of frames that are matched in sequence. You might have a "Raise Right Hand" gesture comprised of two frames. In the first frame your right hand might be at chest level, and in the second frame your right hand might be raised over your head. A gesture is matched when the user's joints match the position of each frame in order within a defined amount of time. By default this interval is .5 seconds. A "fudge factor" or margin of error allows for more or less accuracy when matching frames.
The "Main" screen lets you load a gesture for editing and testing, record a new gesture, or just interact to get a feel for how the Kinect tracks your body.
Click on the "Record" button to create a new gesture.
The "New Gesture" screen lets you define your gesture's properties before recording.
Let's look at these properties.
Name your gesture. For a demo, let's call the first one "Wave". A gesture file will be created in your documents\gesturepak folder called wave.xml.
How many seconds will you need to completely perform your gesture? The default is 5 seconds. Let's leave it at that.
Track Right Hand State
You can pick which joints you want to track on another screen. The hand state tracking feature allows you to track whether the hand is open, closed, or pointing (lasso). If you want the state of your right hand to be used in your gesture, select this option. In our demo, we will leave it un-checked.
Track Left Hand State
If you want the state of the left hand to be used in your gesture, select this option. In our demo, we will leave it un-checked.
Axis Tracking (X, Y, and Z)
You can restrict tracking in two ways: by axis and by joint. Joint selection happens on another screen. Joints can be tracked across, the X (horizontal), Y (vertical), and Z (front to back) axis. You can opt out of any of these to increase accuracy. By default, Z axis tracking is off. That lets you perform gestures with your hands (or other joints) at any distance from your body. In our demo, we will track only X and Y.
Once you are done and ready to record, press the "Start Recording" button.
Yout get 10 seconds to get into position and get ready.
After the countdown you will see "GO" on the screen:
Now you get 5 seconds to perform your gesture. Bring your right hand straight up at eye level, hold it there for a second, and then swipe it to the left so it is just under your head.
After you have completed your gesture, and the 5 seconds are up, GesturePak Recorder advances to the "Edit" screen.
GesturePak Recorder has captured all the frames from the 5 seconds that you performed your gesture.
Use the mouse wheel to move from frame to frame. Press the "Animate" button to see an animation of your body's stick figure performing the gesture.
Trim Unwanted Frames
Keeping all the frames in your gesture file is a good idea. You can use this animation to instruct your
users in how to perform the gesture correctly. However, for the purposes of being succinct, you can trim
off any frames that don't add anything to the animation.
Use the mouse wheel to advance to the frame just before you started the gesture, and press the "Trim Start" button. Now the gesture starts with that frame. Be careful! All of the frames that came before are now gone and removed from your Gesture file.
Use the mouse wheel to advance to the frame just after you completed the gesture and press the "Trim End" button. All remaining frames are removed. Again, double-check before you press the button.
This is a value you can use to increase or decrease accuracy required in order to match frames. If you increase the value, you make it easier to match the frames, but allow for more false positives. Decreasing the value means your user has to be more accurate in positioning their joints close to the recorded positions, but will make it harder to match. We've found that a value of .2 is a good starting place.
GesturePak needs to track at least one joint in order to match your gesture. In this case, we're only interested in the right hand. Hover the mouse over the hand until you see "Right Hand" on the top of the screen, and then click on the "Right Hand" joint to select it for tracking. Notice that the green circle at the hand turns white to indicate it is being tracked.
Selecting Frames to Match
The final step in editing your gesture is to select which frames GesturePak will match against. It would be overkill (and introduce complexity) if GesturePak were to match every frame in your gesture. So, consider which frames are critical for defining this gesture, the fewer the better. Use the mouse wheel to select the first frame you want to match and then click the "Match" button. In my case, I selected frame #7.
Now move to to the next frame in the gesture and again click the "Match" button. In my case, I selected frame #14.
Continue this pattern (in your own gestures) until each frame you want to match has been selected.
Animate Matched Frames
If you want to animate only the matched frames, check the "Matched Frames Only" checkbox and press the "Animate" button.
You can test your gesture right here in GesturePak recorder by pressing the "Test" button. There are no time limits.
Just stand back and perform the gesture. Swipe your hand from right to left just as you did when you recorded the gesture.
When a match is identified, you'll see the word "Match" superimposed in yellow across the screen.
Gestures are saved in \Documents\GesturePak as XML files named with the same name as your gesture. For example, a gesture named "Right Wave" will be saved as "\Documents\GesturePak\Right Wave.xml"
You may use this .xml file with the GesturePak API to recognize it in your own applications. More on that later.
Sometimes you may want to selectively change allowed time for individual frames. There is no way to do this through the GPR, however you can edit the XML file in NotePad (don't use a word processor).
Each frame in the XML file starts like this:
<Frame Name="Frame X" Match="False" xml:lang="en">
Change the values for each frame and save the .XML file.
DurationMax is the Allowed Time, and can be set for each frame. The value is in tens of thousands of milliseconds. 5000000 is 500 milliseconds. Take the number of milliseconds and add 4 zeros to it. Yeah, I know. :)