Hand manipulation video data with hand keypoints
Skilled hand work filmed up close on people’s own phones: assembling, sorting, folding cloth, preparing food, using tools and handling small parts, at real tables and workbenches. Reviewers add hand keypoints to every frame and label every reach, grasp, move and release as a step.
Specification
- Footage
- Close-up phone video from one or two angles (head- or chest-mounted, and a phone on a stand beside the work), at real tables, counters and workbenches
- Tasks
- Assembly, sorting and packing, folding cloth, food prep, using hand tools, plugging in cables and handling small parts
- Hand keypoints
- Finger and wrist keypoints for both hands on every frame, added and checked by reviewers
- Annotation
- Steps with start and end times, action, grasp type, object and hand (left, right or both), success or failure
- People
- Consenting adults, right- and left-handed, novices and people skilled at the task; faces kept out of frame or blurred
- Quality check
- Every step boundary, grasp label and keypoint track is checked by several trained reviewers
Example of the data
What you see, and the file you get.
| pinch grasp | M3 screw | right hand · 00:04.2 |
|---|---|---|
| place | screw into bracket hole | right hand · 00:05.8 |
| power grasp | screwdriver | right hand · 00:07.1 |
| video_file | view | start | end | hand | action | grasp |
|---|---|---|---|---|---|---|
| hands_0031.mp4 | head | 3.6 | 4.2 | right | reach_m3_screw | |
| hands_0031.mp4 | head | 4.2 | 5.8 | right | pick_m3_screw | pinch |
| hands_0031.mp4 | head | 5.8 | 6.9 | right | place_in_bracket_hole | pinch |
| hands_0031.mp4 | head | 5.8 | 11.6 | left | hold_bracket | palm |
| hands_0031.mp4 | head | 7.1 | 11.6 | right | turn_screwdriver | power |
| hands_0031_side.mp4 | side | 7.1 | 11.6 | right | turn_screwdriver | power |
Sample rows. Also as JSON or Excel.
Contact us to get the dataset
Tell us what you need. We walk you through the data, its formats and how it is checked on a short call.
What it is used for
Robot dexterity
Teach robot hands how people grasp, turn and place real objects.
Learning from demonstration
Train manipulation policies on human video with every step and grasp labelled.
Hand pose estimation
Train and test hand tracking on close-up, fast and partly hidden hands.
AR, VR and wearables
Build camera-based hand tracking and gesture input that holds up on real tasks.
Questions
What comes with the video?
Hand keypoints on every frame and step labels with start and end times, grasp type, object and hand, all added by reviewers on the video’s own timeline.
Which tasks are recorded?
Assembly, packing, cloth folding, food prep, tool use, cables and small parts, by novices and by people skilled at the task. Your own task list and grasp taxonomy can be used.
Can I see samples first?
Yes. Request samples with the form on this page and we will send them with the full specification.
Tell us what your model needs.
Request samples and the full specification. We reply within a couple of business days.
Request samples