Egocentric video data of household tasks
Everyday chores filmed through the eyes of the person doing them, in real homes in the US, India, Mexico and Jordan: cooking, tidying, folding laundry, washing up, stocking the fridge and pantry. Every clip is cut into steps, and each step is described in plain words by people, with the objects handled and the hand used.
Specification
- Footage
- First-person video from head- or chest-mounted phones, in real kitchens, living rooms, bedrooms and laundry areas
- Tasks
- Cooking, washing up, tidying, folding laundry, making beds, stocking the fridge and pantry
- People and homes
- Consenting adults in their own homes, with different layouts, appliances and everyday clutter
- Annotation
- Step start and end times, a written description of each step in plain English, objects handled, left or right hand, success or retry
- Privacy
- Other people’s faces, family photos, screens and documents blurred
- Quality check
- Every clip is checked step by step by several trained reviewers
Example of the data
What you see, and the file you get.
| open fridge | 00:03–00:05 | left hand · fridge door |
|---|---|---|
| take out eggs | 00:05–00:09 | right hand · egg carton |
| crack egg into bowl | 00:14–00:18 | both hands · glass bowl |
| video_file | task | step | start | end | description | hand |
|---|---|---|---|---|---|---|
| clip_0012.mp4 | make_omelette | 1 | 3.2 | 5.1 | Open the fridge door | left |
| clip_0012.mp4 | make_omelette | 2 | 5.1 | 9.4 | Take the egg carton from the middle shelf | right |
| clip_0012.mp4 | make_omelette | 3 | 9.4 | 13.8 | Put the carton on the counter and close the fridge | both |
| clip_0012.mp4 | make_omelette | 4 | 14 | 18.6 | Crack two eggs into the glass bowl | both |
| clip_0047.mp4 | fold_laundry | 1 | 0.8 | 6.3 | Pick a T-shirt from the basket and shake it out | both |
Sample rows. Also as JSON or Excel.
Contact us to get the dataset
Tell us what you need. We walk you through the data, its formats and how it is checked on a short call.
What it is used for
Household robots
Teach robots the order of real chores and how people reach, grasp and put things down.
Robot foundation models
Pre-train vision-language-action models on human video from many different homes.
Activity and step recognition
Recognise which step of a task someone is on, and what comes next.
Smart glasses and assistants
Build assistants that see what you see and guide you through a task.
Questions
Which homes are filmed?
Real homes in the US, India, Mexico and Jordan: kitchens, living rooms, bedrooms and laundry areas of every size, with the appliances and clutter people really live with.
Who writes the step descriptions?
People, not models: each step is written in plain English while watching the video, and reviewers check its timing and wording. Your own step vocabulary can be used instead.
Can I see samples first?
Yes. Request samples with the form on this page and we will send them with the full specification.
Tell us what your model needs.
Request samples and the full specification. We reply within a couple of business days.
Request samples