Image & video data · US, India, Mexico & Jordan

Egocentric video data of household tasks

Everyday chores filmed through the eyes of the person doing them, in real homes in the US, India, Mexico and Jordan: cooking, tidying, folding laundry, washing up, stocking the fridge and pantry. Every clip is cut into steps, and each step is described in plain words by people, with the objects handled and the hand used.

Specification

Footage
First-person video from head- or chest-mounted phones, in real kitchens, living rooms, bedrooms and laundry areas
Tasks
Cooking, washing up, tidying, folding laundry, making beds, stocking the fridge and pantry
People and homes
Consenting adults in their own homes, with different layouts, appliances and everyday clutter
Annotation
Step start and end times, a written description of each step in plain English, objects handled, left or right hand, success or retry
Privacy
Other people’s faces, family photos, screens and documents blurred
Quality check
Every clip is checked step by step by several trained reviewers

Example of the data

What you see, and the file you get.

Preview
open fridge00:03–00:05left hand · fridge door
take out eggs00:05–00:09right hand · egg carton
crack egg into bowl00:14–00:18both hands · glass bowl
steps.csvCSV · JSON · Excel
video_filetaskstepstartenddescriptionhand
clip_0012.mp4make_omelette13.25.1Open the fridge doorleft
clip_0012.mp4make_omelette25.19.4Take the egg carton from the middle shelfright
clip_0012.mp4make_omelette39.413.8Put the carton on the counter and close the fridgeboth
clip_0012.mp4make_omelette41418.6Crack two eggs into the glass bowlboth
clip_0047.mp4fold_laundry10.86.3Pick a T-shirt from the basket and shake it outboth

Sample rows. Also as JSON or Excel.

Contact us to get the dataset

Tell us what you need. We walk you through the data, its formats and how it is checked on a short call.

Contact us

What it is used for

  • Household robots

    Teach robots the order of real chores and how people reach, grasp and put things down.

  • Robot foundation models

    Pre-train vision-language-action models on human video from many different homes.

  • Activity and step recognition

    Recognise which step of a task someone is on, and what comes next.

  • Smart glasses and assistants

    Build assistants that see what you see and guide you through a task.

Questions

Which homes are filmed?

Real homes in the US, India, Mexico and Jordan: kitchens, living rooms, bedrooms and laundry areas of every size, with the appliances and clutter people really live with.

Who writes the step descriptions?

People, not models: each step is written in plain English while watching the video, and reviewers check its timing and wording. Your own step vocabulary can be used instead.

Can I see samples first?

Yes. Request samples with the form on this page and we will send them with the full specification.

Tell us what your model needs.

Request samples and the full specification. We reply within a couple of business days.

Request samples