A Multi-Agent Household Simulator

VirtualHome is a multi-agent platform to simulate activities in a household. Agents are represented as humanoid avatars, which can interact with the environment through high-level instructions. You can use VirtualHome to render videos of human activities, or train agents to perform complex tasks. VirtualHome also includes a Knowledge Base, providing instructions to perform a large set of activities.

Card image cap
Interactive Objects

You can move and change the state of the objects in the environment. The environment can be programatically generated.

Card image cap
Humanoid Agents

We provide multiple human avatars as agents, with Path-planning, IK and Animations to generate realistic actions.

Card image cap

You can record activities across different modalities and using multiple camera views simulatenously.

Card image cap

You can add multiple agents in the environment, and control them concurrently.

  • 03/28/2022
    - VirtualHome v2.3.0 is out! Check out the release notes here.
  • 07/27/2020 - We released VirtualHome v2, including multi-agent support and a setup for RL training. Check it out in the repo.
  • 07/27/2020 - The source code for the VirtualHome Simulator is now available.

Activity Knowledge Base

We have collected a knowledge base of activities people do at home. For every activity, we have descriptions of different ways to perform it, and a program that describes how to execute it. The knowledge base also contains information about states of objects in the environment, and typical locations of those objects.

Check out RobotHow for an overview of the knowledge base of activity programs.

VirtualHome in Research

Here is some of the research using VirtualHome

VirtualHome: Simulating Household Activities via Programs

Presents VirtualHome and proposes a model to predict programs from videos or descriptions. The predicted program is finetuned using RL to be executable in the simulator.

X. Puig*, K. Ra*, M. Boben*, J. Li, T. Wang, S. Fidler, A. Torralba.
Link paper
In Proc. Computer Vision and Pattern Recognition (CVPR), 2018.

Synthesizing Environment-Aware Activities via Activity Sketches

Represents activities via Sketches. Proposes a model to generate programs from sketches such that they are consistent with a target environment.

A. Liao*, X. Puig*, M. Boben, A. Torralba, S.Fidler.
Link paper
In Proc. Computer Vision and Pattern Recognition (CVPR), 2019.

Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

Presents a challenge where an agent has to infer a human-like agent's goals, and find the best way to achieve the goals efficiently.

X. Puig, T. Shu, S. Li, Z. Wang, A. Liao, J. Tenenbaum, S. Fidler, A. Torralba.
Link paper
In Proc. International Conference on Learning Representations


Xavier Puig


Kevin Ra

McGill University

Marko Boben

University of Ljulbjana

Yuan-Hong Liao

University of Toronto

Tianmin Shu


Shuang Li


Jiaman Li

University of Toronto

Tingwu Wang

University of Toronto

Kabir Swain

Jordan Ren


Sanja Fidler

University of Toronto

Antonio Torralba


MIT University of Toronto

Reach out to for questions, suggestions, and feedback.