VirtualHome

A Multi-Agent Household Simulator


About
VirtualHome is a multi-agent platform to simulate activities in a household. Agents are represented as humanoid avatars, which can interact with the environment through high-level instructions. You can use VirtualHome to render videos of human activities, or train agents to perform complex tasks. VirtualHome also includes a Knowledge Base, providing instructions to perform a large set of activities.

Features
Card image cap
Interactive Objects

You can move and change the state of the objects in the environment. The environment can be programatically generated.

Card image cap
Humanoid Agents

We provide multiple human avatars as agents, with Path-planning, IK and Animations to generate realistic actions.

Card image cap
Multi-Modal

You can record activities across different modalities and using multiple camera views simulatenously.

Card image cap
Multi-Agent

You can add multiple agents in the environment, and control them concurrently.


News
  • 03/28/2022
    New
    - VirtualHome v2.3.0 is out! Check out the release notes here.
  • 07/27/2020 - We released VirtualHome v2, including multi-agent support and a setup for RL training. Check it out in the repo.
  • 07/27/2020 - The source code for the VirtualHome Simulator is now available.


Activity Knowledge Base

We have collected a knowledge base of activities people do at home. For every activity, we have descriptions of different ways to perform it, and a program that describes how to execute it. The knowledge base also contains information about states of objects in the environment, and typical locations of those objects.

Check out RobotHow for an overview of the knowledge base of activity programs.

VirtualHome in Research

Here is some of the research using VirtualHome


VirtualHome: Simulating Household Activities via Programs

Presents VirtualHome and proposes a model to predict programs from videos or descriptions. The predicted program is finetuned using RL to be executable in the simulator.

X. Puig*, K. Ra*, M. Boben*, J. Li, T. Wang, S. Fidler, A. Torralba.
Link paper
In Proc. Computer Vision and Pattern Recognition (CVPR), 2018.

Synthesizing Environment-Aware Activities via Activity Sketches

Represents activities via Sketches. Proposes a model to generate programs from sketches such that they are consistent with a target environment.

A. Liao*, X. Puig*, M. Boben, A. Torralba, S.Fidler.
Link paper
In Proc. Computer Vision and Pattern Recognition (CVPR), 2019.

Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

Presents a challenge where an agent has to infer a human-like agent's goals, and find the best way to achieve the goals efficiently.

X. Puig, T. Shu, S. Li, Z. Wang, A. Liao, J. Tenenbaum, S. Fidler, A. Torralba.
Link paper
In Proc. International Conference on Learning Representations


Team

Xavier Puig

MIT

Kevin Ra

McGill University

Marko Boben

University of Ljulbjana

Yuan-Hong Liao

University of Toronto

Tianmin Shu

MIT

Shuang Li

MIT

Jiaman Li

University of Toronto

Tingwu Wang

University of Toronto

K. Swain

MIT

Jordan Ren

MIT

Sanja Fidler

University of Toronto

Antonio Torralba

MIT

MIT University of Toronto


Contact
Reach out to xavierpuig@csail.mit.edu for questions, suggestions, and feedback.