Install Unity We have been using Unity 2021.2. for development. We recommend to install it through Unity Hub.

Clone the repository with the Unity Source Code.

git clone

To test the simulator, we also recommend cloning the Unity API.

git clone

Third Party Assets

VirtualHome uses a set of third party assets to include more 3D objects in the environment, provide realistic motions for the agents, and allow procedural generation to build dynamic environments. If you want to further develop the simulator, you will need to purchase these assets separately and include them in the source code. Find here the documentation for how to install the assets.

We use the following assets in VirtualHome, which you will need to purchase to develop the tool:

  • Complete Home Interior Pack: Contains 3D models of objects to populate the scenes, as well as the apartment layouts`.

  • Final IK: Contains character animations, and tools to make the agent motions and object interactions more realistic.`.

  • DunGen: Contains procedural generation tools to create new environments during runtime.`.

Click on each of the asset links listed above, and purchase the assets using your Unity Account. In the Unity Editor, go into the Unity Asset Store and click on My Assets. You should see the 3 Assets you just purchased there. For each asset, click Download and after downloading click Import.

Note that you do not need to purchase the assets if you are only planning to use the executable, which you can download from This section is only needed if you want to build your own executables, or render scripts using the editor.

Interacting with the simulator

After the intial steps have been completed, you should be able to test and interacy with the simulator. Open a scene in VirtualHome using

File > Open Scene > Assets/Story Generator/Scene/Scene_0.unity.

Then, press the play button in Unity and make sure there are no error messages. If you see that the play button stays toggled, you are ready to interact with the simulator.

If you went through the previous steps and managed to press play without issues, you can start interacting with the simulator. For that, you need to use the VirtualHome API.

If you have cloned the repository, you can test it by entering in the repository and starting python.

cd virtualhome

And create a comunication object while Unity is in play mode.

from simulation.unity_simulator.comm_unity import UnityCommunication
comm = UnityCommunication()

This object will communicate with Unity using port 8080. You can change that in the source code or using the Unity Executable.

To make sure that the tool works, try resetting the scene, adding a character, and walking to the fridge. Still in python, run:

comm.render_script(['<char0> [walk] <fridge> (1)'], find_solution=True)

You should see in Unity an agent walking towards the fridge. You can find more examples in VirtualHome API section of the documentation.

Generating an Executable

You may want to interact with the simulator without using Unity. You can generate an executable so that VirtualHome runs without opening or installing Unity. You will then be able to run the tool both on your desktop or in headless servers.

To build an executable, open the project, click on the top menu in Unity and select: File > Build Settings

Make sure that the scene0 (handmade environments) and scene1 (procedural generation environments) are included in the build, select your target platform and architecture and press Build. After a few minutes, an executable will be generated allowing you to run the simulator.