Interaction with VirtualHome

You interact with VirtualHome by communicating with a Unity executable via Python commands. All communication happens with the class unity_simulator.UnityCommunication. We provide here the API of the class, with all the functions to interact with the simulator.

class unity_simulator.UnityCommunication(url='', port='8080', file_name=None, x_display=None, no_graphics=False, logging=True, timeout_wait=30, docker_enabled=False)[source]

Class to communicate with the Unity simulator and generate videos or agent behaviors

  • url (str) – which url to use to communicate
  • port (str) – which port to use to communicate
  • file_name (str) – location of the Unity executable. If provided, it will open the executable, if None, it wil assume that the executable is already running
  • x_display (str) – if using a headless server, display to use for rendering
  • no_graphics (bool) – whether to run the simualtor without graphics
  • logging (bool) – log simulator data
  • timeout_wait (int) – how long to wait until connection with the simulator is called unsuccessful
  • docker_enabled (bool) – whether the simulator is running in a docker container
add_camera(position=[0, 1, 0], rotation=[0, 0, 0])[source]

Add a new scene camera. The camera will be static in the scene.

  • position (list) – the position of the camera, with respect to the agent
  • rotation (list) – the rotation of the camera, with respect to the agent

succes (bool)

add_character(character_resource='Chars/Male1', position=None, initial_room='')[source]

Add a character in the scene.

  • character_resource (str) – which game object to use for the character
  • char_index (int) – the index of the character you want to move
  • position (list) – the position where you want to place the character
  • initial_room (str) – the room where you want to put the character,

if positon is not specified. If this is not specified, it places character in random location :return: succes (bool)

add_character_camera(position=[0, 1, 0], rotation=[0, 0, 0], name='new_camera')[source]

Add a new character camera. The camera will be added to every character you include in the scene, and it will move with the character. This must be called before adding any character.

  • position (list) – the position of the camera, with respect to the agent
  • rotation (list) – the rotation of the camera, with respect to the agent

the name of the camera, used for recording when calling render script


succes (bool)


Returns the number of cameras in the scene, including static cameras, and cameras for each character

Returns:pair success (bool), num_cameras (int)

Returns camera data for cameras given in camera_indexes list

Parameters:camera_indexes (list) – the list of cameras to return, can go from 0 to camera_count-1
Returns:pair success (bool), cam_data: (list): for every camera, the matrices with the camera parameters
camera_image(camera_indexes, mode='normal', image_width=640, image_height=480)[source]

Returns a list of renderings of cameras given in camera_indexes.

  • camera_indexes (list) – the list of cameras to return, can go from 0 to camera_count-1
  • mode (str) – what kind of camera rendering to return. Possible modes are: “normal”, “seg_inst”, “seg_class”, “depth”, “flow”, “albedo”, “illumination”, “surf_normals”
  • image_width (str) – width of the returned images
  • image_heigth (str) – height of the returned iamges

pair success (bool), images: (list) a list of images according to the camera rendering mode


Returns the number of cameras in the scene

Returns:pair success (bool), camera_names: (list): the names of the cameras defined fo rthe characters

Returns environment graph, at the current state

Returns:pair success (bool), graph: (dictionary)
expand_scene(new_graph, randomize=False, random_seed=-1, animate_character=False, ignore_placing_obstacles=False, prefabs_map=None, transfer_transform=True)[source]

Expands scene with the given graph. Given a starting scene without characters, it updates the scene according to new_graph, which contains a modified description of the scene. Can be used to add, move, or remove objects or change their state or size.

  • new_graph (dict) – a dictionary corresponding to the new graph of the form {‘nodes’: …, ‘edges’: …}
  • bool randomize (int) – a boolean indicating if the new positioni/types of objects should be random
  • random_seed (int) – seed to use for randomize. random_seed < 0 means that seed is not set
  • animate_character (bool) – boolean indicating if the added character should be frozen or not.
  • ignore_placing_obstacles (bool) – when adding new objects, if the transform is not specified, whether to consider if it collides with existing objects
  • prefabs_map (dict) – dictionary to specify which Unity game objects should be used when creating new objects
  • transfer_transform (bool) – boolean indicating if we should set the exact position of new added objects or not

pair success (bool), message: (str)


Obtain visible objects accoding to a given camera

Parameters:camera_index (int) – the camera for which you want to check the objects. Between 0 and camera_count-1
Returns:pair success (bool), msg: the object indices visible according to the camera

Return a mapping from rgb colors, shown on seg_inst to object id, specified in the environment graph.

Returns:pair success (bool), mapping: (dictionary)
move_character(char_index, pos)[source]

Move the character char_index to a new position

  • char_index (int) – the index of the character you want to move
  • pos (list) – the position where you want to place the character

succes (bool)

render_script(script, randomize_execution=False, random_seed=-1, processing_time_limit=10, skip_execution=False, find_solution=False, output_folder='Output/', file_name_prefix='script', frame_rate=5, image_synthesis=['normal'], save_pose_data=False, image_width=640, image_height=480, recording=False, save_scene_states=False, camera_mode=['AUTO'], time_scale=1.0, skip_animation=False)[source]

Executes a script in the simulator. The script can be single or multi agent, and can be used to generate a video, or just to change the state of the environment

  • script (list) – a list of script lines, of the form [‘<char{id}> [{Action}] <{object_name}> ({object_id})’]
  • randomize_execution (bool) – randomly choose elements
  • random_seed (int) – random seed to use when randomizing execution, -1 means that the seed is not set
  • find_solution (bool) – find solution (True) or use graph ids to determine object instances (False)
  • processing_time_limit (int) – time limit for finding a solution in seconds
  • skip_execution (int) – skip rendering, only check if a solution exists
  • output_folder (str) – folder to output renderings
  • file_name_prefix (str) – prefix of created files
  • frame_rate (int) – frame rate at which to generate the video
  • image_synthesis (str) – what information to save. Can be multiple at the same time. Modes are: “normal”, “seg_inst”, “seg_class”, “depth”, “flow”, “albedo”, “illumination”, “surf_normals”. Leave empty if you don’t want to generate anythign
  • save_pose_data (bool) – save pose data, a skeleton for every agent and frame
  • image_width (int) – image_height for the generated frames
  • image_height (int) – image_height for the generated frames
  • recoring (bool) – whether to record data with cameras
  • save_scene_states (bool) – save scene states (this will be unused soon)
  • camera_mode (list) – list with cameras used to render data. Can be a str(i) with i being a scene camera index or one of the cameras from character_cameras
  • time_scale (int) – accelerate time at which actions happen
  • skip_animation (bool) – whether agent should teleport/do actions without animation (True), or perform the animations (False)

pair success (bool), message: (str)


Reset scene. Deletes characters and scene chnages, and loads the scene in scene_index

Parameters:scene_index (int) – integer between 0 and 6, corresponding to the apartment we want to load
Returns:succes (bool)