Interaction with VirtualHome

You interact with VirtualHome by communicating with a Unity executable via Python commands. All communication happens with the class unity_simulator.UnityCommunication. We provide the API of the class, with all the functions to interact with the simulator.

class unity_simulator.UnityCommunication(url='127.0.0.1', port='8080', file_name=None, x_display=None, no_graphics=False, logging=True, timeout_wait=30, docker_enabled=False)

Class to communicate with the Unity simulator and generate videos or agent behaviors

Parameters

• url (str) – which url to use to communicate

• port (str) – which port to use to communicate

• file_name (str) – location of the Unity executable. If provided, it will open the executable, if None, it wil assume that the executable is already running

• x_display (str) – if using a headless server, display to use for rendering

• no_graphics (bool) – whether to run the simualtor without graphics

• logging (bool) – log simulator data

• timeout_wait (int) – how long to wait until connection with the simulator is called unsuccessful

• docker_enabled (bool) – whether the simulator is running in a docker container

activate_physics(gravity=- 10)

Activates gravity and realistic collisions in the environment

Parameters

gravity (list) – int of gravity value experienced in the environment

Returns

success (bool)

add_camera(position=[0, 1, 0], rotation=[0, 0, 0], field_view=40)

Add a new scene camera. The camera will be static in the scene.

Parameters

• position (list) – the position of the camera, with respect to the agent

• rotation (list) – the rotation of the camera, with respect to the agent

• field_view (list) – the field of view of the camera

Returns

succes (bool)

add_character(character_resource='Chars/Male1', position=None, initial_room='')

Add a character in the scene.

Parameters

character_resource (str) – which game object to use for the character

# :param int char_index: the index of the character you want to move :param list position: the position where you want to place the character :param str initial_room: the room where you want to put the character, if position is not specified. If this is not specified, it places character in random location

Returns

success (bool)

add_character_camera(position=[0, 1, 0], rotation=[0, 0, 0], name='new_camera')

Add a new character camera. The camera will be added to every character you include in the scene, and it will move with the character. This must be called before adding any character.

Parameters

• position (list) – the position of the camera, with respect to the agent

• rotation (list) – the rotation of the camera, with respect to the agent

Name

the name of the camera, used for recording when calling render script

Returns

succes (bool)

camera_count()

Returns the number of cameras in the scene, including static cameras, and cameras for each character

Returns

pair success (bool), num_cameras (int)

camera_data(camera_indexes)

Returns camera data for cameras given in camera_indexes list

Parameters

camera_indexes (list) – the list of cameras to return, can go from 0 to camera_count-1

Returns

pair success (bool), cam_data: (list): for every camera, the matrices with the camera parameters

camera_image(camera_indexes, mode='normal', image_width=640, image_height=480)

Returns a list of renderings of cameras given in camera_indexes.

Parameters

• camera_indexes (list) – the list of cameras to return, can go from 0 to camera_count-1

• mode (str) – what kind of camera rendering to return. Possible modes are: “normal”, “seg_inst”, “seg_class”, “depth”, “flow”, “albedo”, “illumination”, “surf_normals”

• image_width (int) – width of the returned images

• image_height (int) – height of the returned iamges

Returns

pair success (bool), images: (list) a list of images according to the camera rendering mode

character_cameras()

Returns the number of cameras in the scene

Returns

pair success (bool), camera_names: (list): the names of the cameras defined fo the characters

environment_graph()

Returns environment graph, at the current state

Returns

pair success (bool), graph: (dictionary)

expand_scene(new_graph, randomize=False, random_seed=- 1, animate_character=False, ignore_placing_obstacles=False, prefabs_map=None, transfer_transform=True)

Expands scene with the given graph. Given a starting scene without characters, it updates the scene according to new_graph, which contains a modified description of the scene. Can be used to add, move, or remove objects or change their state or size.

Parameters

• new_graph (dict) – a dictionary corresponding to the new graph of the form {‘nodes’: …, ‘edges’: …}

• randomize (int bool) – a boolean indicating if the new positioni/types of objects should be random

• random_seed (int) – seed to use for randomize. random_seed < 0 means that seed is not set

• animate_character (bool) – boolean indicating if the added character should be frozen or not.

• ignore_placing_obstacles (bool) – when adding new objects, if the transform is not specified, whether to consider if it collides with existing objects

• prefabs_map (dict) – dictionary to specify which Unity game objects should be used when creating new objects

• transfer_transform (bool) – boolean indicating if we should set the exact position of new added objects or not

Returns

pair success (bool), message: (str)

fast_reset(environment=None)

Fast scene. Deletes characters and scene changes

Returns

success (bool)

get_visible_objects(camera_index)

Obtain visible objects according to a given camera

Parameters

camera_index (int) – the camera for which you want to check the objects. Between 0 and camera_count-1

Returns

pair success (bool), msg: the object indices visible according to the camera

instance_colors()

Return a mapping from rgb colors, shown on seg_inst to object id, specified in the environment graph.

Returns

pair success (bool), mapping: (dictionary)

move_character(char_index, pos)

Move the character char_index to a new position

Parameters

• char_index (int) – the index of the character you want to move

• pos (list) – the position where you want to place the character

Returns

succes (bool)

procedural_generation(seed=None)

Generates new environments through procedural generation logic.

Parameters

seed (int) – integer corresponding to the seed given during generation

Returns

success (bool), seed: (integer)

remove_terrain()

remove_terrain. Deletes terrain

Returns

success (bool)

render_script(script, randomize_execution=False, random_seed=- 1, processing_time_limit=10, skip_execution=False, find_solution=False, output_folder='Output/', file_name_prefix='script', frame_rate=5, image_synthesis=['normal'], save_pose_data=False, image_width=640, image_height=480, recording=False, save_scene_states=False, camera_mode=['AUTO'], time_scale=1.0, skip_animation=False)

Executes a script in the simulator. The script can be single or multi agent, and can be used to generate a video, or just to change the state of the environment

Parameters

• script (list) – a list of script lines, of the form [‘<char{id}> [{Action}] <{object_name}> ({object_id})’]

• randomize_execution (bool) – randomly choose elements

• random_seed (int) – random seed to use when randomizing execution, -1 means that the seed is not set

• find_solution (bool) – find solution (True) or use graph ids to determine object instances (False)

• processing_time_limit (int) – time limit for finding a solution in seconds

• skip_execution (int) – skip rendering, only check if a solution exists

• output_folder (str) – folder to output renderings

• file_name_prefix (str) – prefix of created files

• frame_rate (int) – frame rate at which to generate the video

• image_synthesis (list) – what information to save. Can be multiple at the same time. Modes are: “normal”, “seg_inst”, “seg_class”, “depth”, “flow”, “albedo”, “illumination”, “surf_normals”. Leave empty if you don’t want to generate anythign

• save_pose_data (bool) – save pose data, a skeleton for every agent and frame

• image_width (int) – image_height for the generated frames

• image_height (int) – image_height for the generated frames

• recording (bool) – whether to record data with cameras

• save_scene_states (bool) – save scene states (this will be unused soon)

• camera_mode (list) – list with cameras used to render data. Can be a str(i) with i being a scene camera index or one of the cameras from character_cameras

• time_scale (int) – accelerate time at which actions happen

• skip_animation (bool) – whether agent should teleport/do actions without animation (True), or perform the animations (False)

Returns

pair success (bool), message: (str)

reset(environment=None)

Reset scene. Deletes characters and scene changes, and loads the scene in scene_index

Parameters

environment (int) – integer between 0 and 49, corresponding to the apartment we want to load

Returns

succes (bool)

set_time(hours=0, minutes=0, seconds=0)

Set the time in the environment

Parameters

• hours (int) – hours in 24-hour time

• minutes (int) – minutes in 24-hour time

• seconds (int) – seconds in 24-hour time

• scaler (int) – scaler is a multipler that increase/decreases time step

Returns

success (bool)

update_camera(camera_index, position=[0, 1, 0], rotation=[0, 0, 0], field_view=40)

Updates an existing camera, identified by index. :param int camera_index: the index of the camera you want to update :param list position: the position of the camera, with respect to the agent :param list rotation: the rotation of the camera, with respect to the agent :param list field_view: the field of view of the camera

Returns

succes (bool)

update_character_camera(position=[0, 1, 0], rotation=[0, 0, 0], name='PERSON_FRONT')

Update character camera specified by name. This must be called before adding any character.

Parameters

• position (list) – the position of the camera, with respect to the agent

• rotation (list) – the rotation of the camera, with respect to the agent

Name

the name of the camera, used for recording when calling render script

Returns

succes (bool)