Environments

VirtualHome is composed of 7 scenes where activities can be executed. Each scene is encoded as a dictionary, containing a node for every object and edges between them representing relationships. Each environment can be updated by modifying the corresponding dictionary. We call this dictionary EnvironmentGraph.

You can load each environment using unity_simulator.UnityCommunication.reset() function, specifying the matching environment id. You can retrieve the information of an environment using unity_simulator.UnityCommunication.environment_graph(). After loading an environment, you can modify it using unity_simulator.UnityCommunication.expand_scene().

Scenes

Below is a list of the 7 scenes included in VirtualHome.

Scene Id	Apartment View
0
1
2
3
4
5
6

EnvironmentGraph

The EnvironmentGraph is the way to represent a scene in VirtualHome. It is represented as a dictionary of node, corresponding to objects, and edges, corresponding to relationships.

Here is an example:

{
   "nodes":[
      {
         "id":1,
         "class_name":"character",
         "states":[

         ],
         "properties":[

         ],
         "category":"Person"
      },
      {
         "id":2,
         "class_name":"kitchen",
         "states":[

         ],
         "properties":[

         ],
         "category":"Room"
      }
   ],
   "edges":[
      {
         "from_id":1,
         "to_id":2,
         "relation_type":"INSIDE"
      }
   ]
}

The structure of the Graph is documented below.

EnvironmentGraph

EnvironmentGraph

Object Properties

nodes (list(Nodes)) – list of objects
edges (list(Edges)) – list of relationships between objects

Nodes

Object Properties

id (int) – id of object
class_name (str) – class_name of the object
category (str) – cateogry of the object
prefab_name (str) – name of the Unity game object used
states (list(str)) – states of the objects
properties (list(str)) – properties of the objects
bounding_box (list(BoundingBox)) – bounding box of the given object
obj_transform (ObjTransform) – 3D information of the given object

BoundingBox

Object Properties

center (list(float)) – center of the bounding box, in x,y,z
size (list(float)) – size of the bounding box, in x,y,z. The size is axis aligned

ObjTransform

Object Properties

position (list(float)) – x,y,z position of the game object. Does not necessarily match with BoundingBox center.
rotation (list(float)) – x,y,z,w representation a quaternion of the object rotation https://docs.unity3d.com/ScriptReference/Quaternion.html

Edges

Object Properties

from_id (int) – id of Node in the from relationship
to_id (int) – id of Node in the to relationship
relation_type (str) – relationship between the 2 objects.

States

Objects can have the following states

State	Meaning
OPEN	Indicates whether a container is open.
CLOSED	Indicates whether a container is closed.
ON	Indicates whether a light, electronic device is ON.
OFF	Indicates whether a light, electronic device is OFF.

Properties

Objects can have the following properties. Some have effect on the Uniy Simulator and some are just used to categorize the objects. We only indicate meaning for the former.

Property	Meaning
CAN_OPEN	Whether the object can be opened
CLOTHES
CONTAINERS
COVER_OBJECT
CREAM
CUTTABLE
DRINKABLE
EATABLE
GRABBABLE	Whether the object can be grabbed
HANGABLE
HAS_PAPER
HAS_PLUG
HAS_SWITCH	Whether the object can be turned on or off
LIEABLE
LOOKABLE
MOVABLE
POURABLE
READABLE
RECIPIENT
SITTABLE	Whether the agent can sit in this object
SURFACE	Whether the agent can place things on this object

Relationships

Objects can have different relationships between each other, we describe here their meaning

Relation Type	Meaning
ON	Object from_id is on top of object to_id.
INSIDE	Object from_id is inside of object to_id.
BETWEEN	Used for doors. Door connects with room to_id.
CLOSE	Object from_id is close to object to_id (< 1.5 metres).
FACING	Object to_id is visible from objects from_id and distance is < 5 metres. If object1 is a sofa or a chair it should also be turned towards object2.
HOLDS_RH	Character from_id holds object to_id with the right hand.
HOLDS_LH	Character from_id holds object to_id with the left hand.
SITTING	Character from_id is sitting in object to_id.