Actions

Structure of the Programs

Agents interact in VirtualHome via programs, lists of instructions that tell which actions each agent should do, and with which objects.

These commands are called using unity_simulator.UnityCommunication.render_script(), where a script is a list of strings of the form:

<char{id1}> {action1}

with {action1} being of the form

[action_name] # action with no args
[action_name] arg1 # action with a single arg
[action_name] arg1 arg2 # action with 2 args

And args correspond to objects, of the form

<object_name> (object_id)

An example of an action would be

<char0> [put] <glass> (3) <table> (2)

Multiple agents can do actions at the same time. You specify it as follows

<char0> [put] <glass> (3) <table> (2) | <char1> [walk] <kitchen> (5)

Actions available

Below is the list of actions available, notice that after an action is executed, the graph is updated, according to the rules in Environments.

Walk

Walks to a room or object

Arguments

1

Modifier

Preconditions

character is not sitting
obj1 is reachable (not inside some closed container) or obj1 is a room.
obj1 is not grabbed

Postconditions

character will move at a distance close to the object, making sure it is visible.
if obj1 is a room, character will move to the center of the room

Example

[walk] <kitchen> (1)

Run

Runs to a room or object

Arguments

1

Modifier

Preconditions

character is not sitting
obj1 is reachable (not inside some closed container) or obj1 is a room.
obj1 is not grabbed

Postconditions

character will move at a distance close to the object, making sure it is visible.
if obj1 is a room, character will move to the center of the room

Example

[run] <kitchen> (1)

Walktowards

Walks some distance towards a room or object

Arguments

1

Modifier

Preconditions

character is not sitting
obj1 is reachable (not inside some closed container) or obj1 is a room.
obj1 is not grabbed

Postconditions

character will move at a distance close to the object, making sure it is visible.
if obj1 is a room, character will move to the center of the room

Example

[walktowards] <kitchen> (1)
[walktowards] <table> (1) :3:

Walkforward

Walks 1 meter forward.

Arguments

0

Modifier

Preconditions

character is not sitting

Postconditions

character will 1 meter forward according to current orientation.

Example

[walkforward]

TurnLeft

Turns 30 degrees left

Arguments

0

Modifier

Preconditions

character is not sitting

Postconditions

character is turned 30 degree counterclockwise

Example

[turnleft]

TurnRight

Turns 30 degrees right

Arguments

0

Modifier

Preconditions

character is not sitting

Postconditions

character is turned 30 degree clockwise

Example

[turnleft]

Sit

Sit on an object

Arguments

1

Modifier

Preconditions

character is not sitting
character is close to obj1
obj1 has property sittable
number of objects on obj1 must be less than: ‘couch’: 4, ‘bed’: 4, ‘chair’: 1, ‘loveseat’: 2, ‘sofa’: 4, ‘toilet’: 1, ‘pianobench’: 2, ‘bench’: 2

Postconditions

Add directed edges: character sitting obj1

Example

[sit] <chair> (1)

StandUp

Stand Up

Arguments

0

Modifier

Preconditions

character state is sitting

Postconditions

character remove state sitting

Example

[standup]

Grab

Grab an object

Arguments

1

Modifier

Preconditions

obj1 property is grabbable except water
exists edge character close obj1
character is close to obj1
obj1 is reachable (not inside some closed container)
No edge character holds_rh any object or no edge character holds_lh any_object. Character has at least one free hand.

Postconditions

Add directed edges: character holds_rh or hold_lh, obj1
obj1 is not anymore on a surface or inside a container

Example

[grab] <apple> (1)

Open

Open an object

Arguments

1

Modifier

Preconditions

obj1 property is opennable and state is closed.
character is close to obj1
obj1 is reachable (not inside some closed container)
No edge character holds_rh any object or no edge character holds_lh any_object. Character has at least one free hand.

Postconditions

obj1 state is open

Example

[open] <fridge> (1)

Close

Close an object

Arguments

1

Modifier

Preconditions

obj1 property is opennable and state is open.
character is close to obj1
obj1 is reachable (not inside some closed container)
No edge character holds_rh any object or no edge character holds_lh any_object. Character has at least one free hand.

Postconditions

obj1 state is closed.

Example

[close] <fridge> (1)

Put

Put an object on some other object

Arguments

2

Modifier

Preconditions

exists edge character holds_lh obj1 or character holds_rh obj1
exists edge character close obj2

Postconditions

remove directed edges: character holds_lh obj1 or character holds_rh obj1
add directed edges: obj1 on obj2

Example

[putback] <apple> (1) <table> (1)

PutIn

Put an object inside some other object

Arguments

2

Modifier

Preconditions

exists edge character holds_lh obj1 or character holds_rh obj1
exists edge character close obj2
obj2 is not closed

Postconditions

remove directed edges: character holds_lh obj1 or character holds_rh obj1
add directed edges: obj1 inside obj2

Example

[putin] <apple> (1) <fridge> (1)

SwitchOn

Turn an object on

Arguments

1

Modifier

Preconditions

obj1 has property switch
obj1 state is off
exists edge character close obj1

Postconditions

obj1 state is on

Example

[switchon] <stove> (1)

SwitchOff

Turn an object off

Arguments

1

Modifier

Preconditions

obj1 has property switch
obj1 state is on
exists edge character close obj1

Postconditions

obj1 state is off

Example

[switchoff] <stove> (1)

Drink

Drink from an object

Arguments

1

Modifier

Preconditions

obj1 property is drinkable or recipient
exists edge character close obj1

Postconditions

Example

[drink] <waterglass> (1)

Touch

Touch an object

Arguments

1

Modifier

Preconditions

character is close to obj1
obj1 is reachable (not inside some closed container)

Postconditions

Example

[touch] <stove> (1)

LookAt

Look at an object

Arguments

1

Modifier

Preconditions

character is facing obj1
obj1 is reachable (not inside some closed container)

Postconditions

Example

[lookat] <stove> (1)