Carrying an object together

In this tutorial we will go into how two agents can carry an object together, using a custom action.

Take a sneak peek at the endresult below, or immediately jump to the code on Github.

The Goal

The endresult and our goal will look as below. The human (with blue cap) initiates carrying a victim together with the robot (with blue eye). During the carrying, the image changes to represent them carrying the victim.

Human and robot carrying an object (or victim in this case) together.

For this we will need a number of things:

A MATRX world (also called a case).
A human agent, this one will be initiating the picking up.
An artificial agent, this one will be cooperating in carrying the object together with the human.
A new CarryingTogether and DroppingTogether MATRX action.
A number of images for the agents and object we are going to carry.

For each of these we create a seperate folder and file, so we end up with this:

Secondly, make sure your MATRX is up to date:

pip install --upgrade matrx

Now that we are ready, let’s get started.

MATRX world

The world we are going to create is super simple, and contains only three objects: 1) an object we can carry, in this case a orange-coloured human victim 2) a custom human agent we can control and 3) a custom artificial agent that will help us carry it.

As such, we edit the my_experiment/case/carry_togethter_case.py to add an object, an custom human agent, and a custom artificial agent. The result can be seen here.
We don’t have a custom action or human or artificial agent yet so those don’t work, but we’ll make those later.

Note also that at the top of the file we import our custom agents and actions, and link the actions to a key map (press key X, do action Y), that is assigned to the human agent:

# custom code
from my_experiment.agents.agent import CustomAgent
from my_experiment.agents.human_agent import CustomHumanAgent
from my_experiment.actions.custom_actions import CarryObjectTogether, DropObjectTogether

... 

# Controls for the human agent
key_action_map = {  
    'w': MoveNorth.__name__,
    'd': MoveEast.__name__,
    's': MoveSouth.__name__,
    'a': MoveWest.__name__,
    'q': CarryObjectTogether.__name__,
    'e': DropObjectTogether.__name__
}

Now if we run the code, MATRX will complain about our actions, so we will create these next.

The CarryingTogether Actions

By default MATRX supports an action for grabbing and dropping objects. But there is no action for having two agents carry the same object together, as well as that it is technically not possible. In practice though, we can make it seem to the user as if that is happening. To achieve this, our CarryingTogether actions will work as follows:

The Human Agent walks up to the object to be carried and the Artificial Agent.
The agent then executes the CarryTogether action by pressing q (as specified in the key_action_map).
If the object and other agent are close enough, i.e. within grab_range (a value you can specify yourself), and the object is_movable, and some other checks succeed, the action is executed. This action changes the image from the human to an image that looks like the human + agent carrying the victim together (see here), and makes the artificial agent invisible. It also moves the object to the inventory of the human like a normal GrabAction.
The human then carries the victim somewhere, and can then drop the victim by pressing e (specified in key_action_map), initiating the DropObjectTogether action. This action does the reverse of the previous action, by making the agent visible again, teleporting it to the location of the human agent, changing the image of the human back to normal, and moving the victim back into the world.

Step 3 and 4 are programmed as actions in the custom_actions.py file, where for both actions an is_possible function is specified that checks whether the action is possible before executing, and a mutate function that actually executes the action if so.

Also note that these actions extend the normal GrabObject and DropObject functions, and are basically the same with some extra fluff for making it seem as if the object is being carried by two agents.

The Custom Agents

Finally, we create our custom agents. First we create a custom artificial agent that does absolutely nothing, see here. It doesn’t filter any observations, and never decides on an action. This works for us as the human agent is the only one making decisions, but normally you would want the agent also to have some intelligence, see the next section for tips on that part.

The Human Agent is a bit more complex, as it can receive input from the user, check if the key pressed by the user links to a valid action, and finally find an object to pickup if the user pressed q. This all happens in the decide_on_action function, and can be found in the CustomHumanAgent here.

Note the function that is used to find the Victim object:

 obj = state.get_closest_with_property(props={"name": "Victim"})

The state object has a lot of very useful helper functions to find objects in the world, such as get_agents(), get_closest_agents(), get_closest_room_door(), get_team_members(), etc. A list of examples can be found here.

The Result

Now that all code is in place, we can run the main.py and try out our new human-agent team:

Run: python main.py
View in your browser: 127.0.0.1:3000
- Go to the god view and press play
- Switch to human agent view and control the agent with the controls specified in the key action map

The code for this tutorial can be found on Github: matrx-tutorials/carry-object-together at master · matrx-software/matrx-tutorials (github.com)

Next Steps

There are numerous improvements you could make, such as adding intelligence to the artificial agent such that it walks to the human when requested. This is possible if you have an agent that can do path planning, such as the PatrollingAgent from this example case. Sending messages is also possible in MATRX, either programmatically or via the chat. An example of sending messages from an agent brain can be found here, and an example of how to read those messages can be found here.

Another improvement that can be made is to solve the issue of the the artificial agent being visibily teleported to the location of the victim when dropping the victim. This might be solved by having the agent change its position while invisible to follow the human agent. In this way, it will be teleported at most 1 cell.

How to