Whether it’s an apartment key, reading glasses or a wallet: in everyday life it often happens that we misplace objects and then have to look for them. In the future, a robot could also do this. Using two different AI systems, researchers have given it the ability to recognize objects and find them in changing environments. The search robot can search for specific objects on demand, asks what they look like using a large language model and understands where these objects are most likely to be found. This means that the system tracks down lost objects much more efficiently and quickly than a robot that searches its surroundings more randomly.
When robots move through a factory, home, or campus, they must first create a mental map of their surroundings. This then helps them to find their way and navigate efficiently through these environments. However, most rooms do not always stay the same: chairs are moved, dishes are sometimes on the table, sometimes in the kitchen and objects can also change their position outdoors. Robots therefore have to continually update their mental map of the environment. Until now, this has happened relatively randomly, with the systems simply scanning the entire environment at intervals and making before-and-after comparisons. But if robots are to search for objects in this changing environment, additional skills are required: They must be able to recognize the required object and develop a search strategy that leads to the goal as quickly as possible.
Mental map and semantic object recognition
Benjamin Bogenberger and his colleagues from the Technical University of Munich have now developed such a robot. Your robot looks like a broomstick on wheels with a camera mounted on the top end. But behind this reduced appearance lies concentrated technology in the form of artificial intelligence. It enables the robot to track down specific objects on demand and to search specifically in the most likely locations. “Our approach includes two operation modes that make this possible: The first is an active map update,” the team reports. To do this, the robot actively scans the areas of its environment that are most likely to change, such as the surfaces of tables or worktops. However, he checks areas that are more static less frequently. The robot camera delivers two-dimensional images, whose pixels also contain depth information. This creates a centimeter-precise spatial image of the environment that is constantly updated.
The second component is outsourced to a laptop and includes the connection to an AI system based on a large language model. This allows the robot to understand voice input and can link the objects it is looking for with concrete images. In addition, this AI system provides him with information about how and in what context these objects are used and therefore where they are most likely to be found. The system learns that glasses are more likely to be found on the table or windowsill than on the stovetop or in the sink. “The language model records the relationships between the objects and we convert this information into the robot’s language,” explains senior author Angela Schoellig from the Technical University of Munich.
(Video: TU Munich)
Where could the glasses most likely be located?
When the robot starts looking for the glasses, for example, its mental map shows it directly where the chances of finding it are greatest. Two-digit numbers appear everywhere on the map, which quantify the probability of this location. “We taught the robot to understand the environment,” says Schoellig. In a practical test, for example, the robot was given the task of looking for a plate. His surroundings were a room with chairs, a coffee table with a cup on it, and a desk. “The exploration priority map correctly identified the area near the cup on the table as the likely location of the plate,” the researchers report. In further tests, after an initial exploration of the room, the robot was supposed to find a newly placed book on a chair or on a shelf, a keyboard on the desk or a bowl on the table. This showed that compared to conventional search strategies, the robot found the objects 14 percent faster and searched almost 30 percent more efficiently.
According to Bogenberger and his colleagues, their approach opens up the possibility of making robots in factories, households or other environments more efficient and intelligent. The newly developed basic understanding of spaces and objects is “important for all robots that move in spaces that are constantly changing,” says Schoellig. In the next step, the researchers want to further develop their search robot so that it can also find hidden objects – for example in a drawer or behind a door. To do this, however, they first have to give him arms and hands so that he can then open doors or drawers.
Source: Benjamin Bogenberger (Technical University of Munich) et al., IEEE Robotics and Automation Letters, doi: 10.1109/LRA.2026.3656790