New paths to interoperability of robots: Large AI models as an opportunity for collaborative action
Four Levels of Interoperability
Interoperability provides the foundation for communication and cooperation but does not itself bring about joint action. So far, it is primarily achieved through standards for interfaces, protocols, or data formats. Such solutions are complex and often not very flexible. It is particularly challenging to establish a common understanding of information and its significance for actions.
The position paper distinguishes four levels of interoperability: The technical level connects systems with one another, the syntactic level ensures compatible data formats. The semantic level pertains to the meaning of exchanged information, while the pragmatic level involves the understanding of the consequences of actions in a specific context. It is only at this final level that the actions of systems can best be coordinated with each other.
What large AI models can achieve
“Large AI models can help achieve interoperability between different robotics systems in an elegant and efficient manner, on a semantic or even pragmatic level,” says Jürgen Beyerer, managing director as well as director of the Fraunhofer Institute for Optronics, System Technologies, and Image Exploitation (IOSB) in Karlsruhe and holder of the Chair for Interactive Real-Time Systems at the Karlsruhe Institute of Technology (KIT).
In addition to large language models, those designed for logical reasoning or models that integrate language, vision, and action are also suitable for this purpose. They can help categorize information from different systems and make it usable for coordination. Crucial are the world knowledge represented in the model, which is linked to the type and extent of the data in the model training, as well as additional context information made available to the model during operation.
Collaboration Requires Reliable World Knowledge
"Robots can work through complex storylines or missions in a network if they are able to reliably physically interact, exchange, and interpret information. The core of the challenge often lies in the autonomy and the world model of the systems. Today's large generative AI models can fill gaps here and coordinate robots, and future models could even establish new forms of interoperability," explains Sirko Straube, Research Area Manager Emergency Response and Recovery Management at the Robotics Innovation Center of DFKI GmbH.
However, current models have limitations. They often lack activity data as well as domain-specific data. They sometimes generate incorrect statements and have a limited understanding of cause and effect. Additionally, they still do not reliably translate objects, positions, or actions into precise specifications for physical models. This can cause interoperability to "break down" temporarily, posing risks to functional and process safety. Promising future developments are more reliable multimodal AI models that can check facts in real time and better estimate the consequences of actions using physical simulations and world models. However, for practical use, these capabilities must be reliably available.
"Interoperability among robots is a crucial building block for the future that will sustainably advance the practical use and interaction of diverse robotics systems. Research is already developing promising projects and concepts - now it is important to jointly translate these into marketable and application-oriented solutions. This requires a strong alliance between users, robotics manufacturers, and researchers to elevate robotics to the next level of development," summarizes Johannes Hinckeldeyn, Director of Advanced Corte Technologies at the Kion Group.
Translated automatically from German.






























