WP7: Integrative AI for Embodied Systems

Task Lead: UNITN, PI: Luigi Palopoli, Co-PI: Fabio Remondino

We will focus on the foundations of an embodied AI i.e., AI systems working in the physical world. We will create systems combining adaptability with safety, flexibility with reliability. Our solutions will rely on a synergistic combination of model-based techniques with learning from data and humans. 

Connected to TP Adjustable Autonomy and Physical Embodied Intelligence (TP4). 

Effort 36p/m + cascading funds.

Task 2.7.1 Methodologies for integrative AI in the physical world Task leader: Fabio Remondino

We will develop a layered cognitive architecture based on three layers: perception, deliberation and action.

Our idea of integrative AI permeates each of the different layers and defines the way in which the layers are organized and interact. In each layer, we will integrate symbolic and sub-symbolic techniques in order to merge the prior knowledge of physical and semantic nature into the learning process. Furthermore, we will pursue a strong interaction between the layers in order to refine the understanding of the environment and maximize the adequacy of the robot responses at the same time.

At the perception level, we will produce reliable 3D semantic models of the surveyed environment and of the actions that each actor (first and foremost, the humans) performs. To this end, once the data are collected with passive or active sensors, we will exploit semantic priors extracted from existing ontologies, and physical models constraining the position and the dynamic evolution of the environment. We will use this information in different ways. The first possibility is to wire it into the structure of the deep neural networks used to process the sensor data. The second possibility that we will explore is to use it as an external reasoning layer (e.g., based on probabilistic graphical models).

At the deliberation and action layers, we will overcome the distinction between task deliberation, motion planning and motion control. This is needed since the process of understanding semantic information on the environment and human actions is based on a continuous refinement. As new information is discovered and combined with the existing knowledge base, the understanding of the context improves enabling optimized robotic actions. In this framework, motion planning and control will be designed in order to be adapted and “remain” open to multiple final directions as soon as they materialize. This holistic and integrative approach to planning and control will require the synergistic combination of standard reasoning (e.g., to model the different scenarios using semantic information), model based planning and control, and machine learning (e.g., to estimate on the fly the parameters governing the human action and the dynamic changes in the environment).

Task 2.7.2 Integrative AI for robotic systems in human-populated environments Task leader: Luigi Palopoli

Human populated environments will be a perfect arena for the application of the framework developed in Task 2.7.1. We will take inspiration from an assistive application, i.e., using a robotic walker or wheelchair to help an impaired person (or an older person with cognitive deficit) to navigate across a large public space (e.g., a shopping mall, or the historic center of a town). We will use our architecture to create dynamic 3D maps of the surroundings, to interpret the actions and the intent of the assisted person and of all the by-standers, and to decide a course of actions that fulfills the goals of the assisted person. Our behavioral models will allow the robot to adapt to the type of the assisted person, leaving her in charge of the guiding actions when they prove reliable and safe, and taking over control (at least in part) when the commands coming from the person are erratic or dangerous. 

Task 2.7.3 Integrative AI for robotic systems in complex natural environments  Task leader: Luigi Palopoli

The task aims to validate and test the developed framework of integrative AI data processing (T2.7.1) into an outdoor environment - such as a forest - to boost a human-centric solution comprising intelligent collaborative robots. Unmanned aerial and ground vehicles (UAV/UGV), acting as multi-agents in the field, will be in contact with human operators. Through integrative AI, the multi-agents will collect and interpret data, empowering humans in the field with knowledge and preserving security and health. The UAV platform will survey the area with its on-board perception system which will enable, through on-board SLAM methods, to autonomously navigate and understand the scene. At the same time, the UGV will map the area from ground, exchanging information with the UAV. Both agents will leverage satellite-based GNSS signals (GPS, Galileo - enhanced by OSNMA/GBAS/SBAS, EGNOS/EDAS) or low power area network wireless technologies (LPWAN, e.g. LoRa, Sigfox), for positioning and geo-referencing purposes. The fused geo-located semantic information will be shared/transferred to the human in the field using an AR-based head-mounted display (HMD). Therefore an interactive user interface (UI) client manager will be created (e.g. with Unity3D to ensure openness, integration, interaction and compatibility) to issue tailored messages to human agents through HMD for high-level control, augmented perception, deliberation and action.