General-purpose robot systems are needed to solve real-world challenges by combining data-based machine learning with physical, kinematic, dynamic, and interaction models of human-in-the-loop intelligent systems. There has been substantial progress in cross-modal learning deep neural networks and Large Multimodal Models (LMMs) in terms of data-driven benchmarking. However, acquiring large multimodal data for robots in the real world is challenging, and such data-driven systems are computationally costly and not yet interpretable. At the same time, most model-based approaches lack robustness in unstructured, dynamic, and changing environments. My talk will first introduce concepts based on findings in cognitive systems that allow a robot to better understand multimodal scenarios by integrating knowledge and learning. I will then outline the necessary modules to enhance the robot's intelligence level. Following that, I will explain how large multimodal learning methods can be realized in intelligent robots. Finally, I will demonstrate several novel robot systems with skills in dexterous manipulation, robust dynamic walking, and natural human-robot interaction, showcasing their potential for service applications.

Jianwei Zhang is a professor and the Director of Technical Aspects of Multimodal Systems at the Department of Informatics, University of Hamburg, Germany. He is an Academician of the German National Academy of Engineering Sciences, the Academy of Sciences and Humanities in Hamburg, Germany, and a Foreign Member of the Chinese Academy of Engineering. He is also a Distinguished Visiting Professor at Tsinghua University. He received both his Bachelor of Engineering (1986, Computer Control, with distinction) and Master of Engineering (1989, AI) from the Department of Computer Science at Tsinghua University, Beijing, China, and his PhD (1994, Robotics) from the Institute of Real-Time Computer Systems and Robotics, Department of Computer Science, University of Karlsruhe, Germany.

3D point clouds (PCs) become increasingly available for diversified scenarios, thanks to economical creation of a digital twin for almost everything, enabled by substantial advancement of depth sensing, photogrammetry, and deep learning. This offers a practical bridge between our physical world and its expanding virtual counterpart, promote integration of computer vision and computer graphics that have been two separate realms for long, and facilitate mixed reality and multimedia interaction. Therefore, unprecedented possibilities are expected in digital transformation and smart cities, toward robot navigation, autonomous driving, gaming/entertainment, social media, industrial metaverse, BIM, urban surveillance/planning, digital art, cultural heritage preservation, future training/education, crime investigation, and discovery in medical/biological/material sciences. PCs can be also used to generate alternative representation of 3D visual content, like 3D meshes and emerging Gaussian splatting. This talk will present the recent research and development for recreation, representation, processing and evaluation of PCs, with humans and machines as ultimate users, respectively. Related important topics include various filtering, simplification, compression, registration, shape/mesh construction, saliency/quality evaluation, and image-based localization. Possible future research directions will be highlighted and discussed as well.

Weisi Lin is an active researcher in image processing, perception-based signal modelling and assessment, video compression, and multimedia communication. He had been the Lab Head, Visual Processing, Institute for Infocomm Research (I2R), Singapore. He is currently a President’s Chair Professor in College of Computing and Data Science, Nanyang Technological University (NTU), Singapore, where he also serves as the Associate Dean (Research). He is a Fellow of IEEE and IET.