This paper presents a UAV-UGV team designed for autonomous firefighting, developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The system uses LiDAR for localization in GNSS-restricted environments and fuses LiDAR and thermal camera data to track fires. Relative navigation enables successful fire extinguishing. Why it matters: This research demonstrates the potential of robotic systems in autonomous firefighting, addressing challenges in dangerous and inaccessible environments, and advancing robotics research within the UAE.
Team NimbRo presented their UGV solution for autonomous wall building and firefighting at the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The robot integrates a wheeled omnidirectional base, a 6 DoF manipulator arm with a magnetic gripper, a storage system, and a water spraying system. It uses 3D LiDAR, RGB, and thermal cameras to perceive the environment, pick up boxes, construct walls, and detect/extinguish fires. Why it matters: The work highlights advancements in autonomous robotics for complex tasks relevant to construction and disaster response in the UAE and globally.
This paper presents the synthesis of a 1-DoF six-bar gripper mechanism for aerial grasping, designed for a task in the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The synthesis process involves selecting the mechanism class, determining the number of links and joints using algebraic methods, and optimizing link dimensions via geometric programming. The gripper was modeled in CAD software, additively manufactured, and mounted on a UAV with a DC motor for gripping spherical objects. Why it matters: The research contributes to advancements in robotics and aerial manipulation, with potential applications in various industries, particularly for tasks requiring remote object retrieval and manipulation.
This paper presents two robotic systems developed for the MBZIRC 2020 competition, designed for autonomous wall construction. The systems utilize a UGV with 3D LiDAR for precise brick pose estimation and a UAV employing real-time visual servoing. The authors report results from the competition and lab experiments, discussing lessons learned from the autonomous wall-building task. Why it matters: The work highlights advancements in mobile manipulation and autonomous robotics, with potential applications in construction and infrastructure development in the region.
This paper presents a fully autonomous micro aerial vehicle (MAV) developed to pop balloons using onboard sensing and computing. The system was evaluated at the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The MAV successfully popped all five balloons in under two minutes in each of the three competition runs. Why it matters: This demonstrates the potential of autonomous robotics and computer vision for real-world applications in challenging environments.
A robotics team from KAUST's Robotics, Intelligent Systems, and Control (RISC) lab won the "Best Air Team" special award and the European Global Navigation Satellite Systems Agency special prize at the European Robotics League Emergency Robots Challenge in Sevilla, Spain. The KAUST team, led by Kuat Telegenov and advised by Professor Jeff Shamma, competed against international teams in aerial robotic challenges. The competition aimed to encourage advancements in autonomous capabilities and seamless outdoor/indoor navigation for robots. Why it matters: The awards recognize KAUST's contributions to robotics research and highlight the importance of developing autonomous systems for emergency response and complex environments.
The Robotics, Intelligent Systems, and Control (RISC) lab at KAUST is developing swarm robotics, enabling robots to work together on collaborative tasks with limited human supervision. RISC is using game theory to improve how robots make coordinated decisions in scenarios like engaging intruders or tracking oil spills. The lab is also researching programmable self-assembly for robot swarms. Why it matters: This research advances autonomous multi-agent systems for critical applications like search and rescue and environmental monitoring in the region.
Two KAUST Ph.D. students participated in Junction 2018, a 48-hour international hackathon in Helsinki. One student, Yan Gong, was part of a winning team that addressed the "Adaptive Smart Heating" challenge. Another student, Torsten Hädrich, and his team designed a VR-based solution for healthcare using motion tracking to analyze patient motor skills. Why it matters: This participation highlights KAUST's engagement in global technology innovation and the application of its research to real-world challenges in energy and healthcare.
KAUST researchers in the Image and Video Understanding Lab are applying machine learning to computer vision for automated navigation, including self-driving cars and UAVs. They tested their algorithms on KAUST roads, aiming to replicate the brain's efficiency in tasks like activity and object recognition. The team is also exploring the possibility of creative algorithms that can transfer skills without direct training. Why it matters: This research contributes to the advancement of autonomous systems and explores the fundamental questions of replicating human intelligence in machines within the GCC region.
The article discusses Team NimbRo's approaches to challenges involving micro aerial vehicles (MAV) at the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017. The challenges included landing on a moving vehicle and a treasure hunt task requiring mission planning and multi-robot coordination. The team's system achieved a third place in both subchallenges and contributed to winning the MBZIRC Grand Challenge. Why it matters: This demonstrates advanced robotics capabilities developed and tested in the UAE, pushing the boundaries of autonomous aerial vehicle operation and multi-robot collaboration.
Team NimbRo's robot Mario won the MBZIRC 2017 Challenge 2 by autonomously manipulating a valve stem using a wrench. The robot uses an omnidirectional base for locomotion and a 3D laser scan detector to find the manipulation panel. A deep neural network detects and selects the correct tool from grayscale images, and motion primitives are adapted to turn the valve stem. Why it matters: This work demonstrates advanced robotic manipulation capabilities relevant for industrial automation and hazardous environment operations in the region.
KAUST Ph.D. students David Evangelista and Xianjin Yang won best paper awards at international conferences this summer for their work in mean-field game theory. Evangelista's paper focused on solutions for stationary mean-field games with congestion, while Yang's paper developed numerical methods for homogenization problems. The awards were presented at the 18th International Symposium on Dynamic Games and Applications in France and the 12th American Institute of Mathematical Sciences (AIMS) Conference in Taiwan. Why it matters: The recognition highlights KAUST's strength in applied mathematics and computational science, specifically in the emerging field of mean-field games with applications across various domains.
This paper introduces a decentralized multi-agent decision-making framework for search and action problems under time constraints, treating time as a budgeted resource where actions have costs and rewards. The approach uses probabilistic reasoning to optimize decisions, maximizing reward within the given time. Evaluated in a simulated search, pick, and place scenario inspired by the Mohamed Bin Zayed International Robotics Challenge (MBZIRC), the algorithm outperformed benchmark strategies. Why it matters: The framework's validation in a Gazebo environment signals potential for real-world robotic applications, particularly in time-sensitive and cooperative tasks within the robotics domain in the UAE.
Researchers present RUR53, an unmanned ground vehicle (UGV) capable of autonomous navigation, object recognition, and tool manipulation. The UGV uses a modular software architecture, enabling it to perform complex tasks like detecting panels, docking, and manipulating tools such as wrenches and valve stems. RUR53 was tested at the 2017 Mohamed Bin Zayed International Robotics Challenge where it ranked third in the Grand Challenge as part of a collaboration. Why it matters: This research demonstrates advanced robotics capabilities applicable to various industrial and inspection tasks, highlighting the UAE's focus on robotics innovation.
This paper presents a decentralized multi-agent unmanned aerial system designed for search, pickup, and relocation of objects. The system integrates multi-agent aerial exploration, object detection/tracking, and aerial gripping. The decentralized system uses global state estimation, reactive collision avoidance, and sweep planning for exploration. Why it matters: The system's successful deployment in demonstrations and competitions like MBZIRC highlights the potential of integrated robotic solutions for complex tasks such as search and rescue in the region.
MBZUAI researchers have developed a new action tokenization method called LipVQ-VAE to improve in-context robot learning. LipVQ-VAE combines VQ-VAE with a Lipschitz constraint to generate smoother robotic motions, addressing limitations of traditional methods. The technique was tested on simulated and real robots, showing improved performance in imitation learning. Why it matters: This research advances robot learning by enabling more fluid and successful robot actions through improved action representation, drawing inspiration from NLP techniques.
MBZUAI researchers have developed a "divide-and-conquer" technique to improve learning from demonstration in robotics. The approach breaks down complex dynamical systems into independently solvable subsystems, modeled as linear parameter-varying systems. This method aims to simplify computations while maintaining stability and accurately capturing joint interactions for robots in complex environments. Why it matters: The research addresses a key challenge in robotics, potentially enabling more efficient and safer robot learning from human demonstrations.
MBZUAI graduate Maryam Mohamed Buty Alghfeli's master's research focused on using the metaverse to enhance intelligent transportation systems and vehicular networks. Her work proposed a framework for sensing, communication, and task offloading in the vehicular metaverse, addressing challenges related to computing and communication resource constraints. The research also considered self-sustainability and proactive learning approaches to improve network operation and serve autonomous vehicles. Why it matters: This research contributes to the development of more efficient and resilient autonomous vehicle networks, which are crucial for the future of smart cities and transportation in the UAE and beyond.
Gregory Chirikjian presented an overview talk on applying probability, harmonic analysis, and geometry to robotics, emphasizing the need for robots to function beyond traditional industrial programming. He discussed a new approach where robots define affordances of objects, using simulation to 'imagine' object use and enabling reasoning about novel objects. Probabilistic methods on Lie-groups, initially developed for mobile robot state estimation, are now adapted for one-shot learning of affordances, with plans to integrate large language models. Why it matters: This research direction aims to enhance robot intelligence and adaptability, crucial for service robots in dynamic environments and aligning with broader goals of advanced AI integration in robotics.
Yoshihiko Nakamura from the University of Tokyo discusses the computational challenges of humanoid robots, extending beyond sensing and control to understanding human movement, sensation, and relationships. The talk covers recent research on mechanical humanoid robots with a focus on actuators and computational problems related to human movements. Nakamura highlights the need for humanoid robots to interpret human actions and interactions for effective application. Why it matters: Addressing these computational challenges is crucial for developing more sophisticated and human-compatible robots for use in various human-centered applications within the region and globally.
Vaneet Aggarwal from Purdue University presented new research on discrete and continuous submodular bandits with full bandit feedback. The research introduces a framework transforming discrete offline approximation algorithms into sublinear α-regret methods using bandit feedback. Additionally, it introduces a unified approach for maximizing continuous DR-submodular functions, accommodating various settings and oracle access types. Why it matters: This research provides new methods for optimization under uncertainty, which is crucial for real-world AI applications in the region, such as resource allocation and automated decision-making.
Cyrill Stachniss from the University of Bonn presented recent work on agricultural robotics and self-driving cars. The talk covered autonomous field robots and their ability to perceive, model, and predict future developments in complex farming environments. The presentation also included developments in supervised and unsupervised learning for autonomous car perception systems. Why it matters: This highlights the growing interest in robotics research at MBZUAI and the potential for AI to transform key sectors in the GCC region like agriculture and transportation.
Saber Salehkaleybar from EPFL presented a talk on causal discovery, focusing on learning causal relationships from observational data and through interventions. He discussed an approximation algorithm for experiment design under budget constraints, with applications in gene-regulatory networks. The talk also covered improvements to reduce the computational complexity of experiment design algorithms. Why it matters: Causal AI systems can lead to more intelligent decision-making in various fields.
Jingjin Yu from Rutgers University presented research on multi-robot coordination and robotic manipulation at MBZUAI. The talk covered Rubik Table algorithms for collision-free path planning for multiple robots in dense settings. It also discussed algorithms for long-horizon manipulation tasks like rearrangement and object retrieval. Why it matters: Advancements in multi-robot coordination and manipulation are crucial for deploying robots in various sectors within the UAE and beyond, such as logistics and elder care.
A presentation discusses the evolution of Vision-and-Language Navigation (VLN) from benchmarks like Room-to-Room (R2R). It highlights the role of Large Language Models (LLMs) such as GPT-4 in enabling more natural human-machine interactions. The presentation showcases work using LLMs to decode navigational instructions and improve robotic navigation. Why it matters: This research demonstrates the potential of merging vision, language, and robotics for advanced AI applications in navigation and human-computer interaction.
Gregory Chirikjian presented an overview of research on robot navigation in unstructured environments, using computer vision, sensor tech, ML, and motion planning. The methods use multi-modal observations from RGB cameras, 3D LiDAR, and robot odometry for scene perception, along with deep RL for planning. These methods have been integrated with wheeled, home, and legged robots and tested in crowded indoor scenes, home environments, and dense outdoor terrains. Why it matters: This research pushes the boundaries of robotics in complex environments, paving the way for more versatile and autonomous robots in the Middle East.
Michael Yu Wang, Chair Professor and Founding Dean of the School of Engineering at Great Bay University, argues for combining "good old fashioned engineering" (GOFE) with learning-based approaches like LLMs for robot skill acquisition, particularly in manipulation. He suggests a modular framework that integrates engineering principles with learning, drawing inspiration from human hand-eye coordination and tactile perception. Wang emphasizes the need to address engineering features of robot tactile sensors, such as spatial and temporal resolutions, to achieve human-like robot manipulation skills. Why it matters: This perspective highlights the importance of hybrid approaches combining traditional engineering with modern AI for advancing robotics, especially in complex manipulation tasks relevant to industries in the GCC region.
Eliseo Ferrante from NYU Abu Dhabi presented work on increasing the controllability of swarm robotics systems. The research covers microscopic control via implicit intelligent leaders and macroscopic control via automated generation of swarm behaviors. Grammatical evolution and generative AI methods are used to produce collective behaviors aligned with human specifications. Why it matters: This research enhances the applicability of swarm robotics in real-world scenarios by improving control and coordination, potentially impacting industries like logistics, environmental monitoring, and disaster response in the region.
Tetsunari Inamura's talk explores using VR to collect HRI data and tailor assistive robotic functionalities to individual users. He discusses symbol emergence via multimodal interaction, interactive behavior generation through symbol manipulation, and VR for data collection. The talk emphasizes long-term human capability enhancement and avoiding over-reliance on technology. Why it matters: This research promotes independence and growth in human-robot interactions, potentially revolutionizing assistive technologies in the region.
Song Chaoyang from the Southern University of Science and Technology (SUSTech) presented research on Vision-Based Tactile Sensing (VBTS) for robot learning, combining soft robotic design with learning algorithms to achieve state-of-the-art performance in tactile perception. Their VBTS solution demonstrates robustness up to 1 million test cycles and enables multi-modal outputs from a single, vision-based input, facilitating applications such as amphibious tactile grasping and industrial welding. The talk also highlighted the DeepClaw system for capturing human demonstration actions, aiming for a universal interaction interface. Why it matters: This research advances embodied intelligence by improving robot dexterity and adaptability through enhanced tactile sensing, which is crucial for complex manipulation tasks in various sectors such as manufacturing and healthcare within the region.
Mykel Kochenderfer from Stanford University gave a talk on building robust decision-making systems for autonomous systems, highlighting the challenges of balancing safety and efficiency in uncertain environments. The talk addressed computational tractability and establishing trust in these systems. Kochenderfer outlined methodologies and research applications for building safer systems, drawing from his work on air traffic control, unmanned aircraft, and automated driving. Why it matters: The development of safe and reliable autonomous systems is crucial for various applications in the region, and insights from experts like Kochenderfer can guide research and development efforts at institutions like MBZUAI.
Dr. Hao Dong from Peking University presented research on addressing the challenge of limited large-scale training data in embodied AI, particularly for manipulation, task planning, and navigation. The presentation covered simulation learning and large models. Dr. Dong is a chief scientist of China's National Key Research and Development Program and an area chair/associate editor for NeurIPS, CVPR, AAAI, and ICRA. Why it matters: Overcoming data scarcity is crucial for advancing embodied AI research and enabling more sophisticated robotic applications in the region.
Krishna Murthy, a postdoc at MIT, researches computational world models to enable robots to understand and operate effectively in the physical world. His work focuses on differentiable computing approaches for spatial perception and interfaces large image, language, and audio models with 3D scenes. Murthy envisions structured world models working with scaling-based approaches to create versatile robot perception and planning algorithms. Why it matters: This research could significantly advance robotics by enabling more sophisticated perception, reasoning, and action capabilities in embodied agents.
Giuseppe Loianno from NYU presented research on creating "Super Autonomous" robots (USARC) that are Unmanned, Small, Agile, Resilient, and Collaborative. The research focuses on learning models, control, and navigation policies for single and collaborative robots operating in challenging environments. The talk highlighted the potential of these robots in logistics, reconnaissance, and other time-sensitive tasks. Why it matters: This points to growing research interest in advanced robotics in the region, especially given the focus on smart cities and automation.
The article discusses research on fine-tuning text-to-image diffusion models, including reward function training, online reinforcement learning (RL) fine-tuning, and addressing reward over-optimization. A Text-Image Alignment Assessment (TIA2) benchmark is introduced to study reward over-optimization. TextNorm, a method for confidence calibration in reward models, is presented to reduce over-optimization risks. Why it matters: Improving the alignment and fidelity of text-to-image models is crucial for generating high-quality content, and addressing over-optimization enhances the reliability of these models in creative applications.
Nicu Sebe from the University of Trento presented recent work on video generation, focusing on animating objects in a source image using external information like labels, driving videos, or text. He introduced a Learnable Game Engine (LGE) trained from monocular annotated videos, which maintains states of scenes, objects, and agents to render controllable viewpoints. Why it matters: This talk highlights advancements in cross-modal AI, potentially enabling new applications in gaming, simulation, and content creation within the region.
Mingyu Ding from UC Berkeley presented research on endowing robots with human-like commonsense and physical reasoning capabilities. The talk covered multimodal commonsense reasoning integrating vision, world models, and language-based task planners. It also discussed physical reasoning approaches for robots to infer dynamics and physical properties of objects. Why it matters: Enhancing robots with these capabilities can improve their ability to generalize across everyday tasks, leading to greater social benefits and impact.
Researchers are exploring methods for evaluating the outcome of actions using off-policy observations where the context is noisy or anonymized. They employ proxy causal learning, using two noisy views of the context to recover the average causal effect of an action without explicitly modeling the hidden context. The implementation uses learned neural net representations for both action and context, and demonstrates outperformance compared to an autoencoder-based alternative. Why it matters: This research addresses a key challenge in applying AI in real-world scenarios where data privacy or bandwidth limitations necessitate working with noisy or anonymized data.
Dr. Yali Du from King's College London will give a presentation on learning to cooperate in multi-agent systems. Her research focuses on enabling cooperative and responsible behavior in machines using reinforcement learning and foundation models. She will discuss enhancing collaboration within social contexts, fostering human-AI coordination, and achieving scalable alignment. Why it matters: This highlights the growing importance of research into multi-agent systems and human-AI interaction, crucial for developing AI that integrates effectively and ethically into society.
Qingbiao Li from the Oxford Robotics Institute is researching decentralized multi-robot coordination using Graph Neural Networks (GNNs). The approach builds an information-sharing mechanism within a decentralized multi-robot system through GNNs and imitation learning. It also uses visual machine learning-assisted navigation with panoramic cameras to guide robots in unseen environments. Why it matters: This research could improve the effectiveness of automated mobile robot systems in urban rail transit and warehousing logistics in the GCC region, where smart city initiatives are growing.
Patrick van der Smagt, Director of AI Research at Volkswagen Group, discussed the use of generative machine learning models for predicting and controlling complex stochastic systems in robotics. The talk highlighted examples in robotics and beyond and addressed the challenges of achieving quality and trust in AI systems. He also mentioned his involvement in a European industry initiative on trust in AI and his membership in the AI Council of the State of Bavaria. Why it matters: Understanding control in robotics, along with trust in AI, are key issues for further development of autonomous systems, especially in industrial applications within the GCC region.
A Marie Curie Fellow from Inria and UIUC presented research on stochastic gradient descent (SGD) through the lens of Markov processes, exploring the relationships between heavy-tailed distributions, generalization error, and algorithmic stability. The research challenges existing theories about the monotonic relationship between heavy tails and generalization error. It introduces a unified approach for proving Wasserstein stability bounds in stochastic optimization, applicable to convex and non-convex losses. Why it matters: The work provides novel insights into the theoretical underpinnings of stochastic optimization, relevant to researchers at MBZUAI and other institutions in the region working on machine learning algorithms.
This paper addresses exploration in reinforcement learning (RL) in unknown environments with sparse rewards, focusing on maximum entropy exploration. It introduces a game-theoretic algorithm for visitation entropy maximization with improved sample complexity of O(H^3S^2A/ε^2). For trajectory entropy, the paper presents an algorithm with O(poly(S, A, H)/ε) complexity, showing the statistical advantage of regularized MDPs for exploration. Why it matters: The research offers new techniques to reduce the sample complexity of RL, potentially enhancing the efficiency of AI agents in complex environments.
An associate professor of Statistics at the University of Toronto gave a talk on how ensemble learning stabilizes and improves the generalization performance of an individual interpolator. The talk focused on bagged linear interpolators and introduced the multiplier-bootstrap-based bagged least square estimator. The multiplier bootstrap encompasses the classical bootstrap with replacement as a special case, along with a Bernoulli bootstrap variant. Why it matters: While the talk occurred at MBZUAI, the content is about ensemble learning which is a core area for improving AI model performance, and is of general interest to the AI research community.
Tailin Wu from Stanford presented research on using machine learning to accelerate scientific discovery and simulation at MBZUAI. The work covers learning theories from dynamical systems with improved accuracy and interpretability. It also introduces LAMP, a deep learning model optimizing spatial resolutions in simulations. Why it matters: Efficient AI-driven scientific simulation has broad implications for research in physics, biomedicine, materials science and engineering across the region.
Ivan Laptev from INRIA Paris presented a talk at MBZUAI on embodied multi-modal visual understanding, covering advancements in video understanding tasks like question answering and captioning. The talk highlighted recent work on vision-language navigation and manipulation. He argued that detailed understanding of the physical world through vision is still in early stages, discussing open research directions related to robotics and video generation. Why it matters: The discussion of robotics applications and future research directions in embodied AI could influence the direction of AI research and development in the UAE, particularly at MBZUAI.
MBZUAI researchers are using federated learning to optimize energy production and use in microgrids, balancing individual and grid-level needs with a focus on sustainability. They presented a multi-agent framework called MAHTM at the ICLR 2023 workshop, aiming to minimize the carbon footprint of electrical grids. The system uses three layers of decision-making agents to minimize cost, decrease carbon impact, and balance production. Why it matters: This research offers a novel approach to integrating renewable energy sources into existing grids, potentially accelerating the transition to more sustainable energy systems in the region and globally.
Bruno Ribeiro from Purdue University presented a talk on Asymmetry Learning and Out-of-Distribution (OOD) Robustness. The talk introduced Asymmetry Learning, a new paradigm that focuses on finding evidence of asymmetries in data to improve classifier performance in both in-distribution and out-of-distribution scenarios. Asymmetry Learning performs a causal structure search to find classifiers that perform well across different environments. Why it matters: This research addresses a key challenge in AI by proposing a novel approach to improve the reliability and generalization of classifiers in unseen environments, potentially leading to more robust AI systems.
Giulia De Masi, Principal Scientist at the Technology Innovation Institute (TII) in Abu Dhabi, specializes in Collective Intelligence and Swarm Robotics. Her work focuses on designing emergent behaviors in robot swarms through local interactions, drawing inspiration from social insects. De Masi's background includes positions at academic institutions in the UAE and a PhD from the University of Rome La Sapienza. Why it matters: This highlights the growing focus on swarm robotics and collective intelligence research within the UAE, with potential applications in various industries.
Ian Reid, a Professor of Computer Science at the University of Adelaide, gave a talk at MBZUAI on leveraging deep learning to go beyond geometric SLAM. The talk covered using prior domain knowledge to improve map and shape estimation and enabling navigation in unvisited environments. The research aims to turn cameras into devices for flexible, large-scale situational awareness or "Spatial AI" sensors. Why it matters: Integrating deep learning with SLAM could significantly advance robotic navigation and spatial understanding, with applications for autonomous systems in various industries.