Taking Mobile Manipulators into the Real World

Robotics: Science and Systems Workshop

Friday, July 14 2023

Join In

The workshop was held on Friday, July 14 2023. Recordings of the talks can be downloaded from the Schedule below.


This workshop seeks to progress research in the topic of real-world manipulation and mobile manipulation, that is manipulators operating in environments which may be outdoors, naturally occurring, and unstructured. These real-world environments have unique characteristics making them non-trivial and difficult for robot operation compared to typical laboratory environments. Challenges include travelling over uneven and unpredictable terrain, working with deformable and non-rigid structures such as trees and other natural phenomena, and operating with high uncertainty due to environmental conditions.

This workshop aims to bring together researchers from diverse sub-communities within robotics including manipulation, mobile manipulation, field robotics, legged robotics, robot learning, SLAM, and robotic vision to discuss the following topics:

  • Is a benchmark or dataset required for progress in this area?
  • If so, should this be a real-world test environment, a high fidelity simulation, or low fidelity simulations with high domain randomisation etc.?
  • What gaps are holding outdoor and real-world mobile manipulation progress back from the rest of the manipulation field?
  • Is progress in this area going to be best achieved with data driven, analytic solutions, or a combined approach?
  • Why are manipulators slow and unsatisfying at completing tasks?
  • Mobile manipulators typically move in an unintuitive and unnatural manner, is this an issue for real-world environments?
  • What kind of sensing and contact models would be useful for environmental representation and mobile manipulation?
  • How do you handle uncertainty in real-world environments?
  • Are there specific challenges for long-term autonomy in the context of field robotics?
  • What is the role of human input in field mobile manipulation?


From Interaction to Integration: Advancing Optimal Human-Robot Interfaces for Underwater Manipulation

Paulo Padrao, Jose E Fuentes, Tero H Kaarlela, Alfredo Bayuelo, Leonardo Bobadilla

Autonomous Excavator System for Construction Automation

Ruitao Song, Samuel Ong, Liuwang Kang, Shiyu Jin, Yuan-Chih Peng, Zhenpeng He, Lingfeng Qian, Liangjun Zhang

Looking Good: Visually Informative Motion Generation for Mobile Manipulation

Sophie C Lueth, Snehal Jauhri, Georgia Chalvatzaki

IndoorSim-to-OutdoorReal: Learning to Navigate Outdoors without any Outdoor Experience

Joanne Truong, April Zitkovich, Sonia Chernova, Dhruv Batra, Tingnan Zhang, Jie Tan, Wenhao Yu

Reinforcement Learning Based Escape Route Generation in Low Visibility Environments

Hari N Srikanth


Danica Kragic

Danica Kragic
Royal Institute of Technology

Luis Sentis

Luis Sentis
University of Texas

Firas Abi-Farraj

Firas Abi-Farraj
Enchanted Tools

Georgia Chalvatzaki

Georgia Chalvatzaki
TU Darmstadt

Lukas Kaul

Lukas Kaul
Toyota Research Institute

Fabio Ramos

Fabio Ramos
NVIDIA, University of Sydney


1:30 - 1:35

Jesse Haviland


1:35 - 2:00

Fabio Ramos

Differentiable Physics Simulators for Manipulation: Challenges and Opportunities

The availability of differentiable programming languages enables the development of differentiable simulators capable of providing derivatives with respect to physics parameters and dynamics states. This opens numerous opportunities to connect modern probabilistic inference with the simulation process to infer physical parameters from real data, or to learn policies more effectively. In this talk I will present examples of methods that leverage differentiable simulators for manipulation of deformable objects, robot cutting, locomotion, and learning the structure of articulated objects. I will show that when combined with probabilistic inference, real2sim (adapting the simulator to match real data) can be done robustly with just a few observations.

2:00 - 3:00

Georgia, Fabio, Firas, Lukas, Rika

Panel/Round Table. Chaired by Tirtha

Identify the open challenges in real-world and field: manipulation, interaction and mobile manipulation. This will identify the technical, scientific and application related bottlenecks to deployment of robots to real-world environments.

3:00 - 3:30

Poster Session and Break

3:30 - 3:50

Georgia Chalvatzaki

Structuring Robot Learning for Mobile Manipulation

The increasing demand for intelligent robotic assistants in unstructured and human-inhabited environments, such as homes, hospitals, and warehouses, necessitates the development of more efficient, scalable, and safe robot learning methods. In this talk, I will discuss our research at the intersection of classical robotics and machine learning, focusing on structured learning approaches that enable mobile manipulation robots to better understand and interact with their environments. By exploiting the structure of the problem or by imposing structure as inductive bias, we enhance the learning process for real-world applications of mobile manipulation.

3:50 - 4:10

Danica Kragic

Learning Interaction for Robotics Tasks

4:10 - 4:20

Leonardo Bobadilla

Spotlight Talk From Interaction to Integration: Advancing Optimal Human-Robot Interfaces for Underwater Manipulation

4:20 - 4:30

Liangjun Zhang

Spotlight Talk Autonomous Excavator System for Construction Automation

4:30 - 4:40

Sophie Lueth

Spotlight Talk Looking Good: Visually Informative Motion Generation for Mobile Manipulation

4:40 - 5:00

Lukas Kaul

Mobile Manipulation @TRI: From the (corporate) Lab into the Real World

Challenge tasks (akin to the DARPA Robotics Challenge) can effectively drive fundamental research, while real-world testing provides an invaluable feedback mechanism to ensure research efforts translate into meaningful results. This talk will introduce TRI's approach to mobile manipulation research and reflect on the state and future of mobile manipulation research.

5:00 - 5:20

Firas Abi-Farraj

Walking the Tightrope: Simplifying the World vs. Complexifying the Robots

While mobile robots are (slowly) finding their way out of the lab, mobile manipulators are lagging behind. In this talk, I'll briefly discuss the reasons why and focus on our recent efforts to re-shape the mobile manipulation problem in a way that allows for deploying mobile manipulators in social environments (as soon as 2025!).

5:20 - 5:40

Luis Sentis

Control and Intelligent Legged Manipulation in Unstructured Natural Environments

Humanoid robots remain one of the most complex robotic systems to control and perform useful tasks. There are two forces driving the need for humanoid robots. On one hand blue collar jobs such as cleaning, inspecting, and moving objects around. On the other hand, with the rise of large language models there are chances that human-centered robots could play a role in the knowledge economy. In any case, performing relatively simple dexterous manipulation tasks in floating based robots (e.g. humanoids) remains a difficult challenge. Another dichotomy is the duality between optimal control methods and deep learning methods for control of human-centered robots. In this talk we focus on control and embodiment by exploring the combination of model based and model free approaches. Optimal control methods are used to track trajectories of humanoid robots with high fidelity. At the same time, we employ imitation learning techniques to accomplish two capabilities: social locomotion in crowded environments and dual arm manipulation of complex objects using humanoid robots.


Jesse Haviland

Jesse Haviland
Queensland University of Technology, CSIRO Data61

Ben Burgess-Limerick

Ben Burgess-Limerick
Queensland University of Technology, CSIRO Data61

Peter Corke

Peter Corke
Queensland University of Technology

Tirthankar Bandyopadhyay

Tirthankar Bandyopadhyay
CSIRO Data61

Rika Antonova

Rika Antonova
Stanford University


For any questions, please contact us at j.haviland@qut.edu.au