Header Image

Workshop Information:

Benchmarking has generally been identified as an important instrument for evaluating robot performance and predicting how robots will satisfy the special needs of users. This becomes increasingly important for human-centered robots, such as humanoid and wearable robots, as they move out of the lab and have to perform in real-life situations. Recent benchmarking efforts included several international robotics competitions which received a lot of attention, such as the Cybathlon, DARPA Robotic Challenge, RoboCub or Rockin. While they often address a more qualitative performance of specific tasks, there still is a clear lack of unified benchmarking scenarios and quantified key performance indicators.

The workshop is organized in the context of the EUROBENCH project, which aims to create the first unified benchmarking framework for robotic systems in Europe. This framework will allow companies and researchers to test the performance of their robots at any stage of development and to get reliable information about robots they may consider purchasing. The project is mainly focused on bipedal locomotion and therefore on humanoid robots as well as lower limb exoskeletons and prostheses, but the general methodology can also be extended to other robotic technologies. The EUROBENCH project is not only establishing two benchmarking facilities (one for humanoids and one for wearable robots) that will be accessible to the entire academic and industrial robotics community. It is also creating unified benchmarking testbeds, standardized tests and related performance indicators and a benchmarking software, that will allow using the tests in robotics labs worldwide. Benchmarks, on the one hand, measure elementary locomotion skills with respect to stability, robustness, and motions on different terrains and are on the other hand inspired by real-world use cases.

Topics of interest:

Workshop Organizers:

Katja Mombaur Optimization, Robotics & Biomechanics Heidelberg University
Felix Aller Optimization, Robotics & Biomechanics Heidelberg University
Diego Torricelli Neural Rehabilitation Group Spanish National ResearchCouncil (CSIC)

Workshop Objectives:

Since the EUROBENCH project applies a cascade funding scheme, many research teams are contributing through sub-projects which are selected in two separate calls. Projects of the first call (which have recently started), are contributing to the definition and setup of the benchmarking scheme, while projects of the second call (due next year) will be evaluating the benchmarking facility and at the same time improve their robotic systems. The objectives of this planned Humanoids 2019 workshop are the following:

Workshop Schedule:

09:00 - 09:30 Katja Mombaur Welcome & Introduction to Benchmarking and the EUROBENCH project
09:30 - 10:00 Francesco Ferro A robotics platform for benchmarking bipedal locomotion skills
10:00 - 10:30 Lorenzo Saccares Toward market-driven design requirements for humanoids and wearable robots
10:30 - 11:00 Coffee Break
11:00 - 11:30 Felix Aller Design choices and architecture of a methodological benchmarking framework
11:30 - 12:00 Vittorio Lippi Benchmarking Humanoid postural control–techniques and technologies inspired by human experiments
12:00 - 12:30 Jawad Masood TestEd – A Reconfigurable Benchmark Solution for Narrow Use-cases
12:30 - 14:00 Lunch Break
14:00 - 14:30 Francesca Negrello DYSTURBANCE - developing a framework to benchmark legged systems balance
14:30 - 15:00 Michael Lorenz Magnetometer-free Inertial Motion Capture System with Visual Odometry
15:00 - 15:30 Coffee Break
15:30 - 16:00 Christian Schumacher Generating normative data on mechanically induced upper-body perturbations in the BeStaBle project
16:00 - 16:30 Davide Giusino Development of a Human Factors protocol for lower-limb exoskeletons benchmarking
16:30 - 16:45 Yu-Chi Lin Evaluation of the Robustness of Humanoid Footstep Sequences and Humanoid Contact Planning using Learned Capturability Prediction
16:45 - 17:00 Francisco Javier Andrade Chavez A Benchmarking of DCM Based Architectures for Position, Velocity and Torque Controlled Humanoid Robots
17:00 Final Discussion

Confirmed Speakers:

Vittorio Lippi Technische Universität Berlin
Title: Benchmarking Humanoid postural control–techniques and technologies inspired by human experiments
Abstract: This work describes the technical implementation of a testbed to perform state of the art posture control and balance experiments and provides versatile tools for the design of new experiments. The benchmarking is based on measured variables representing body kinematics performed under perturbed conditions. The proposed system is inspired by the experience with human posture control experiments oriented to to characterize and quantify the balancing behavior across subjects of different body size and weight, and to relate balancing problems to insufficiencies in sensors, actuators, or specific disturbances. On this basis it aims to provide quantitative measures of the performance.
Lorenzo Saccares IUVO Pisa
Title: EUROBENCH – Toward market-driven design requirements for wearable robots
Abstract: The aim of EUROBENCH project is to create the first unified benchmarking platform for humanoids and wearable robots in Europe. Such ecosystem will allow companies and researchers to test the performance of their robots at any stage of their development. To this end, main objectives of EUROBENCH project are: (i) the design of a unified benchmarking software, which will allow in each laboratory worldwide to test, score and compare robotic systems; (ii) the construction of two benchmarking facilities, one for wearable robots and the other for humanoid robots, to allow for standardized tests in unique locations, saving resources and time. EUROBENCH is a EU project with 10 partners from 5 different countries. More than 15 third parties are collaborating with the consortium to complete skill and resources. In this talk, the contribution of IUVO in EUROBENCH will be summarized and presented. In particular, the experience matured by IUVO on the requirements for wearable robots in different application domains will be discussed. Finally, current trends, research direction and future challenges of benchmarking for wearable robots will be addressed.
Francesca Negrello Università di Pisa
Title: DYSTURBANCE - developing a framework to benchmark legged systems balance
Abstract: Important aspects that characterizes a legged robot system are the robustness of its static balance and of its dynamic walking gait. Here we present the preliminary results of DYSTURBANCE, a sub-project of EUROBENCH, towards the development of a perturbation system to be used for the systematic characterization of walking robots and locomotion systems. Specifically, the system enables reproducible issuing of both dynamic and static perturbations forces, for the systematic study of gait and balance resilience and robustness. The proposed framework consists in test-bench inspired from classical Resilience testing machines used in standardized testing of material samples (as the Charpy test rig), and is completed by a set of key performance indicators which are extracted during the tests. Preliminary results of dynamic perturbations on Alter-EGO, a humanoid robot provided with a two-wheeled base, are presented.
Michael Lorenz Technische Universität Kaiserslautern
Title: Magnetometer-free Inertial Motion Capture System with Visual Odometry
Abstract: The IMCVO Eurobench sub-project aims to develop a wearable, easy mountable, sensory system, which is capable to precisely track the 3D kinematics of the lower body of bipedal robots, exoskeletons or even humans, to create consistent benchmark data in a large variety of surroundings. Our proposed solution builds on recent advances in magnetometer-free inertial motion capture and a tracking camera, which provides visual odometry information. Additional foot pressure soles are planned, to provide information about the ground contact of the walking object or subject. Based on little information about the robot / human, e.g. the segment lengths, we are able to provide 3D kinematics for benchmarking motions of humans or arbitrary bipedal robots.
Jawad Masood CTAG - Automotive Technology Centre of Galicia
Title: TestEd – A Reconfigurable Benchmark Solution for Narrow Use-cases
Abstract: The need to improve the quality of life and to reduce job-related musculoskeletal diseases, exoskeleton technology is becoming a popular tool among different industry sectors. The latest market research shows the exoskeleton technology is at 110 million USD in 2019 and forecasts 950 million USD in 2024 . This is 56.3% compound annual growth rate for the next five years. At CTAG, we are testing industrial exoskeleton technology for the last three years. We believe the biggest bottleneck for this technology to be industrially sustainable and to achieve ambitious market goals is the lack of benchmarking and standardization. For the last six months, CTAG is working on a project called TestEd that is centred around developing a modular and reconfigurable testbed for mobility in the narrow spaces. I will talk about the mechanical specification of this testbed with the help of three use-cases taken from the manufacturing and construction industry
Francesco Ferro PAL Robotics
Title: A robotics platform for benchmarking bipedal locomotion skills
Abstract: tba.
Davide Giusino Università di Bologna
Title: Development of a Human Factors protocol for lower-limb exoskeletons benchmarking
Abstract: Currently, benchmarking of wearable robotic systems for bipedal locomotion is usually based on technical performance indicators or effects on the user’s motor functions. As a consequence, there is a lack of understanding of the users’ experience in this domain. To fill this gap, the Eurobench sub-project STEPbySTEP will integrate the Human Factors’ perspective to develop a framework for the evaluation of assistive lower-limb exoskeletons (LLE) in a stairs-based testbed scenario. The designed protocol features three execution phases (before, during and after task), as well as a mixed-method approach entailing the collection of qualitative, quantitative, subjective and physiological data. The methodology encompasses semi-structured interviews gathering users’ feedback about safety and design, observational recordings of the human-exoskeleton interaction, a dual task experimental paradigm to detect cognitive-motor interference, surveys about perceptions of local musculoskeletal pressure, exosystem usability and trust in human-robot collaboration, and recordings of the user electrodermal activity and heart rate as mental workload parameters. The set of metrics will be validated in medical rehabilitation settings by sampling patients with several neuromotor disorders. The aim is to define a standard scoring procedure and introduce a new user-centred index for LLE performance. The ultimate goal is to outline a benchmarking scheme in order to compare exosystems and define which one is most suitable for the final user.
Christian Schumacher Technische Universität Darmstadt
Title: Generating normative data on mechanically induced upper-body perturbations in the BeStaBle project
Abstract: In the BeStaBle project, as part of EuroBench, we aim to compare mechanically and self-induced perturbations in terms of their suitability for benchmarking postural control. A first step is to collect normative data of human subjects receiving different types of mechanical perturbations. This talk will cover the benchmarking of postural balance of the upper body. In research, common perturbation protocols involve moving platforms or force pulls and pushes during standing. However, these tasks perturb the person’s balance as a whole. In order to study upper-body balance in an isolated manner, we applied artificial and impulse-like torques at the upper body of human subjects during quiet standing. For this, we used an Angular Momentum Perturbator (AMP) that, in contrast to common devices, perturbed the upper-body posture with only minimal effect on Centre of Mass (CoM) excursions. By measuring kinetic, kinematic and electromyographic data, we were able to assess the subjects’ response behavior, including muscle activity of major leg muscles. We will present results on the reactive contribution of different mono- and biarticular muscles in the leg to restore upright upper-body postures. Finally, implications for the design of new, simpler benchmarking platforms will be discussed.
Katja Mombaur Universität Heidelberg
Title: Welcome & Introduction to Benchmarking and the EUROBENCH project
Abstract: tba
Felix Aller Universität Heidelberg
Title: Design choices and architecture of a methodological benchmarking framework
Abstract: tba

Workshop Contributions:

Call for Poster Contribution:

The workshop is soliciting posters for a poster session and a spotlight talk for all participants. To participate, please submit a poster title and 1-2 page(s) draft in two-column format in the IEEE style (in PDF format, max 5MB) via email to: Felix Aller (felix.aller(at)ziti.uni-heidelberg.de), titled: “Benchmarking WS Poster, Humanoids 2019”. All contributions will undergo a brief review by the organizers, and the authors will be notified.

Submissions Schedule:

Poster abstract submission: September 20th, 2019
Notification of acceptance: September 22nd, 2019 (earlier submissions will receive an earlier response)
Workshop: October 15th, 2019


This workshop is supported by the project EUROBENCH (European Robotic Framework for Bipedal Locomotion Benchmarking) funded by H2020 Topic ICT 27-2017 under grant agreement number 779963.