Towards safe and efficient transfer of robot policies from simulation to real world

Reinforcement learning approaches have demonstrated great promise for flexible and efficient robot learning. Yet, current data-driven algorithms require large amounts of data to learn even simple robotic tasks and leave open safety concerns when training on real hardware. Physics simulators offer several advantages in this regard, allowing to train robot policies entirely in simulation before the final deployment onto the real world. However, while this solves the problem of fast and safe data collection, any small inaccuracy or parameter discrepancy in simulation may potentially lead to policies which do not directly transfer to the real system. Domain randomization has recently gained a lot of traction as a method to overcome the reality gap experienced by transferred policies, encouraging robustness to domain shifts by randomizing physical parameters in the simulated scene. As current applications of this method require tedious manual engineering to find optimal randomization ranges, we introduce a novel algorithm to automatically identify the parameter distributions to train on, based on limited real-world data safely collected through human demonstrations. We show that the optimized distributions are capable of compensating for unmodelled phenomena in simulation. Furthermore, we evaluate our method on two real robots demonstrating a successful domain transfer and improved performance over prior methods.