Reinforcement learning approaches have demonstrated great promise for flexible and efficient robot learning. Yet, current data-driven algorithms require large amounts of data to learn even simple robotic tasks and leave open safety concerns when training on real hardware. Physics simulators offer several advantages in this regard, allowing to train robot policies entirely in simulation before the final deployment onto the real world. However, while this solves the problem of fast and safe data collection, any small inaccuracy or parameter discrepancy in simulation may potentially lead to policies which do not directly transfer to the real system. Domain randomization has recently gained a lot of traction as a method to overcome the reality gap experienced by transferred policies, encouraging robustness to domain shifts by randomizing physical parameters in the simulated scene.