Simulated environments with latent dynamic parameters are used in robotics to train controllers with Reinforcement Learning by sampling the parameters at the start of each episode, a technique known as Domain Randomization. This has proven to enable a smoother transfer of the trained controllers from a simulated environment to a real environment, compared to controllers trained in non-randomized environments. In the Domain Randomization literature, we found an established connection between adaptivity to the environment and history-based policies, whereas Markovian policies are assumed not to be adaptive and are often qualified as robust. Adaptivity in this context refers to the ability to infer the value of the dynamic parameters and deploy an optimal strategy for the inferred value.