Autonomous exploration is a fundamental challenge in robotics, with broad implications for operations in remote or hazardous environments. Diffusion policies, generative models that can predict robot actions, have emerged as powerful tools for navigation. However, these models are typically trained with imitation learning (IL) and often fail to generalize beyond their demonstrations. Furthermore, fine-tuning diffusion policies with reinforcement learning (RL) is challenging, as backpropagating through the denoising chain is non trivial, and sample collection in the real world is costly. This thesis addresses such challenges by adapting Q-weighted Variational Policy Optimization (QVPO) to fine-tune Navigation with Goal Masked Diffusion (NoMaD), a state-of-the-art diffusion-based navigation model that unifies goal-conditioned navigation and task-agnostic exploration through goal masking, predicting multimodal action sequences directly from past RGB frames.