Heating, Ventilation, and Air Conditioning (HVAC) systems account for a significant proportion of the world's total energy consumption. In recent years, research in the HVAC area has focused on developing new control systems based on artificial intelligence, particularly reinforcement learning. Reinforcement learning algorithms are well suited to the prior objective of HVAC: reducing energy consumption while maintaining good thermal comfort. However, the commercial application of such technology is still uncommon. This study aims to investigate the most effective training methods for algorithms with a view to future real-world applications. Three different reinforcement learning algorithms (DDPG, TD3, and SAC) were trained in a simulated residential environment using Energym, each in three different ways: online as is, offline, and offline with online fine-tuning.