Reinforcement learning (RL) algorithms are investigated in the context of the El Farol Bar problem. Adapting a RL algorithm designed for potential stochastic games to a non-stochastic scenario reveals intriguing dynamics governed by two distinct time scales. Numerical analysis demonstrates that the relative speeds of these time scales critically influence convergence behavior. Specifically, having a Q-function which evolves in a fast regime favors convergence to the (unique) symmetric mixed strategies Nash equilibrium. Conversely, when the fast scale is the one governing the learning of strategies, the convergence is instead to one of the pure strategies Nash equilibria. This fact highlights the intricate interplay between learning dynamics in RL algorithms and stability of equilibria, offering insights into their convergence properties.