% abstract, choose between abstract and summary

Multi-agent reinforcement learning (MARL) is a promising paradigm for learning problems that involve multiple decision makers. Contrary to centralized MARL with a central controller, decentralized (independent) MARL is a more practical paradigm in terms of scalibility, privacy, and computational cost, yet more challenging due to non-stationarity of the environment from an agent’s perspective. The non-stationarity challenge arises as the evolution of the environment and each agent's payoffs will depend on the behavior of other agents. In value-based MARL, two-timescale learning is shown to address this issue. In such a learning dynamics, agents update their value function estimates at a timescale slower than their local Q-function estimates, and therefore, the game is rendered locally stationary with respect to the strategy of other agents. However, two-timescale dynamics in decentralized Q-learning has been studied only in two-player zero-sum games. In this thesis, we focus on a newly emerged and important class of stochastic games, stochastic potential games (SPG). We develop a many-player extension of the two-timescale decentralized Q-learning algorithm as the first game-agnostic value-based MARL algorithm in stochastic games, and analyze its asymptotic converges to Nash equilibria in SPGs. We evaluate the empirical performance of the algorithm on two SPG benchmarks, network routing games and distancing games.


