\chapter{Discussion and Conclusion}
\label{chap:conc}

In the final chapter of this thesis, we discuss an important property of the decentralized $Q$-learning dynamics, i.e., game-agnostic convergence. We conclude the thesis by summarizing our contributions, and outlining the potential directions for future research. 

\section[Game-agnostic Learning Dynamics in MARL]{Game-agnostic Learning Dynamics in MARL}

In a very recent paper, Ding et al. \cite{ding2022independent} discuss a desirable property that a learning dynamics may possess, named \emph{game-agnostic} convergence. This property indicates that the learning dynamics converges regardless of the type of the game being played or the players' awareness of this type. Although convergence has been established for both zero-sum and potential games when players adopt fictitious play \cite{monderer1996potential,hofbauer2002global} or $Q$-learning \cite{leonardos2021exploration,leonardos2022exploration} in non-stochastic matrix games, this property has not been explored in the realm of MARL and stochastic games. Ding et al. \cite{ding2022independent}, for the first time, show that a gradient-based algorithm, named Optimistic Gradient Descent/Ascent \cite{wei2021last} converges to Nash equilibrium in both Markov zero-sum and potential two-player Markov cooperative games (an identical-interest case of Markov potential games). Therefore, our results indicate that decentralized $Q$-learning is potentially the first game-agnostic value-based MARL algorithm, and the first one to converge in multi-player SPGs.

\section[Conclusion and Future Work]{Conclusion and Future Work}

In this thesis, for the first time, we applied a purely value-based MARL approach, named decentralized $Q$-learning, to stochastic potential games, a class of Markov games with the ability to model many real-world scenarios. Following the previous established convergence in zero-sum games, our results indicate that decentralized $Q$-learning is a game-agnostic MARL algorithm. Possible directions for future research include: analyzing the finite-time convergence properties of decentralized $Q$-learning, extending the algorithm to continuous state spaces using function approximation, and delving deeper into studying game-agnostic convergence of the algorithm by considering other families of games.