Understanding how neural networks shift from memorization to generalization is crucial to modern AI, particularly in the context of emergent behaviors such as In-Context Learning (ICL) and grokking. While prior work has studied these phenomena mainly through accuracy and structural analysis, the role of uncertainty in shaping this transition has been largely overlooked. In this work, we present a probabilistic framework to investigate ICL and grokking from a Bayesian perspective, focusing on how uncertainty evolves during training and influences generalization dynamics. First, we develop a reproducible experimental pipeline based on modular arithmetic tasks, enabling controlled evaluation of in-distribution and out-of-distribution generalization in Transformer models.