The growing diffusion of Large Language Models (LLMs) has stimulated increasing interest in their application to supply chain management, a field where managerial decisions require precision, efficiency, and adaptability. Despite the widespread use of general-purpose benchmarks such as MMLU or HELM, the literature highlights the absence of systematic evaluation frameworks specifically designed for supply chain contexts. This thesis addresses that gap by developing a set of benchmarks to assess the reliability, efficiency, and managerial usefulness of LLMs. The research is guided by two central questions: (i) which combinations of datasets, evaluation metrics, and prompting strategies enable the construction of meaningful benchmarks for supply chain tasks; (ii) which language model currently offers the best balance among accuracy, speed, and cost.