\chapter{Introduction}
\label{chap:intro}

Artificial Intelligence (\gls{AI}) has recently achieved remarkable progress, largely enabled by the Transformer architecture \cite{NIPS2017_3f5ee243}, which replaced recurrence and convolutions with self-attention, resulting in both faster training and state-of-the-art performance across language tasks. Building upon this foundation, \gls{LLM}s such as GPT-3 \cite{NEURIPS2020_1457c0d6} have demonstrated unprecedented few-shot and zero-shot capabilities, highlighting the power of scale in model training. Scaling laws further reveal that performance in language modeling follows smooth power-law trends with respect to model size, dataset size, and compute \cite{kaplan2020scalinglawsneurallanguage}. While these trends suggest that ever larger models can achieve better generalization, they also imply unsustainable growth in computational cost, energy usage, and memory demand. The resulting tension between performance and efficiency forms a central challenge in the continued development and deployment of \gls{LLM}s.

Approximate computing has emerged as a promising paradigm to address this challenge by deliberately introducing controlled imprecision into computation. Instead of performing all operations with exact, high-precision arithmetic, approximate methods aim to reduce the redundancy inherent in over-parameterized models and exploit the tolerance of neural networks to small perturbations. Techniques such as quantization \cite{gholami2021surveyquantizationmethodsefficient}, pruning \cite{han2016deepcompressioncompressingdeep}, \gls{LoRA} \cite{hu2021loralowrankadaptationlarge}, stochastic perturbation (noise injection)\cite{gao2024spuqperturbationbaseduncertaintyquantification}, and stochastic memory masking \cite{alistarh2017qsgdcommunicationefficientsgdgradient} each target different aspects of efficiency. Quantization reduces bit-width, trading a small drop in accuracy for dramatic improvements in latency and memory footprint. Pruning eliminates redundant parameters, reducing model size and energy consumption \cite{han2016deepcompressioncompressingdeep}. \gls{LoRA} introduces low-rank reparameterizations into pre-trained transformers, achieving task adaptation with only a fraction of trainable parameters and negligible inference overhead \cite{hu2021loralowrankadaptationlarge}. Stochastic perturbation and memory masking further explore non-traditional representations and reduced memory precision to unlock additional gains in efficiency  \cite{alistarh2017qsgdcommunicationefficientsgdgradient}.

The motivation of this thesis lies in bridging the gap between theoretical advances in approximation methods and their systematic application to modern large language models. Rather than retraining models from scratch, which is computationally prohibitive, this work adopts a training-time approximation framework. In this setup, techniques such as quantization (implemented through a hybrid approach where weights are quantized once during initialization and activations are quantized dynamically during inference), \gls{LoRA}, pruning, stochastic perturbations, and stochastic memory masking are integrated directly into the fine-tuning process. The quantization strategy follows a BF16 computation pipeline: quantized weights and activations are dequantized before matrix multiplication, computed in BF16 precision, and then requantized so that the next layer receives quantized activations. This unified approach extends beyond classical post-training quantization \gls{PTQ} or \gls{QAT}, enabling a systematic study of how these approximation strategies individually and in combination affect training dynamics, generalization, and deployment-time efficiency. By leveraging prior insights on model scaling, compression, and efficient adaptation, the thesis establishes a benchmark framework for approximated \gls{LLM}s, providing a clear view of the trade-offs between accuracy, energy consumption, and computational cost.

To ensure a fair comparison, the evaluation methodology distinguishes between training and inference efficiency. While training cost is influenced by optimizer states, gradient accumulation, and checkpointing strategies, inference time and generated output size directly reflect real-world deployment scenarios. Besides, this thesis emphasizes inference metrics such as \gls{BLEU}, \gls{ROUGE-L}, SBERT and F1 score \cite{yang2018adaptationsrougebleubetter,alon2023detectinglanguagemodelattacks,yacouby-axman-2020-probabilistic}. This balance between efficiency and quality provides a reliable basis for assessing approximation techniques in practice.

\section{Objectives}
\label{sec:objectives}

The remarkable progress of large \gls{LLM}s has been accompanied by a rapid growth in model size, dataset requirements, and computational cost \cite{NEURIPS2020_1457c0d6, kaplan2020scalinglawsneurallanguage}. While scaling laws provide strong evidence that larger models yield improved performance, they also reveal a widening gap between model capabilities and the resources required to train and deploy them. This gap makes it increasingly difficult to reproduce state-of-the-art results outside of large industrial labs, raising concerns about accessibility, fairness, and sustainability in the development of \gls{AI}.

In this context, approximate computing has emerged as a promising research direction. The central objective of this thesis is to investigate how approximation techniques including quantization \cite{gholami2021surveyquantizationmethodsefficient,jacob2017quantizationtrainingneuralnetworks}, pruning \cite{han2016deepcompressioncompressingdeep,han2015learningweightsconnectionsefficient}, low-rank adaptation through \gls{LoRA} \cite{hu2021loralowrankadaptationlarge,zeng2024expressivepowerlowrankadaptation,zhao2024loraland310finetuned}, stochastic perturbation\cite{bengio2013estimatingpropagatinggradientsstochastic,chen2024nlperturbatorstudyingrobustnesscode,chinova2025stochastic}, and memory-efficient mechanisms \cite{alistarh2017qsgdcommunicationefficientsgdgradient} affect the training dynamics and deployment efficiency of modern large language models. These methods target different aspects of efficiency, respectively reducing arithmetic precision, eliminating redundant connections, introducing low-rank reparameterizations, or exploiting alternative noise and memory-optimized representations.

Another key objective is to design and implement a unified benchmarking framework that allows these approximation methods to be consistently applied across different transformer-based models. Current literature often evaluates approximations in isolation, using distinct datasets, baselines, and metrics, which makes comparisons difficult and conclusions fragmented. By contrast, this work provides a standardized platform where approximations can be directly compared, facilitating a clearer understanding of their strengths, weaknesses, and synergies.

Finally, the thesis aims to go beyond isolated analysis by exploring combinatorial approximation strategies. While most prior studies focus on one technique at a time, real-world deployment often benefits from layered approaches such as combining quantization with pruning, or integrating \gls{LoRA} with stochastic memory masking. An important research question addressed here is whether such combinations unlock efficiency gains without excessively degrading model quality. The evaluation balances performance metrics such as perplexity, \gls{BLEU}, \gls{ROUGE-L}, F1, SBERT scores with efficiency measures like inference time and generated text output size, providing a holistic view of the trade-offs involved.

\section{Contributions}
\label{sec:contributions}

Following these objectives, the contributions of this thesis advance both the methodological and empirical understanding of approximation for \gls{LLM}s. The first contribution is the development of a unified and extensible benchmarking framework that integrates multiple approximation methods into a single experimental environment. The framework supports quantization, pruning, \gls{LoRA}, stochastic perturbation, and stochastic memory masking. Its modular design enables fair, side-by-side comparisons of diverse techniques and allows straightforward extension to new approximation methods, architectures, and datasets. By consolidating previously fragmented practices in the literature, this framework lays the foundation for standardized evaluation of efficiency-oriented methods in large language models.

The second contribution lies in applying the framework to modern instruction-tuned transformer models, specifically LLaMA-3.2-1B-Instruct and Gemma-3-1B-Instruct. This demonstrates portability across architectures and highlights model-specific sensitivities to approximation techniques. While this work focuses on decoder-only architectures, it also acknowledges that encoder–decoder models such as T5 and BART may interact differently with approximation methods like \gls{LoRA} and quantization due to their two-stage structure. Concentrating on decoder-only models ensures that the findings are directly applicable to the current generation of widely deployed \gls{LLM}s.

The third contribution is a systematic analysis of inference-time efficiency versus accuracy trade-offs. By evaluating not only task performance (perplexity, \gls{BLEU}, \gls{ROUGE-L}, F1 , SBERT) but also inference time and generated output size, the study explicitly acknowledges efficiency as a first-class evaluation dimension. This approach aligns with the growing demand for sustainable and resource-efficient \gls{AI}.

The fourth contribution is an exploration of combined approximation strategies. By investigating whether layering techniques such as quantization with pruning or \gls{LoRA} with stochastic memory masking can achieve superior efficiency without unacceptable accuracy loss, the thesis contributes practical insights into whether approximations interact additively, synergistically, or adversarially in real-world deployments.

Finally, the contributions are validated across multiple instruction-tuned datasets, including Alpaca, Databricks-Dolly-15k, and AgentInstruct. This cross-dataset evaluation ensures that the conclusions are not tied to a single benchmark but instead generalize across different training regimes. Taken together, these contributions represent both a methodological advance through the creation of a unified approximation benchmark and a substantive empirical contribution through the systematic study of approximation trade-offs in modern \gls{LLM}s.
