\chapter{Methodology}
\label{chap:methodology}

This chapter outlines the methodology adopted in this thesis. Building upon the theoretical background presented in Chapter~\ref{chap:related}, it describes the datasets, baseline models, implemented approximation techniques, evaluation metrics, and experimental protocol. The goal is to establish a reproducible framework for systematically studying efficiency–accuracy trade-offs in instruction-tuned \gls{LLM}s.

A modular approximation pipeline was developed to integrate and evaluate efficiency-oriented techniques in transformer architectures. The framework supports five primary approximation methods, which are quantization, pruning, stochastic perturbation, memory masking, and \gls{LoRA}, each of which can be activated individually or in combination. This modular design enables controlled experimentation, allowing direct comparison of approximation effects across datasets and models. The workflow consists of four major stages which are model selection and initialization, approximation integration, fine-tuning and evaluation, and metric collection and analysis.

Firsly, two compact instruction-tuned transformer models were selected which are LLaMA-3.2-1B-Instruct and Gemma-3-1B-it. Both were chosen for their open availability, manageable computational footprint, and compatibility with the approximation methods. These architectures provide a representative testbed for studying approximation in mid-scale \gls{LLM}s. Secondly, the following approximate computing methods were implemented.

\begin{itemize}
    \item \textbf{Quantization} is implemented as a simulated low-precision scheme (INT8, INT4, NF4) in which weights are quantized statically and activations dynamically, while computations remain in BF16 to ensure numerical stability and hardware independence. Weights are quantized once at model initialization or via a dedicated \texttt{quantize\_model()} pass, whereas activations are quantized dynamically before each forward pass. Matrix multiplications are performed in BF16, and outputs are subsequently requantized before propagation to the next layer.
    
    \item \textbf{Pruning} enforces structured sparsity by removing a fixed fraction of weights with the lowest magnitudes according to an $L_1$-norm criterion. This reduces the parameter count and computational load while maintaining the overall network topology and representational capacity.
    
    \item \textbf{Stochastic perturbation} injects Gaussian noise into activations before self-attention projections(before query, key and value projections)  within the MLP gate layers, simulating arithmetic uncertainty and the behavior of stochastic hardware accelerators. By targeting activation projections in the MLP rather than self-attention layers, this approach enhances robustness to noise-induced variability while maintaining training stability and controllable experimental conditions.

    \item \textbf{Memory masking} randomly deactivates a small subset of tensor elements within self-attention outputs, simulating approximate or lossy memory access.
    
    \item \textbf{\gls{LoRA}} introduces efficient trainable subspaces for fine-tuning large models by reparameterizing each pretrained weight matrix $W_0$ as 
    \[
        W = W_0 + \frac{\alpha}{r}BA,
    \]
    where $A \in \mathbb{R}^{r \times d}$ and $B \in \mathbb{R}^{d \times r}$ are low-rank adapters with rank $r \ll d$. LoRA is applied to the query and value projection matrices (\texttt{q\_proj}, \texttt{v\_proj}) within each attention block to enable efficient adaptation with minimal parameter overhead. Only the LoRA parameters $(A, B)$ are trainable, while all pretrained weights remain frozen. The applied configuration uses $r = 8$, $\alpha = 16$, and dropout $p = 0.05$, corresponding to $\alpha = 2r$.
\end{itemize}


All modules were parameterized through a unified configuration system, allowing flexible combinations such as INT8 + \gls{LoRA}, Pruned + \gls{LoRA}, or INT8 + Pruned + \gls{LoRA}. This ensured reproducible comparisons under identical training conditions.

Thirdly, each approximated model underwent fine-tuning on three instruction-following datasets which are Alpaca, Databricks-Dolly-15k, and AgentInstruct.  
Due to GPU memory and runtime constraints, the Alpaca and Dolly datasets were subsampled to 15{,}000 examples each while maintaining diversity across instruction types. Fine-tuning was conducted using the AdamW optimizer with cosine learning rate decay and linear warmup. Models were trained for 3–5 epochs, using mixed batch sizes (2–16 samples) depending on memory availability.  All experiments used a sequence length of 128 tokens. Approximation parameters were fixed throughout training to ensure stability and fair comparison. 

Finally, to assess the effectiveness of approximation, complementary quality and efficiency metrics, which are \gls{BLEU} score, \gls{ROUGE-L} score, F1 score, SBERT score inference time, output size and final model size, were collected. The evaluation protocol was identical for all configurations, measuring both linguistic quality and efficiency. 

% ------------------------
\section{Datasets}
\label{sec:datasets}

To evaluate approximation techniques under diverse instruction-following conditions, we employ three representative instruction-tuning datasets with complementary properties: \textbf{Alpaca}, \textbf{Databricks-Dolly-15k}, and \textbf{AgentInstruct}. These datasets differ in scale, construction methodology, and task coverage, which together enable a systematic assessment of robustness and efficiency trade-offs.

\subsection{Alpaca}
The Alpaca dataset~\cite{alpaca} was introduced by Stanford’s Center for Research on Foundation Models (\gls{CRFM}) as an accessible, low-cost resource for instruction tuning. It contains approximately 52,000 instruction--response pairs generated synthetically using OpenAI’s text-davinci-003 (GPT-3.5). The construction pipeline follows the Self-Instruct paradigm~\cite{wang2023selfinstructaligninglanguagemodels}, where a seed set of 175 manually written prompts was used to bootstrap GPT-3.5 into producing thousands of novel instructions and corresponding demonstrations.

To reduce cost and maximize diversity, several modifications were made compared to the original Self-Instruct pipeline: (i) using the more powerful text-davinci-003 engine, (ii) employing aggressive batch decoding (20 generations per batch), (iii) discarding the distinction between classification and non-classification instructions, and (iv) generating a single demonstration per instruction rather than multiple. This yielded a large corpus for under \$500, making Alpaca one of the most cost-effective open instruction datasets.

\paragraph{Data Fields.} Each Alpaca entry has:
\begin{itemize}
    \item \texttt{instruction}: natural-language description of the task.
    \item \texttt{input}: optional context (present in $\sim$40\% of examples).
    \item \texttt{output}: the response generated by GPT-3.5.
    \item \texttt{text}: concatenation of instruction, input, and output using a standardized prompt template.
\end{itemize}

From a theoretical standpoint, Alpaca exemplifies the synthetic data distillation paradigm, where a strong proprietary teacher model transfers its knowledge into an open student model. While synthetic generation ensures fluency and uniformity, it also propagates biases and factual errors from the teacher model. Furthermore, Alpaca’s license restricts it to non-commercial use. Despite these caveats, Alpaca is widely adopted for controlled studies of approximation, since its scale and relative cleanliness allow systematic exploration of pruning stability, quantization degradation, and \gls{LoRA}.

\subsection{Databricks-Dolly-15k}
The Databricks-Dolly-15k dataset~\cite{DatabricksBlog2023DollyV2} represents a contrasting design philosophy. Instead of synthetic generations, it consists of approximately 15,000 human-authored instruction--response pairs contributed by thousands of Databricks employees. Prompts were crafted across eight categories, closely following the taxonomy outlined in the InstructGPT paper~\cite{ouyang2022traininglanguagemodelsfollow}: brainstorming, classification, closed QA, open QA, summarization, information extraction, creative writing, and free-form instructions. Annotators were instructed not to use generative \gls{AI} systems when writing responses, and to ground factual categories (e.g., QA, summarization) in Wikipedia.

The annotation process was deliberately lightweight: employees were given short descriptions and examples but little formal rubric, leading to linguistic richness and pragmatic variety. Midway through collection, contributors could also answer prompts written by colleagues, further increasing lexical diversity and paraphrasing variety. As a result, Dolly-15k reflects human creativity, ambiguity, and stylistic idiosyncrasies often missing in synthetic corpora.

\paragraph{Data Fields.} Each Dolly entry has:
\begin{itemize}
    \item \texttt{instruction}: the human-authored prompt.
    \item \texttt{context}: optional supporting text (e.g., a Wikipedia passage).
    \item \texttt{response}: the human-written answer.
\end{itemize}

Dolly is notable for three reasons. First, it is the first large-scale, open-source, human-generated instruction dataset released under a permissive CC-BY-SA 3.0 license, enabling both academic and commercial use. Second, its linguistic diversity makes it a stronger testbed for approximation robustness: stochastic masking, pruning, and quantization must handle real-world variation. This positions Dolly-15k as a benchmark for generalization under authentic, human-like conditions.

\subsection{AgentInstruct}
AgentInstruct~\cite{zeng2023agenttuning} originates from the AgentTuning project, designed to equip LLMs with agentic capabilities. Unlike Alpaca and Dolly, which are primarily single-turn, AgentInstruct emphasizes multi-step interaction trajectories involving planning, reasoning, and tool use. A typical entry may require decomposing a high-level instruction into sub-tasks, invoking external tools (e.g., search or calculator), and integrating intermediate results into a coherent final answer.

The dataset was constructed by generating candidate agentic trajectories (partly via GPT-4), filtering them through execution success and alignment criteria, and mixing them with general-purpose sources such as ShareGPT. The final release includes $\sim$1,800 high-quality trajectories. Although smaller in scale, each trajectory is semantically dense, capturing structured reasoning and long-context dependencies.

\paragraph{Data Fields.} 
Each entry in the AgentInstruct dataset is structured as a multi-turn conversation between a human (or user) and an \gls{AI} assistant. The data is represented as a sequence of messages, each of which contains three core attributes:

\begin{itemize}
    \item \textbf{from}: the speaker role, which is either the \emph{human/user} (corresponding to the instruction) or the \emph{assistant/gpt} (corresponding to the model’s response).
    \item \textbf{value}: the textual content of the message. For user turns, this contains the natural language instruction or follow-up question. For assistant turns, it contains the model’s intermediate reasoning, tool invocation, or final answer.
    \item \textbf{loss}: a supervision flag (Boolean or null), which specifies whether a given assistant step is included as part of the supervised training signal.
\end{itemize}

For downstream formatting, the dataset can be distilled into instruction–response pairs by selecting the latest message authored by the \emph{human/user} as the instruction, and the subsequent \emph{assistant/gpt} message as the response. This ensures a standardized alignment of prompts with their corresponding model-generated answers while preserving the conversational and multi-step reasoning nature of the dataset.

From an approximation perspective, AgentInstruct is a natural stress test for memory-saving techniques such as stochastic masking or low-rank compression. Multi-step reasoning requires retaining intermediate states over long contexts, and lossy approximations may compromise logical consistency and planning depth.

\paragraph{Summary.}
Together, these datasets span complementary aspects of instruction-following:
\begin{itemize}
    \item \textbf{Alpaca:} Large-scale, synthetic, and clean. Facilitates controlled experiments on approximation sensitivity.
    \item \textbf{Dolly-15k:} Medium-scale, human-authored, and diverse. Tests robustness against natural human variation.
    \item \textbf{AgentInstruct:} Small-scale, multi-step, agentic reasoning. Evaluates approximation impact on reasoning depth and memory.
\end{itemize}

By combining synthetic uniformity (Alpaca), authentic human diversity (Dolly), and structured agentic complexity (AgentInstruct), this suite provides a balanced foundation for analyzing approximation methods' training in \gls{LLM}s.

% ------------------------
% ------------------------
\section{Models}
\label{sec:models}

This work employs two compact instruction-tuned \gls{LLM}s as baselines: \textbf{LLaMA-3.2-1B-Instruct}, released by Meta, and \textbf{Gemma-3-1B-Instruct}, released by Google DeepMind. Both models are representative of modern transformer-based \gls{LLM}s, while being sufficiently lightweight to enable experimentation with approximate training techniques under constrained resources. Despite their relatively small size compared to frontier-scale \gls{LLM}s (tens or hundreds of billions of parameters), they preserve the architectural sophistication of current-generation autoregressive transformers, making them highly suitable for controlled benchmarking of approximation methods such as pruning, quantization, and stochastic memory masking.

\subsection{LLaMA-3.2-1B-Instruct}
The LLaMA (Large Language Model Meta \gls{AI}) family has become a cornerstone of open research in generative \gls{AI}, providing a transparent and accessible alternative to proprietary models~\cite{touvron2023llamaopenefficientfoundation}. LLaMA-3.2-1B-Instruct is a compact, instruction-tuned variant of the LLaMA 3.2 family, with approximately 1 billion parameters. Although far smaller than the largest models in the LLaMA-3.2 series (e.g., 70B), the 1B variant incorporates the same core architectural innovations, ensuring structural comparability across scales.

\paragraph{Architecture.}
LLaMA-3.2 models adopt a decoder-only transformer architecture, optimized for autoregressive next-token prediction. Key architectural components include:
\begin{itemize}
    \item \textbf{Grouped-Query Attention (\gls{GQA})}: Instead of allocating a unique set of key-value projections per attention head, \gls{GQA} shares key-value projections across groups of heads~\cite{shazeer2019fasttransformerdecodingwritehead}. This reduces memory footprint and improves inference efficiency while maintaining competitive quality.
    \item \textbf{Rotary Positional Embeddings (\gls{RoPE})}: Positional encodings are incorporated directly into the attention mechanism via rotation in complex space, enabling extrapolation to longer context windows compared to absolute embeddings~\cite{su2023roformerenhancedtransformerrotary}.
    \item \textbf{SwiGLU Activation Functions}: The feed-forward networks (\gls{FFN}s) employ a SwiGLU nonlinearity~\cite{shazeer2020gluvariantsimprovetransformer}, which has been shown to improve training stability and generalization compared to ReLU or GELU.
    \item \textbf{Long-Context Extensions}: LLaMA-3.2 incorporates efficient scaling of context length, enabling windows of up to 128k tokens in larger variants, with proportionally scaled-down limits for smaller models.
\end{itemize}

\paragraph{Instruction Tuning.}
The Instruct variant is obtained by supervised fine-tuning of the pretrained model on prompt–response pairs derived from instruction datasets (e.g., Alpaca, Dolly-15k). Further refinements may include reinforcement learning from human feedback (\gls{RLHF}) or Direct Preference Optimization (\gls{DPO}), which align model behavior with human-preferred responses~\cite{ouyang2022traininglanguagemodelsfollow}. Instruction tuning improves zero-shot usability, ensuring that the model interprets prompts as natural instructions rather than arbitrary text completion tasks.

\paragraph{Role in Benchmark.}
LLaMA-3.2-1B-Instruct serves as a canonical open-weight, instruction-tuned baseline. Its compact size allows us to fine-tune with multiple approximation strategies under realistic hardware constraints. Furthermore, its widespread use in open-source research ensures that approximation results are broadly comparable to the literature.

\subsection{Gemma-3-1B-Instruct}
Gemma is Google DeepMind’s family of lightweight, open-source language models introduced in 2025~\cite{gemmateam2025gemma3technicalreport}. The Gemma-3-1B-Instruct model is the 1B-parameter instruction-tuned variant of Gemma 3, explicitly designed for efficiency, multilinguality, and modularity. It is positioned as a counterpart to LLaMA
, offering an alternative open foundation with strong performance in constrained environments.

\paragraph{Architecture.}
Like LLaMA, Gemma models are decoder-only transformers. However, Gemma introduces a number of design choices that emphasize deployment efficiency and adaptability:
\begin{itemize}
    \item \textbf{Grouped-Query Attention (\gls{GQA})} and \textbf{multi-query variants} are employed to further reduce memory usage during inference.
    \item \textbf{Extended Context Windows}: The 1B-it variant supports sequences of up to 32k tokens, which is unusually long for such a compact model, enabling experiments on long-context approximation trade-offs.
    \item \textbf{Optimized Embedding Layers}: Gemma models apply more parameter-efficient embedding layers, reducing memory bottlenecks for multilingual vocabularies.
    \item \textbf{Parameter-Efficient Scaling}: Architectural choices are explicitly tuned to maximize FLOPs utilization at small parameter scales, providing a more competitive baseline compared to other lightweight open models.
\end{itemize}

\paragraph{Instruction Tuning.}
The Gemma-3-1B-Instruct model undergoes structured instruction tuning using both supervised datasets (e.g., human-authored instructions, synthetic prompt–response pairs) and reinforcement signals. The instruction format is standardized around explicit role tokens (\texttt{<start\_of\_turn>}, \texttt{<end\_of\_turn>}), ensuring consistency across tasks and improving robustness to conversational multi-turn alignment~\cite{gemmateam2025gemma3technicalreport}. This contrasts with LLaMA’s simpler Alpaca-style instruction-response formatting, highlighting methodological diversity in instruction-tuning pipelines.

\paragraph{Role in Benchmark.}
Gemma-3-1B-Instruct provides an architectural and methodological contrast to LLaMA-3.2-1B-Instruct. While both models are decoder-only transformers, differences in training data, tokenization, and instruction formatting allow us to probe the portability of approximation methods. For instance, quantization strategies may interact differently with Gemma’s embedding layers compared to LLaMA.

\paragraph{Comparative Analysis.}
The two models complement each other along several dimensions:
\begin{itemize}
    \item \textbf{Scale}: Both are $\sim$1B parameter models, light enough for systematic benchmarking, yet architecturally representative of frontier-scale LLMs.
    \item \textbf{Instruction-Tuning Paradigm}: LLaMA emphasizes Alpaca-style distillation and \gls{DPO} alignment, whereas Gemma uses structured conversation templates and \gls{RLHF} pipelines. This diversity ensures our approximations are tested across multiple alignment strategies.
    \item \textbf{Architectural Efficiency}: LLaMA prioritizes efficient scaling via \gls{GQA} and \gls{RoPE}, while Gemma emphasizes parameter-efficient embeddings and long-context extensions.
    \item \textbf{Research Relevance}: LLaMA serves as a widely-used, standard benchmark model, while Gemma represents a newer generation of compact, multilingual, and efficiency-oriented open models.
\end{itemize}

Together, these models establish a balanced evaluation environment: LLaMA-3.2-1B-Instruct as a stable, widely-adopted open benchmark, and Gemma-3-1B-Instruct as a forward-looking, efficiency-driven alternative. Their shared compactness makes them well suited for the experimental focus of this thesis, benchmarking approximation-aware training, while their architectural and methodological contrasts provide insight into the generalizability of approximation techniques across model families.

% ------------------------
\section{Approximation Techniques}
\label{sec:approximation}

The approximation techniques applied in this thesis directly follow the theoretical foundations outlined in Chapter~\ref{chap:related}. Each method is adapted into a practical implementation pipeline. 

\subsection{Quantization}
\label{sec:quantization}

Quantization in this thesis is implemented using a simulated quantization. Pretrained transformer models such as LLaMA-3.2-1B-Instruct and Gemma-3-1B-Instruct are quantized into low-precision representations (NF4, INT4, or INT8) while maintaining floating-point computations for numerical stability. 

\paragraph{Quantization pipeline.}
Weights are quantized once either at model initialization or via a dedicated \texttt{quantize\_model()} pass while activations are quantized dynamically before each forward pass. Computations, including matrix multiplications, are performed in BF16, and outputs are requantized before being propagated to subsequent layers. This scheme strikes a balance between numerical efficiency and hardware independence, preserving the accuracy advantages of simulated quantization while avoiding integer-only computation constraints~\cite{gholami2021surveyquantizationmethodsefficient, jacob2017quantizationtrainingneuralnetworks}.

\paragraph{Quantization operator.}
Let $W \in \mathbb{R}^{m \times n}$ denote a weight matrix. Quantization maps $W$ to a discrete set of representable values $\hat{W}$ using a scale factor $s$ and zero-point $Z$:
\begin{equation}
    \hat{W} = \text{Quantize}(W; s, Z)
    = \text{clip}\!\left(\left\lfloor \frac{W}{s} \right\rceil + Z, q_{\min}, q_{\max}\right),
\end{equation}
where $q_{\min}$ and $q_{\max}$ define the integer range (e.g., $[-127,127]$ for INT8), and $\lfloor\cdot\rceil$ denotes rounding to the nearest integer.  
The corresponding dequantization step is given by:
\begin{equation}
    \tilde{W} = s \cdot (\hat{W} - Z),
\end{equation}
yielding an approximation error $\varepsilon = W - \tilde{W}$. In this work, such quantization is performed for all linear weights, while dynamic quantization is applied to activations during inference.

\paragraph{Comparison with \gls{PTQ} and \gls{QAT}.}
The proposed method differs from classical approaches:
\begin{itemize}
    \item \textbf{\gls{PTQ}:} Post-training quantization applies $\text{Quantize}(\cdot)$ directly to pretrained weights without retraining. While computationally efficient, it often degrades accuracy below 8-bit precision~\cite{frantar2023gptqaccurateposttrainingquantization}.
    \item \textbf{\gls{QAT}:} Quantization-aware training retrains all model parameters under simulated quantization in both forward and backward passes~\cite{gholami2021surveyquantizationmethodsefficient}. It achieves high fidelity but is computationally prohibitive for billion-scale LLMs.
\end{itemize}

\paragraph{Uniform quantization.}
Uniform symmetric quantization maps each real-valued weight $w_i$ into an integer $\hat{w}_i$ according to:
\begin{equation}
    \hat{w}_i = \text{round}\!\left(\frac{w_i}{s}\right), 
    \quad s = \frac{\max(|W|)}{2^{b-1}-1},
\end{equation}
where $b$ denotes the bit width. For 8-bit and 4-bit quantization, the representable ranges are $[-127,127]$ and $[-7,7]$, respectively. Although INT8 provides stable approximations with minimal degradation, INT4 may introduce higher quantization error, particularly in attention and feed-forward layers with skewed weight distributions.  
Implementation follows:
\begin{verbatim}
def quantize_tensor_uniform(tensor, bits=8):
    qmax = 2 ** (bits - 1) - 1
    max_val = tensor.abs().max(dim=-1, keepdim=True).values
    scale = torch.clamp(max_val / qmax, min=1e-8)
    q = torch.clamp((tensor / scale).round(), -qmax, qmax)
    return q, scale
\end{verbatim}

\paragraph{NF4 quantization.}
While the original NF4 (NormalFloat-4) scheme~\cite{dettmers2023qloraefficientfinetuningquantized} defines a statistically optimal 4-bit codebook derived from a Gaussian prior, our implementation adopts an NF4-inspired design that follows similar principles but employs a distinct computational structure for practical GPU execution. Specifically, the quantizer still relies on a fixed 16-value codebook which approximates equal-probability bins of a standard normal distribution, providing higher resolution around zero where most weights concentrate.  
\begin{equation}
\mathcal{C}_{\text{NF4}} =
\begin{aligned}[t]
\{& -1.0000, -0.6962, -0.5251, -0.3949, \\
  & -0.2844, -0.1848, -0.0911, 0.0000, \\
  &     0.0911, 0.1848, 0.2844, 0.3949, \\
  &     0.5251, 0.6962, 0.8682, 1.0000 \}.
\end{aligned}
\label{eq:nf4_codebook_final}
\end{equation}

\begin{align}
    \alpha_b &= \max_{i,j} |W_{b,ij}|, \\
    \tilde{W}_b &= \frac{W_b}{\alpha_b}, \\
    \hat{W}_b &= Q_{\text{NF4}}(\tilde{W}_b), \\
    W_b^{q} &= \alpha_b \cdot \hat{W}_b.
\end{align}

Unlike the original QLoRA formulation, which performs blockwise quantization with precomputed codebook mappings, our NF4 variant implements element-wise nearest-neighbor assignment directly on GPU. Each tensor is divided into blocks of 64 elements for scaling, followed by a streaming search over the 16 codebook entries per block. This eliminates large intermediate tensors and avoids out-of-memory (OOM) errors common in naive 4D broadcasting approaches. The quantized indices and per-block scales are stored compactly and later dequantized via direct codebook lookup. This approach preserves the statistical intuition of NF4 while offering a GPU-efficient, memory-safe quantization routine that can be applied dynamically during training or inference. In contrast to Dettmers et al.~\cite{dettmers2023qloraefficientfinetuningquantized}, our variant prioritizes runtime efficiency over exact Gaussian optimality, making it well-suited for large-scale experimentation under resource constraints.


\subsection{Pruning}
\label{sec:pruning}
Pruning reduces model complexity by eliminating redundant parameters. In this benchmark, we implement unstructured magnitude pruning, where individual weights are removed based on their absolute magnitude.

\paragraph{Unstructured magnitude pruning.}
Weights are pruned under an $L_1$ threshold:
\begin{equation}
    \hat{W}_{ij} = 
    \begin{cases}
        W_{ij}, & |W_{ij}| \geq \tau, \\
        0, & |W_{ij}| < \tau,
    \end{cases}
\end{equation}
where $\tau$ is dynamically chosen to achieve a target sparsity level. This fine-grained approach maximizes 
parameter removal but leads to irregular sparsity patterns that are difficult to exploit on standard hardware.

The following Python snippet shows our implementation of unstructured pruning integrated into the training pipeline.
\begin{verbatim}
if config.get("prune"):
    prune.l1_unstructured(gate, name="weight", amount=0.05)
\end{verbatim}

In our implementation, pruning is applied exclusively to the MLP components of transformer blocks, specifically the gate projection. This choice is motivated by three factors: (i) the MLP layers constitute the majority of parameters in LLaMA and Gemma models, so sparsification here yields the largest reduction in size; (ii) attention projections are highly sensitive to pruning and even small perturbations can destabilize the softmax distributions, whereas MLP transformations exhibit higher redundancy and pruning tolerance; and (iii) PyTorch’s pruning utilities integrate seamlessly with linear MLP layers, making them a natural starting point. Concentrating pruning in MLPs thus balances efficiency gains with model robustness, consistent with observations in prior pruning studies~\cite{frantar2023gptqaccurateposttrainingquantization}.

\paragraph{Note on structured pruning.}
Structured pruning (removing entire neurons, channels, or projections) is not implemented in the current benchmark but remains an important future extension, as it produces hardware-friendly sparsity.

\subsection{\gls{LoRA}}
\label{sec:lora}

\gls{LoRA} introduces efficient trainable subspaces to fine-tune large models~\cite{hu2021loralowrankadaptationlarge}. 
Given a pretrained weight matrix $W_0$, \gls{LoRA} reparameterizes:
\begin{equation}
    W = W_0 + \frac{\alpha}{r} B A,
\end{equation}
where $A \in \mathbb{R}^{r \times d}$ and $B \in \mathbb{R}^{d \times r}$ are low-rank adapters with rank $r \ll d$. During fine-tuning, only $A$ and $B$ are updated, while $W_0$ remains frozen.

\paragraph{Application to Transformers.}
In this thesis, \gls{LoRA} is applied to the query and value projection matrices (\texttt{q\_proj}, \texttt{v\_proj}) within each attention block, following empirical best practices for instruction-tuned models. This ensures efficient adaptation with minimal parameter overhead. The applied configuration uses $r=8$, $\alpha=16$, and dropout $p=0.05$, corresponding to $\alpha = 2r$. The implementation via the Hugging Face's \gls{PEFT} library is as follows:
\begin{verbatim}
lora_cfg = LoraConfig(
    r=8, lora_alpha=16, lora_dropout=0.05, bias="none",
    target_modules=["q_proj", "v_proj"],
    task_type=TaskType.CAUSAL_LM)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()
\end{verbatim}
Only \gls{LoRA} adapters $(A, B)$ are trainable, while all pretrained parameters are frozen.

\subsection{Stochastic Perturbations}
\label{sec:stochastic}
To probe robustness against arithmetic noise, we inject Gaussian perturbations into activations before self-attention projections (the \texttt{q\_proj}, \texttt{k\_proj}, and \texttt{v\_proj} matrices) within the MLP gate layers.

\begin{equation}
    \hat{y} = (x + \epsilon) W^\top + b, 
    \qquad \epsilon \sim \mathcal{N}(0, \sigma^2 I).
\end{equation}
This simulates the uncertainty of stochastic computing and approximate arithmetic units, which may arise in low-power hardware accelerators. The following Python snippet shows our implementation of stochastic perturbations injected into MLP gate projections using a custom stochastic linear operator.

\begin{verbatim}
def stochastic_linear(x, weight, bias=None, noise_std=0.01):
    noise = torch.randn_like(x) * noise_std
    return F.linear(x + noise, weight, bias)
\end{verbatim}
 
In our benchmark, Gaussian noise was injected only into the MLP gate projections. This design reflects three considerations: (i) the MLP layers account for the majority of model parameters and provide redundancy that makes them more tolerant to stochastic perturbations; (ii) attention projections are highly sensitive to noise, where even small perturbations can destabilize token alignment and degrade coherence; and (iii) implementation is straightforward in MLP gate projections, which are single linear transformations, while attention layers involve multiple interdependent projections. Thus, targeting MLP layers ensures both robustness and experimental controllability when evaluating stochastic approximation.

\subsection{Stochastic Memory Masking}
\label{sec:memory}
While true memory approximation typically involves lossy compression of activations or quantization of key-value (KV) caches, we adopt a simplified proxy approach to study robustness. Specifically, we introduce stochastic masking in the outputs of attention layers, mimicking the effect of unreliable or lossy memory propagation. 

Formally, given attention activations $y$, a Bernoulli mask $M$ is sampled element-wise:
\begin{equation}
    \hat{y} = y \odot M, \quad M \sim \text{Bernoulli}(1-p),
\end{equation}
where $p$ is the masking probability. This resembles dropout applied during inference, but is interpreted as random activation loss rather than a training regularizer. Unlike pruning, which permanently removes weights, stochastic masking introduces random erasures at each forward pass. To integrate this into the benchmark, we wrap the forward pass of self-attention modules such that a fraction $p=0.05$ of the activations are masked:
\begin{verbatim}
if config.get("memory_approx") and hasattr(layer, "self_attn"):
    drop_p = 0.05
    def wrapped_forward(*args, orig=orig, drop_p=drop_p, **kw):
        out = orig(*args, **kw)
        mask = (torch.rand_like(out[0]) > drop_p)
        return (out[0] * mask,) + out[1:]
    layer.self_attn.forward = wrapped_forward
\end{verbatim}

This stochastic memory masking does not perform actual compression, but it provides a controlled way to approximate the behavior of lossy memory. By introducing transient activation loss, it allows us to probe the resilience of \gls{LLM}s under degraded memory fidelity.

\subsection{Combinations of Approximations}
\label{sec:combos}

A major contribution of this work lies in enabling composable approximation strategies.  
Rather than evaluating each method in isolation, the benchmark systematically explores combinations of techniques, mirroring real-world scenarios where efficiency, compression, and adaptability must be jointly optimized.  
By composing quantization, pruning, stochastic perturbations, memory masking, and \gls{LoRA}, we study both pairwise and higher-order interactions between methods.

\paragraph{Double Combinations.}
Pairwise combinations are used to test complementary effects between structural, arithmetic, and representational approximations:
\begin{itemize}
    \item \textbf{Quantization + \gls{LoRA}:}  
    The base model is quantized (INT8, INT4, or NF4) while the \gls{LoRA} adapters are applied.  
    This hybrid configuration maintains the adaptability of \gls{LoRA} while substantially reducing memory and storage requirements.
    \item \textbf{Pruning + \gls{LoRA}:}  
    Sparse pruned backbones are augmented with trainable \gls{LoRA} adapters, allowing the model to recover expressivity lost through weight removal.
    \item \textbf{Stochastic Perturbations + \gls{LoRA}:}  
    Perturbations are applied to the linear layers while \gls{LoRA} adapts to the induced stochasticity during training, testing robustness under noisy computation.
    \item \textbf{Memory Masking + \gls{LoRA}:}  
    Stochastic masking is applied to attention activations while \gls{LoRA} adapters fine-tune the frozen backbone, capturing adaptation under partial information degradation.
\end{itemize}

\paragraph{Triple Combinations.}
To investigate cumulative interactions, aggressive three-way configurations are explored:
\begin{itemize}
    \item \textbf{INT8 + Pruned + \gls{LoRA}:}  
    A jointly compressed setup where pruning enforces sparsity, INT8 quantization reduces memory bandwidth, and \gls{LoRA} fine-tunes the model for recovery.  
    \item \textbf{INT4 + Pruned + \gls{LoRA}:}  
    Represents the most extreme configuration, combining aggressive quantization and structural sparsity with minimal fine-tuning overhead.  
\end{itemize}

This compositional evaluation demonstrates how hybrid approximations can jointly optimize memory, latency, and performance, providing a unified framework for controlled degradation and recovery in \gls{LLM}s.


\paginavuota