\chapter{Related Works}
\label{chap:related}

The rapid expansion of \gls{LLM}s has sparked diverse efforts to improve efficiency in both training and inference. Approximate computing strategies play a central role, deliberately trading small amounts of accuracy for substantial reductions in computation, memory, and energy. This chapter surveys the theoretical foundations and recent advances of the techniques explored in this thesis: quantization, pruning, low-rank adaptation, stochastic perturbations, and stochastic memory masking. Each section reviews classical approaches, extensions to transformers, and connections to our own framework.

\section{Quantization}
\label{sec:quantization}

Quantization is one of the most effective approximation techniques for reducing the computational and memory footprint of neural networks. Instead of storing weights and activations in BF16 or FP32, quantization maps them to low-bit representations such as INT8 or INT4, which are directly supported on modern accelerators. This section reviews the theoretical foundation of quantization, following the integer-arithmetic formulation introduced by Jacob et al.~\cite{jacob2017quantizationtrainingneuralnetworks}, and describes recent advances tailored for \gls{LLM}s.

\subsection{Affine Quantization}
The central idea is to approximate a real value $r \in \mathbb{R}$ by an integer $q$ within a fixed range $[q_{\min}, q_{\max}]$ through an affine mapping:
\begin{equation}
    r \approx S (q - Z),
\end{equation}
where $S \in \mathbb{R}^+$ is the scale factor and $Z \in \mathbb{Z}$ is the zero-point. The zero-point ensures that zero in real space can be exactly represented in quantized space, which is important for avoiding bias shifts.

Given a real value $r$, its quantized integer representation is obtained as:
\begin{equation}
    q = \text{clip}\left( \text{round}\left(\frac{r}{S} + Z \right), q_{\min}, q_{\max} \right),
\end{equation}
where $\text{round}(\cdot)$ is nearest integer rounding and $\text{clip}(\cdot)$ ensures that $q$ remains within the target bit range.

The dequantization step maps $q$ back into floating-point:
\begin{equation}
    \hat{r} = S \cdot (q - Z).
    \label{eq:affine_mapping}
\end{equation}
This $\hat{r}$ is the quantized approximation of $r$~\cite{jacob2017quantizationtrainingneuralnetworks}.

\subsection{Integer-Only Inference}

A basic requirement for efficient deployment of quantized neural networks is that all arithmetic during inference can be implemented using integer only operations on quantized values. For example, for the 8-bit quantization, $q \in \{0, \dots, 255\}$ (unsigned) or $q \in \{-128, \dots, 127\}$ (signed). Biases are typically quantized to 32-bit integers to avoid overflow. This mapping ensures that zero-padding operations in convolution or matrix multiplication can be implemented exactly, since the zero-point $Z$ guarantees a quantized representation of $r=0$.

\vspace{1em}
\noindent\textbf{Integer-Only Matrix Multiplication.}
Consider the multiplication of two square matrices of size $N \times N$:
\begin{equation}
    r_3 = r_1 r_2,
\end{equation}
where $r_1, r_2, r_3 \in \mathbb{R}^{N \times N}$.

Let $(S_m, Z_m)$ denote the quantization parameters for matrix $r_m$, with
quantized entries $q^{(m)}_{ij}$. Applying Eq.~\eqref{eq:affine_mapping}:
\begin{equation}
    r^{(m)}_{ij} = S_m \big(q^{(m)}_{ij} - Z_m\big), \qquad m \in \{1,2,3\}.
    \label{eq:quantized_entry}
\end{equation}

By definition of matrix multiplication:
\begin{equation}
    r^{(3)}_{ik} = \sum_{j=1}^N r^{(1)}_{ij} \cdot r^{(2)}_{jk}.
\end{equation}

Substituting Eq.~\eqref{eq:quantized_entry}:
\begin{equation}
    S_3 \big(q^{(3)}_{ik} - Z_3\big) = 
    \sum_{j=1}^N S_1 \big(q^{(1)}_{ij} - Z_1\big) \cdot S_2 \big(q^{(2)}_{jk} - Z_2\big).
    \label{eq:matmul_quantized}
\end{equation}

Factoring out the scales:
\begin{equation}
    S_3 \big(q^{(3)}_{ik} - Z_3\big) =
    S_1 S_2 \sum_{j=1}^N \big(q^{(1)}_{ij} - Z_1\big)\big(q^{(2)}_{jk} - Z_2\big).
\end{equation}

Rearranging, the integer-only update rule becomes:
\begin{equation}
    q^{(3)}_{ik} = Z_3 + M \cdot 
    \sum_{j=1}^N \big(q^{(1)}_{ij} - Z_1\big)\big(q^{(2)}_{jk} - Z_2\big),
    \label{eq:quantized_matmul_final}
\end{equation}
where the multiplier $M$ is defined as:
\begin{equation}
    M = \frac{S_1 S_2}{S_3}.
    \label{eq:multiplier}
\end{equation}

\noindent\textbf{Fixed-Point Multiplier Representation.}
The only non-integer term in Eq.~\eqref{eq:quantized_matmul_final} is the constant $M$.
Jacob et al.\cite{jacob2017quantizationtrainingneuralnetworks} show that $M \in (0,1)$ in practice, and it can be expressed as:
\begin{equation}
    M = 2^{-n} M_0, \quad M_0 \in [0.5, 1), \; n \in \mathbb{Z}_{\geq 0}.
    \label{eq:multiplier_decomp}
\end{equation}

Here:
\begin{itemize}
    \item $M_0$ is stored as a fixed-point integer (e.g., 32-bit). Since $M_0 \geq 0.5$, it always has at least 30 bits of relative accuracy when represented in 32-bit fixed point.
    \item $2^{-n}$ is implemented as an efficient \texttt{bit-shift}, ensuring integer-only hardware execution.
\end{itemize}

Thus, multiplication by $M$ is implemented as:
\begin{equation}
    \text{int32\_accum} \cdot M \;\;\approx\;\; 
    \Big(\text{int32\_accum} \cdot M_0^{(\text{int})}\Big) \gg n,
\end{equation}
where $\gg n$ denotes bit-shift by $n$ bits.


\begin{itemize}
    \item The inner sum in Eq.~\eqref{eq:quantized_matmul_final} involves only integer additions and multiplications.
    \item The multiplier $M$ is a single precomputed constant, applied with fixed-point arithmetic and shifts.
    \item This makes inference fully integer-only, with scales and zero-points applied as lightweight rescaling operations.
    \item Bias terms are usually quantized to INT32, ensuring no loss of accuracy in accumulation.
\end{itemize}

This affine quantization scheme underpins integer-only inference in TensorFlow Lite,
ONNX Runtime, and PyTorch QNNPACK, and remains the basis for modern \gls{PTQ} and \gls{QAT} pipelines~\cite{frantar2023gptqaccurateposttrainingquantization}.

\subsection{Post-Training Quantization, Quantization-Aware Training, and Zero-Shot Quantization}

Quantization often requires model adaptation after weights are mapped to low-precision. 
This can be achieved either by fine-tuning the model (\gls{QAT}), or directly quantizing a pretrained model without retraining (\gls{PTQ}). 
A more recent category, \gls{ZSQ}, eliminates the need for training data altogether, using either analytical approximations or synthetic data generation. 

\paragraph{Quantization-Aware Training (\gls{QAT}).}  
Given a pretrained model, quantization introduces perturbations to parameters, potentially shifting the model away from the local optimum found during full-precision training. 
To address this, \gls{QAT} retrains the model with quantization simulated in both the forward and backward passes. 
Specifically, weights are updated in floating point, but after each gradient update they are projected back into the quantized space as in Figure~\ref{fig:QAT}.
\begin{equation}
    W \leftarrow \text{Quantize}(W - \eta \nabla L),
\end{equation}
where $\eta$ is the learning rate. 

\begin{figure}
    \centering
    \includegraphics[width=0.5\linewidth]{images/QAT.png}
    \caption{Illustration of Quantization-Aware Training procedure~\cite{gholami2021surveyquantizationmethodsefficient}.}
    \label{fig:QAT}
\end{figure}

A central difficulty in \gls{QAT} is that the quantization operator is non-differentiable. 
For example, rounding is piecewise constant and its derivative is zero almost everywhere. 
To circumvent this, \gls{QAT} uses the Straight-Through Estimator (STE)~\cite{bengio2013estimatingpropagatinggradientsstochastic}, which approximates the gradient of quantization as the identity function:
\begin{equation}
    \frac{\partial \, \text{round}(x)}{\partial x} \approx 1.
\end{equation}
This coarse approximation nonetheless works well in practice, as it correlates with the true population gradient in expectation~\cite{gholami2021surveyquantizationmethodsefficient}. 
Figure~\ref{fig:qat_ptq} illustrates the difference between \gls{PTQ} and \gls{QAT}.

Several refinements to \gls{QAT} have been proposed. PACT~\cite{choi2018pactparameterizedclippingactivation} learns activation clipping ranges as part of training, LSQ~\cite{esser2020learnedstepsizequantization} learns scale factors through gradient descent, and AdaRound~\cite{nagel2021whitepaperneuralnetwork} adapts rounding boundaries instead of using naive round-to-nearest. 
Despite these advances, \gls{QAT} remains computationally expensive, as it often requires retraining for hundreds of epochs to recover accuracy, particularly at ultra-low precision (e.g., 2--4 bits).

\paragraph{Post-Training Quantization (\gls{PTQ}).}  
In \gls{PTQ}, pretrained weights are quantized directly without additional retraining:
\begin{equation}
    \hat{W} = \text{clip}\left(\text{round}\left(\frac{W}{S}\right), -q_{\max}, q_{\max}\right) \cdot S,
\end{equation}
where $S$ is the scaling factor.  
\gls{PTQ} is computationally efficient and requires no labeled training data, making it attractive for deployment. 
However, it often suffers from accuracy degradation at low bit-widths, especially below INT8.

To mitigate this, several improvements have been proposed:
\begin{itemize}
    \item \textbf{Bias correction:} Adjusting mean and variance shifts induced by quantization~\cite{banner2019posttraining4bitquantizationconvolution}.
    \item \textbf{Range equalization:} Aligning weight ranges across layers to reduce scale mismatch~\cite{nagel2021whitepaperneuralnetwork}.
    \item \textbf{Analytical Clipping (ACIQ):} Computing optimal clipping thresholds using an assumed Gaussian or Laplace distribution of activations~\cite{choukroun2019lowbitquantizationneuralnetworks}.
    \item \textbf{Outlier Channel Splitting (OCS):} Duplicating channels with extreme values to reduce quantization error~\cite{zhao2019improvingneuralnetworkquantization}.
    \item \textbf{Adaptive rounding (AdaRound):} Learning rounding decisions to minimize quantization error~\cite{nagel2021whitepaperneuralnetwork}.
\end{itemize}

These refinements enable \gls{PTQ} to approach \gls{QAT}-level accuracy in some cases, particularly at 8-bit and 6-bit precision, but \gls{PTQ} still struggles at ultra-low bit-widths (e.g., INT4) without retraining~\cite{frantar2023gptqaccurateposttrainingquantization}.

\paragraph{Zero-Shot Quantization (\gls{ZSQ}).}  
In many practical scenarios, access to the original training data is impossible due to privacy, proprietary constraints, or size limitations (e.g., large web-scale datasets). 
\gls{ZSQ} methods address this by quantizing models without access to training data.  

There are two main levels:
\begin{enumerate}
    \item \textbf{\gls{ZSQ}+\gls{PTQ}:} No data, no fine-tuning. Quantization parameters are chosen analytically or with heuristics (e.g., bias correction, equalization).
    \item \textbf{\gls{ZSQ}+\gls{QAT}:} No real data, but fine-tuning is performed on synthetic data.
\end{enumerate}

A popular branch of research generates synthetic calibration data. 
Early work used GANs to synthesize samples that approximate the model’s decision boundaries. 
More advanced methods (e.g., ZeroQ, GDFQ) directly optimize synthetic inputs such that the internal statistics (e.g., BatchNorm means and variances) match those observed in the pretrained model:
\begin{equation}
    \min_x D_{\text{KL}}\big(\mu_{\text{BN}}(x), \mu_{\text{real}}\big) + D_{\text{KL}}\big(\sigma_{\text{BN}}^2(x), \sigma_{\text{real}}^2\big).
\end{equation}

These synthetic inputs are then used to calibrate quantization scales or even fine-tune the quantized model via knowledge distillation from the full-precision teacher. 
Recent work also integrates generative models to produce higher-fidelity calibration data~\cite{gholami2021surveyquantizationmethodsefficient}.

\paragraph{Summary.}  
\begin{itemize}
    \item \textbf{\gls{QAT}} achieves the best accuracy, especially at 4-bit or below, but requires expensive retraining.  
    \item \textbf{\gls{PTQ}} is fast and data-free, suitable for deployment when INT8 suffices, but often suffers at ultra-low precision.  
    \item \textbf{\gls{ZSQ}} extends \gls{PTQ}/\gls{QAT} to data-free scenarios, relying on synthetic data or analytical approximations.  
\end{itemize}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.75\linewidth]{images/QAT-PTQ.png}
    \caption{Comparison between \gls{QAT} (left, requires retraining) and \gls{PTQ} (right, data-free)~\cite{gholami2021surveyquantizationmethodsefficient}.}
    \label{fig:qat_ptq}
\end{figure}

\subsection{Simulated Quantization (Fake Quantization)}
\label{sec:simulated_quantization_theory}

Simulated quantization, also known as fake quantization, is a technique for emulating low-precision arithmetic within a floating-point computational framework. 
Unlike integer-only quantization, which performs all multiplications and accumulations using fixed-point arithmetic on quantized tensors, simulated quantization preserves floating-point computation but introduces quantization and dequantization operations during the forward pass to replicate the precision loss of integer quantization~\cite{gholami2021surveyquantizationmethodsefficient, jacob2017quantizationtrainingneuralnetworks}. 
This approach is widely used in quantization-aware training (\gls{QAT}) and mixed-precision fine-tuning to enable gradient-based optimization of quantized models without requiring hardware support for low-bit arithmetic.

\paragraph{Mathematical Formulation.}
Given a real-valued tensor $r \in \mathbb{R}^n$, quantization is characterized by a scale factor $S$ and discrete quantization levels $Q = \{q_{\min}, \ldots, q_{\max}\}$. 
In integer quantization, the quantized tensor $q$ is represented and stored in integer form. 
In contrast, simulated quantization applies both quantization and dequantization within the floating-point domain:
\begin{equation}
    \hat{r} = \text{DeQuantize}\!\left(\text{Quantize}(r)\right)
    = S \cdot \text{clip}\!\left( \text{round}\!\left( \frac{r}{S} \right), q_{\min}, q_{\max} \right),
    \label{eq:fake_quant}
\end{equation}
where $\hat{r}$ is the quantized approximation of $r$ in floating-point format. 
This operation simulates the effect of integer quantization by injecting discretization noise while maintaining differentiability of the computational graph.

\paragraph{Forward and Backward Pass.}
During the forward pass, rounding introduces a quantization error
\begin{equation}
    \varepsilon_q = \hat{r} - r,
\end{equation}
which models the information loss due to finite precision. 
However, since the quantization function is non-differentiable, simulated quantization relies on the \textit{Straight-Through Estimator} (STE)~\cite{bengio2013estimatingpropagatinggradientsstochastic} to propagate gradients during backpropagation:
\begin{equation}
    \frac{\partial \, \hat{r}}{\partial r} \approx 1.
\end{equation}
This approximation treats the quantization function as an identity mapping in the backward pass, allowing gradient flow to proceed as if the quantization step were continuous. 
Although simplistic, the STE has been shown to effectively align the training dynamics of quantized models with their full-precision counterparts~\cite{gholami2021surveyquantizationmethodsefficient}.

\begin{figure}
    \centering
    \includegraphics[width=0.5\linewidth]{images/FP-INT-Comparison.png}
    \caption{Comparison between full-precision inference (Left), simulated quantization
with float operations (Middle), and integer-only quantization with fixed-point
arithmetic (Right) [4].}
    \label{fig:SimQuant}
\end{figure}

\paragraph{Computation Model.}
The computational structure of simulated quantization, depicted in Figure~\ref{fig:SimQuant} (Middle), follows a hybrid FP32 workflow:
\begin{enumerate}
    \item Weights $W$ are quantized once to $W_q$ and dequantized to FP32 ($W_d$) for computation.
    \item Activations $x$ are quantized dynamically at runtime to $x_q$ and dequantized to $x_d$.
    \item Matrix multiplication and accumulation are performed in full precision:
    \begin{equation}
        y = \text{Linear}(x_d, W_d) + b.
    \end{equation}
    \item The outputs are then re-quantized to produce low-precision activations for the next layer.
\end{enumerate}
This sequence ensures that each layer experiences quantization-induced noise consistent with its low-bit equivalent, even though the arithmetic remains FP32.

\paragraph{Advantages and Limitations.}
Simulated quantization provides several theoretical and practical benefits:
\begin{itemize}
    \item \textbf{Hardware independence:} It can be performed on any floating-point hardware, without integer matrix-multiplication units.
    \item \textbf{Differentiability:} Through STE, it enables gradient-based optimization of quantized networks.
    \item \textbf{Faithful quantization noise modeling:} It accurately emulates rounding and clipping errors seen in integer arithmetic.
    \item \textbf{Layer-wise flexibility:} It supports a mixture of static (for weights) and dynamic (for activations) quantization strategies.
\end{itemize}

However, simulated quantization does not yield real latency or energy savings during training or inference, since all computations still occur in floating-point. 
It is primarily an analytical tool for approximating quantization behavior, useful for model fine-tuning, mixed-precision training, and benchmarking~\cite{gholami2021surveyquantizationmethodsefficient, frantar2023gptqaccurateposttrainingquantization}.

\paragraph{Comparison with Integer-Only Inference.}
The difference between simulated and integer-only quantization can be summarized as follows:
\begin{align}
    \textbf{Integer-only:} &\quad y_q = \text{ReQuantize}\!\left(\sum_j (x_q - Z_x)(w_q - Z_w)\right), \\
    \textbf{Simulated:} &\quad y = D(x_q)^\top D(w_q),
\end{align}
where $D(\cdot)$ denotes dequantization to FP32. 
Thus, while integer-only inference executes integer multiplications and accumulations on hardware accelerators, simulated quantization performs identical arithmetic in expectation but remains entirely in the FP32 domain.

\paragraph{Implementation in this Thesis.}
Simulated quantization provides a theoretically grounded bridge between low-bit arithmetic and BF16 computation. 
It allows analysis of quantization noise, gradient behavior, and numerical stability under controlled conditions. 
For this reason, it is the standard paradigm in quantization-aware training frameworks~\cite{gholami2021surveyquantizationmethodsefficient, jacob2017quantizationtrainingneuralnetworks}, as well as the basis of the approximation strategy employed in this thesis.

\subsection{Advanced Quantization Formats for LLMs}
\label{sec:advanced_quantization}

Scaling large language models has motivated the development of advanced quantization schemes that go beyond standard INT8. These methods enable sub-8-bit precision, improve hardware efficiency, and reduce the accuracy gap traditionally observed at ultra-low precision.

\paragraph{Simulated vs. Integer-Only Quantization.}
There are two primary modes of deploying quantized models. In simulated quantization (also called fake quantization), weights and activations are stored in reduced precision, but computations such as matrix multiplications are carried out in floating-point. This requires dequantization before each operation and limits hardware efficiency. In contrast, integer-only quantization (fixed-point quantization) performs all operations using integer arithmetic, eliminating floating-point overheads and enabling direct use of low-precision logic on accelerators~\cite{jacob2017quantizationtrainingneuralnetworks}. Integer-only quantization can be further enhanced with dyadic quantization, where all scaling factors are constrained to dyadic numbers (i.e., rational values with powers of two in the denominator). This makes scaling efficient via integer multiplications and bit-shifts.

\paragraph{Mixed-Precision Quantization.}
Uniformly quantizing an entire model to a fixed bit-width (e.g., INT4) can cause severe accuracy degradation. Mixed-precision quantization alleviates this problem by assigning higher precision (e.g., INT8) to sensitive layers and lower precision (e.g., INT4) to less sensitive ones. The sensitivity of layers can be determined by second-order information such as the trace of the Hessian, as in HAWQ~\cite{dong2019hawqv2hessianawaretraceweighted}, or by reinforcement learning and differentiable NAS approaches. For LLMs, mixed-precision has been shown to deliver up to 50\% inference speedup on GPUs while minimizing accuracy loss.

\paragraph{Hardware-Aware Quantization.}
Not all quantization configurations map equally well to hardware. For example, some accelerators support INT8 efficiently but lack optimized kernels for INT6 or INT2. Hardware-aware quantization incorporates latency and energy models of the target device when selecting bitwidths, ensuring that accuracy–efficiency trade-offs translate into real-world gains. This approach is increasingly important for deploying LLMs in production environments, where GPU memory bandwidth and cache hierarchies dominate performance~\cite{jacob2017quantizationtrainingneuralnetworks}.

\paragraph{Distillation-Assisted Quantization.}
Knowledge distillation can help quantized models recover accuracy lost at low precision. A high-precision teacher supervises the low-precision student by matching soft probability distributions or intermediate features. The combined loss is:
\begin{equation}
    \mathcal{L} = \alpha \, H(y, \sigma(z_s)) + \beta \, H(\sigma(z_t, T), \sigma(z_s, T)),
\end{equation}
where $z_s$ and $z_t$ are logits of the student and teacher, $T$ is the distillation temperature, and $H$ is cross-entropy~\cite{hinton2015distillingknowledgeneuralnetwork}. Distillation has been shown to substantially improve quantization accuracy in LLMs, particularly below 8 bits.

\paragraph{Extreme Quantization.}
Binarization (1-bit) and ternarization (2-bit) represent the most aggressive forms of quantization, reducing model size by up to $32\times$ compared to FP32. While early methods such as BinaryConnect and XNOR-Net demonstrated the feasibility of bit-wise operations, applying extreme quantization to LLMs remains challenging due to severe accuracy loss. Recent advances mitigate this degradation using scaling factors, wider architectures, and better gradient approximations~\cite{chen2023gradientbasedbileveloptimizationdeep}. Despite the challenges, extreme quantization is an active research direction for bringing transformer-based models to edge devices.

\paragraph{Vector Quantization.}
Inspired by classical signal processing, vector quantization clusters groups of weights and replaces them with centroids from a learned codebook:
\begin{equation}
    \min_{c_1,\ldots,c_k} \sum_i \| w_i - c_j \|^2, \quad w_i \in \text{cluster } j.
\end{equation}
This approach achieves up to $8\times$ compression with limited accuracy loss~\cite{han2016deepcompressioncompressingdeep}. Extensions such as product quantization apply clustering to submatrices and recursively quantize residuals. For LLMs, vector quantization can be combined with pruning and Huffman coding for further compression.

\paragraph{Implementation in this Thesis.}
In this work, all quantization schemes (INT8, INT4, and NF4) are implemented as GPU-based simulated quantizers integrated into the model’s fine-tuning pipeline. 
The implementation performs quantization and dequantization directly on the GPU to avoid CPU bottlenecks and excessive memory transfers. 
For uniform quantization (INT8/INT4), tensors are quantized symmetrically per tensor using the function \texttt{quantize\_tensor\_uniform()}, which computes the scale parameter $s$ as:
\begin{equation}
    s = \frac{\max |W|}{q_{\max}},
\end{equation}
where $q_{\max}$ is $127$ for INT8 or $7$ for INT4. 
Each tensor element is then mapped to its nearest quantized level using rounding and clamping operations, followed by dequantization via \texttt{dequantize\_tensor()} to restore floating-point form for computation. 
This procedure preserves the model’s numerical behavior while emulating quantization effects during training and evaluation.

For NF4 quantization, the method follows the NormalFloat-4 specification introduced by Dettmers et al.~\cite{dettmers2023qloraefficientfinetuningquantized}, but is reimplemented to operate in a fully GPU-streamed manner, and because of our GPU memory limitations. 
The quantization function \texttt{quantize\_tensor\_nf4()} divides each tensor into contiguous blocks of 64 elements, computes per-block scaling factors, and performs nearest-neighbor assignment to the fixed 16-level NF4 codebook:
\begin{equation}
\mathcal{C}_{\text{NF4}} =
\begin{aligned}[t]
\{& -1.0000, -0.6962, -0.5251, -0.3949, \\
  & -0.2844, -0.1848, -0.0911, 0.0000, \\
  &     0.0911, 0.1848, 0.2844, 0.3949, \\
  &     0.5251, 0.6962, 0.8682, 1.0000 \}.
\end{aligned}
\label{eq:nf4_codebook_final}
\end{equation}

Quantization is computed per element by iterating over the 16 codebook entries, which avoids forming large 4D broadcast tensors that could lead to GPU memory exhaustion. 
The corresponding \texttt{dequantize\_tensor\_nf4()} function restores BF16 weights by mapping each quantized index back to its codebook value and rescaling with the stored blockwise scale.

Unlike uniform quantization, which uses a linear mapping between real and integer domains, NF4 performs nonuniform quantization with higher resolution near zero, aligning with the Gaussian distribution of transformer weights. 
In all cases, quantized weights are computed once (static quantization), while activations are quantized dynamically during each forward pass, followed by BF16 computation and re-quantization of outputs. 
This ensures that each layer experiences quantization-induced effects without compromising training stability.

Overall, these quantization operators provide an efficient, memory-safe, and hardware-agnostic foundation for evaluating low-bit approximation in large language models. 
They combine realistic quantization error modeling with the flexibility of BF16 computation, enabling consistent benchmarking across INT8, INT4, and NF4 configurations.

    \section{Pruning}
    \label{sec:pruning}
    
    Pruning is a classical model compression technique that reduces the number of parameters in a neural network by removing redundant or less important connections. The motivation is rooted in the observation that overparameterized models contain significant redundancy, and that a sparse subnetwork can often achieve comparable accuracy to the dense model. Early work such as Optimal Brain Damage~\cite{NIPS1989_6c9882bb} introduced second-order methods for pruning, while Deep Compression~\cite{han2016deepcompressioncompressingdeep} popularized magnitude-based pruning, showing that large neural networks can be reduced by up to an order of magnitude in size without significant accuracy loss.
    
    \subsection{Unstructured Pruning}
    Unstructured pruning removes individual weights based on a saliency criterion, most commonly weight magnitude. A typical rule is:
    \begin{equation}
    \hat{W}_{ij} = 
    \begin{cases}
    W_{ij}, & |W_{ij}| > \tau, \\
    0, & \text{otherwise},
    \end{cases}
    \end{equation}
    where $\tau$ is a threshold chosen to achieve the desired sparsity level.  
    This approach is highly flexible and can achieve very high sparsity ratios~\cite{han2016deepcompressioncompressingdeep}. However, because the resulting sparsity pattern is irregular, specialized hardware kernels are often required to realize actual inference-time speedups.
    
    \subsection{Structured Pruning}
    Structured pruning instead removes entire groups of weights, such as neurons, channels, or attention heads. This preserves dense tensor structures and is therefore more compatible with modern hardware accelerators. For transformers, structured pruning has been applied in several forms:
    \begin{itemize}
        \item \textbf{Attention head pruning} selectively removes full heads, leveraging the redundancy across multi-head attention~\cite{michel2019sixteenheadsreallybetter}.
        \item \textbf{Feedforward pruning} eliminates full hidden neurons in the MLP layers~\cite{shim2021layerwisepruningtransformerattention}.
    \end{itemize}
    Structured pruning is less flexible in terms of sparsity ratios compared to unstructured pruning, but has the advantage of translating directly into inference-time efficiency gains without specialized kernels.
    
    \subsection{Pruning and the Lottery Ticket Hypothesis}
    The Lottery Ticket Hypothesis~\cite{frankle2019lotterytickethypothesisfinding} further strengthened the theoretical foundation of pruning by showing that large networks contain ``winning subnetworks'' sparse subnetworks that, when trained in isolation with the same initialization, can match the performance of the full dense model. This observation connects pruning to broader discussions on overparameterization and generalization in deep learning.
    
    \paragraph{Implementation in this Thesis.}
    In this thesis, \texttt{L1-unstructured pruning} is implemented within the transformer feedforward layers. Specifically, pruning is applied to the gate projection \texttt{gate\_proj} inside each MLP block. Following the \texttt{torch.nn.utils.prune} API, we prune 5\% of the smallest-magnitude weights using L1-norm saliency:
    \begin{equation}
    \text{prune}(W) = W \odot M, \quad M_{ij} = \mathbf{1}(|W_{ij}| > \tau).
    \end{equation}
    This approach creates a binary pruning mask that zeroes out low-magnitude weights while preserving higher-magnitude connections. The method corresponds to magnitude-based unstructured pruning as in Deep Compression~\cite{han2016deepcompressioncompressingdeep}. Although this does not yield direct inference-time speedups on commodity hardware, it provides a lightweight yet principled approximation method that integrates seamlessly into the unified benchmarking framework developed in this thesis.

\begin{figure}
    \centering
    \includegraphics[width=0.5\linewidth]{images/Pruning.png}
    \caption{Synapses and neurons before and after
pruning~\cite{article_prun}.}
    \label{fig:placeholder}
\end{figure}

    \section{Low-Rank Adaptation (\gls{LoRA})}
    \label{sec:lora}
    
    \gls{LoRA} is a parameter-efficient fine-tuning (\gls{PEFT}) method that freezes the pretrained weights and learns task-specific, low-rank updates alongside them. Consider a pretrained weight $W_0 \in \mathbb{R}^{d \times k}$. \gls{LoRA} reparametrizes the adaptation as a rank-$r$ matrix product,
    \begin{equation}
    W \;=\; W_0 + \Delta W \;=\; W_0 + \frac{\alpha}{r}\,B A,
    \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},
    \label{eq:lora-param}
    \end{equation}
    where only $A,B$ are trainable, $B$ is typically initialized to zero and $A$ to a small Gaussian so that $\Delta W$ starts at zero; $\alpha$ is a scaling hyperparameter that stabilizes training across different $r$~\cite{hu2021loralowrankadaptationlarge}. The forward pass is then
    \begin{equation}
    h \;=\; W_0 x \;+\; \frac{\alpha}{r}\, B A x,
    \end{equation}
    i.e., a residual low-rank correction on top of the frozen base. In transformers, \gls{LoRA} is commonly applied to the attention projections $W_q, W_k, W_v, W_o$, while keeping MLP blocks frozen for maximal parameter efficiency~\cite{hu2021loralowrankadaptationlarge}. Inference can be latency-free: at serve time one may merge $BA$ into $W_0$ ($W \leftarrow W_0 + BA$), so the compute graph is identical to a dense model~\cite{hu2021loralowrankadaptationlarge}. 

    Recent theory analyzes when low-rank adapters suffice to match a target model exactly and how approximation error scales with rank. For fully connected networks, under mild non-singularity assumptions, a frozen model augmented with rank-$R$ adapters can exactly realize a smaller target model once $R$ exceeds a layerwise threshold; below threshold, the best-achievable error relates to the $(RL{+}1)$-st singular value of a certain layerwise “gap” matrix (extending Eckart–Young–Mirsky to products of low-rank-updated matrices)~\cite{zeng2024expressivepowerlowrankadaptation}. For transformers, the analysis shows that, again under mild conditions, rank on the order of half the embedding size is sufficient to exactly match a target transformer of the same size, giving concrete guidance on adapter rank selection~\cite{zeng2024expressivepowerlowrankadaptation}. 
    
    \paragraph{Practical efficiency.}
    Relative to full fine-tuning, \gls{LoRA} reduces optimizer state and gradient traffic for frozen parameters, cutting training VRAM and storage substantially. The original report shows up to $\sim\!3\times$ lower GPU memory during training and $\sim\!10{,}000\times$ fewer trainable parameters at rank $r{=}4$ when adapting only $W_q$/$W_v$ (e.g., GPT-3 175B: 1.2\,TB $\to$ 350\,GB training memory; checkpoints $\sim$350\,GB $\to$ 35\,MB for adapters) while introducing no extra inference latency when merged~\cite{hu2021loralowrankadaptationlarge}.
    
    \paragraph{Scalability.}
    Because adapters are tiny and swappable, serving multiple fine-tuned behaviors on a single shared base becomes practical. An open-source Multi-\gls{LoRA} server (LoRAX) demonstrates dynamic adapter loading, multi-adapter batching, and tiered caching to host many \gls{LoRA} models concurrently on one GPU with robust time-to-first-token and throughput under load. In the \gls{LoRA} Land deployment, 25 LoRA-adapted Mistral-7B models were served to thousands of users on a single 80\,GB A100, with detailed latency benchmarks reported~\cite{zhao2024loraland310finetuned}. 
    
    \paragraph{Empirical effectiveness.}
    A large-scale study across 10 base models and 31 tasks (310 \gls{LoRA} fine-tunes) reports that \gls{LoRA} adapters trained on 4-bit-quantized bases ``QLoRA-style'' consistently improve task performance; on average, 4-bit \gls{LoRA} models exceed their untuned bases and, on that task suite, even surpass GPT-4 under the study’s prompting and scoring setup, while enabling economical multi-adapter serving~\cite{zhao2024loraland310finetuned}. On the other hand, the authors also note limitations in prompt engineering, evaluation scope, and model variety, encouraging replication~\cite{zhao2024loraland310finetuned}. 
    
    \paragraph{Connections to other \gls{PEFT} variants.}
    \gls{LoRA} is compatible with other  \gls{PEFT} ideas: SVD-pruned updates and decomposition variants (e.g., DoRA) target improved conditioning; QLoRA combines \gls{LoRA} with 4-bit NF4 and paged optimizers to cut memory further; in practice, many works apply \gls{LoRA} primarily to attention projections for the best quality–efficiency trade-off~\cite{hu2021loralowrankadaptationlarge,zeng2024expressivepowerlowrankadaptation,zhao2024loraland310finetuned}. 
    
    \paragraph{Implementation in this Thesis.}
    In this work, \gls{LoRA} is applied to the self-attention layers of the model following the configuration used in~\cite{hu2021loralowrankadaptationlarge}. 
    Specifically, \gls{LoRA} is integrated into the query and value projection matrices (\texttt{q\_proj} and \texttt{v\_proj}) of the transformer attention mechanism, while all other parameters remain frozen. 
    Each modified projection layer is decomposed into a pair of trainable low-rank matrices $(A, B)$, where $A \in \mathbb{R}^{r \times d}$ and $B \in \mathbb{R}^{d \times r}$, initialized such that $BA \approx 0$ at the start of fine-tuning:
    \begin{equation}
        W' = W + \Delta W, \qquad \Delta W = B A,
    \end{equation}
    where $W$ denotes the frozen pretrained weight and $\Delta W$ the learned low-rank update. In this implementation, the \gls{LoRA} configuration is defined by the tuple $(r, \alpha, p)$, corresponding respectively to the rank, scaling factor, and dropout probability. 
    The adopted parameters are $r = 8$, $\alpha = 16$, and dropout $p = 0.05$, as shown in the code excerpt below:
    \begin{verbatim}
    lora_cfg = LoraConfig(
        r=8, lora_alpha=16, lora_dropout=0.05, bias="none",
        target_modules=["q_proj", "v_proj"],
        task_type=TaskType.CAUSAL_LM)
    model = get_peft_model(model, lora_cfg)
    model.print_trainable_parameters()
    \end{verbatim}
    
    The \gls{LoRA} scaling factor $\alpha$ is set proportionally to the rank ($\alpha = 2r$), which stabilizes the magnitude of the low-rank updates during training. 
    This design ensures that only a small subset of parameters (the low-rank adapters) are trainable, while the vast majority of the model remains frozen, leading to efficient fine-tuning with minimal computational and memory overhead.

\section{Stochastic Perturbations}
\label{sec:stochastic}

\gls{LLM}s often produce overconfident predictions, even when they are wrong, a phenomenon commonly linked to hallucinations~\cite{zhang2024alleviatinghallucinationslargelanguage}. To address this, uncertainty quantification techniques aim to measure the reliability of model outputs. While traditional sampling methods target aleatoric uncertainty (inherent data variability), they often fail to capture epistemic uncertainty, which stems from the model’s limited knowledge or sensitivity to perturbations~\cite{H_llermeier_2021}. 

Stochastic perturbations provide a lightweight means to approximate epistemic uncertainty by injecting controlled randomness into model inputs or computations. The SPUQ framework~\cite{gao2024spuqperturbationbaseduncertaintyquantification} formalizes this by generating perturbed inputs $(T_i, x_i)$ from an original input $(T_0, x_0)$, where $T$ denotes temperature and $x$ the prompt:
\begin{equation}
    \{(T_i, x_i)\}_{i=1}^k = \mathcal{P}(T_0, x_0),
\end{equation}
with $\mathcal{P}$ denoting the perturbation module (e.g., paraphrasing, dumour tokens, system message variations). The perturbed inputs are fed into the \gls{LLM}s to yield predictions $\{y_i\}_{i=0}^k$, where $i=0$ corresponds to the unperturbed input.

\subsection{Perturbation-Driven Variance}

For additive noise perturbations, as used in our implementation, inputs are modeled as:
\begin{equation}
    \tilde{x} = x + \epsilon, \qquad \epsilon \sim \mathcal{N}(0, \sigma^2 I).
\end{equation}
The perturbed output of a linear layer becomes:
\begin{equation}
    \tilde{y} = \tilde{x} W^\top + b = (x + \epsilon) W^\top + b.
\end{equation}
The expectation is unbiased:
\begin{equation}
    \mathbb{E}[\tilde{y}] = x W^\top + b,
\end{equation}
but the variance encodes sensitivity to perturbations:
\begin{equation}
    \mathrm{Var}[\tilde{y}_j] = \sigma^2 \|W_{:,j}\|_2^2.
\end{equation}
Hence, columns of $W$ with larger norms amplify epistemic uncertainty.

\subsection{Uncertainty Aggregation}

SPUQ generalizes stochastic perturbations by combining them with sampling-based uncertainty. Confidence is derived by comparing outputs $\{y_i\}$ across perturbed and unperturbed prompts. Two aggregation methods are proposed~\cite{gao2024spuqperturbationbaseduncertaintyquantification}:

\paragraph{Inter-sample aggregation.}
Confidence for output $y_j$ is computed via textual similarity with other samples:
\begin{equation}
    c_{\text{inter}}(y_j) = 
    \frac{\sum_{i=0,i \neq j}^k s(y_j,y_i) \, w_i}{\sum_{i=0,i \neq j}^k w_i},
\end{equation}
where $s(\cdot,\cdot)$ is a similarity metric (e.g., RougeL, BERTScore), and $w_i$ weighs perturbations based on their closeness to the original prompt.

\paragraph{Intra-sample aggregation.}
Confidence is averaged across sample-wise uncertainty estimates:
\begin{equation}
    c_{\text{intra}} = \frac{1}{k+1}\sum_{i=0}^k c(x_i,y_i),
\end{equation}
where $c(x_i,y_i)$ may be derived from likelihood, perplexity, or even self-verbalized uncertainty.

Noise injection complements sampling (aleatoric) by revealing instability across perturbed inputs (epistemic). SPUQ reduces Expected Calibration Error (ECE) by up to 50\% across \gls{LLM}s. Repeated stochastic forward passes provide estimates of mean and variance (Monte Carlo Estimation):
    \begin{align}
        \hat{\mu}(x) &= \frac{1}{M} \sum_{m=1}^M \tilde{y}^{(m)}, \\
        \hat{\sigma}^2(x) &= \frac{1}{M} \sum_{m=1}^M \left(\tilde{y}^{(m)} - \hat{\mu}(x)\right)^2,
    \end{align}
    enabling epistemic uncertainty quantification.
\end{itemize}

\subsection{Stochastic Regularization Methods}

The idea of injecting noise into neural networks has a long history as a form of stochastic regularization. 
Rather than perturbing prompts as in SPUQ, our approach is closer to activation-level noise injection, which has been widely explored in deep learning for both efficiency and robustness.

\paragraph{Connection to Dropout.} 
Dropout~\cite{JMLR:v15:srivastava14a} is perhaps the most well-known stochastic regularization method, where a random binary mask $m \sim \text{Bernoulli}(p)$ is applied to activations:
\begin{equation}
    \tilde{x} = m \odot x,
\end{equation}
forcing the network to learn redundant representations. 
Our Gaussian perturbation variant can be viewed as a smoother alternative, replacing discrete masking with continuous noise:
\begin{equation}
    \tilde{x} = x + \epsilon, \qquad \epsilon \sim \mathcal{N}(0, \sigma^2 I).
\end{equation}
Both approaches trade deterministic activations for stochastic ones, encouraging robustness to perturbations.

\paragraph{Bayesian Interpretation.} 
Gal and Ghahramani~\cite{pmlr-v48-gal16} interpreted dropout as approximate Bayesian inference, where multiple stochastic forward passes correspond to Monte Carlo sampling from a posterior distribution over models. 
Analogously, Gaussian perturbations applied during inference provide a lightweight Monte Carlo estimate of predictive uncertainty:
\begin{align}
    \hat{\mu}(x) &= \frac{1}{M} \sum_{m=1}^M f(x + \epsilon_m), \\
    \hat{\sigma}^2(x) &= \frac{1}{M} \sum_{m=1}^M \big(f(x + \epsilon_m) - \hat{\mu}(x)\big)^2,
\end{align}
where $\epsilon_m$ are independent Gaussian samples. 
Thus, our method extends stochastic regularization into the transformer MLP blocks and enables uncertainty-aware evaluation without modifying prompts.

\paragraph{Approximate Computing Perspective.} 
From an efficiency viewpoint, stochastic perturbations reduce the need for exact, high-precision computation by introducing noise-tolerant approximations~\cite{han2016deepcompressioncompressingdeep}. 
Since the injected noise dominates small floating-point errors, the model can tolerate lower-precision arithmetic in perturbed layers. 
This aligns with the goals of approximate computing: trading precision for efficiency while preserving accuracy in expectation.

\subsection{Implementation in This Thesis}

In this work, we adopt a simplified stochastic perturbation scheme by injecting Gaussian noise into activations before linear projections:
\begin{verbatim}
def stochastic_linear(x, weight, bias=None, noise_std=0.01):
    noise = torch.randn_like(x) * noise_std
    return F.linear(x + noise, weight, bias)
\end{verbatim}
While SPUQ leverages input-level perturbations for epistemic uncertainty, our activation-level Gaussian noise injection connects naturally to stochastic regularization and Bayesian interpretations of deep networks. 
It offers a simple yet effective approximation method that integrates seamlessly into the model architecture and can be combined with quantization, pruning, or \gls{LoRA} in our unified framework.

\section{Stochastic Memory Masking}
\label{sec:memory}

Scaling \gls{LLM}s inevitably runs into memory bottlenecks. 
During both training and inference, the dominant cost is not always arithmetic computation, but rather storing and moving activations, parameters, and intermediate results across limited GPU memory and bandwidth. 
For example, the quadratic complexity of self-attention means that even for moderate sequence lengths, memory usage grows prohibitively large. 
This motivates research into memory approximation methods that deliberately reduce the memory footprint of models at the expense of exactness, thereby enabling deployment at larger scales or under resource-constrained conditions.

While system-level techniques such as FlashAttention~\cite{alizadeh2024llmflashefficientlarge} address the problem by optimizing tiling and IO patterns, in this thesis we focus on algorithmic approaches. 
Among these, stochastic memory masking represents a lightweight yet effective strategy to approximate storage by introducing controlled noise directly into activations. 
Unlike checkpointing or reversible layers, which reduce memory through recomputation, stochastic masking directly enforces sparsity in stored values.

\subsection{Checkpointing and Sparsification}
One classical strategy for memory reduction is activation checkpointing~\cite{chen2016trainingdeepnetssublinear}. 
The key idea is to avoid storing all intermediate activations during the forward pass. 
Instead, only a small subset of ``checkpoints'' is retained, and missing activations are recomputed during backpropagation. 
This reduces memory usage from $O(L)$ to $O(\sqrt{L})$ for $L$ layers, at the cost of extra forward computations.  

Reversible residual networks (\gls{RevNets})~\cite{gomez2017reversibleresidualnetworkbackpropagation} take a different approach by designing layers that are invertible. 
Here, activations can be reconstructed exactly from later outputs, eliminating the need for explicit storage.  

Stochastic methods, by contrast, sacrifice exactness. 
For example, QSGD~\cite{alistarh2017qsgdcommunicationefficientsgdgradient} quantizes each gradient coordinate to a low-bit stochastic representation while preserving unbiasedness, and DropConnect~\cite{pmlr-v28-wan13} randomly masks weights during training. 
These approaches highlight the broader principle: random masking or quantization can approximate limited storage conditions while retaining useful learning signals.

\subsection{Stochastic Memory Masking for Attention}
Building on this line of work, we define stochastic memory masking as a simple but powerful approximation technique applied to attention activations in transformer models.  
Given an attention output vector $y \in \mathbb{R}^d$, we apply an element-wise Bernoulli mask:
\begin{equation}
    \hat{y}_i =
    \begin{cases}
        \dfrac{y_i}{1-p}, & \text{with probability } 1-p, \\
        0, & \text{with probability } p,
    \end{cases}
\end{equation}
where $p$ is the drop probability.  
The scaling factor ensures that the expectation is unbiased:
\begin{equation}
    \mathbb{E}[\hat{y}_i] = y_i,
\end{equation}
while the variance grows with $p$:
\begin{equation}
    \text{Var}[\hat{y}_i] = \frac{p}{1-p} y_i^2.
\end{equation}

In practice, we often omit the rescaling, deliberately introducing a small bias that simulates lossy compression. 
This makes the approximation closer to realistic storage errors, where magnitudes are systematically underestimated.

\subsection{Implementation in This Thesis}
From an approximate computing perspective, stochastic memory masking trades determinism for efficiency. 
It does not physically reduce GPU memory allocation, but it conceptually simulates a reduced-memory regime by discarding activations at random.  
This provides a tunable knob ($p$) that controls the fidelity–efficiency trade-off.  

Unlike structured methods, stochastic masking varies across forward passes, probing the model’s robustness to dynamic activation loss. 
Thus, it offers insight into how resilient large language models are to lossy memory approximations, complementing system-level optimizations like FlashAttention with an algorithmic robustness test.

\section{Evaluation Metrics and Efficiency Measures}

Evaluating approximation techniques for \gls{LLM}s requires both
quality-oriented metrics, which capture linguistic fidelity, and 
efficiency-oriented measures, which quantify computational resource demands.
This dual perspective reflects the standard in \gls{NLP} evaluation
and robustness analysis \cite{yang2018adaptationsrougebleubetter,alon2023detectinglanguagemodelattacks,yacouby-axman-2020-probabilistic}.

\subsection{Quality Metrics}

\paragraph{\gls{BLEU}.}  
Bilingual Evaluation Understudy (\gls{BLEU}) is an $n$-gram
precision-based metric with a brevity penalty to avoid favoring short hypotheses:
\begin{equation}
\text{\gls{BLEU}} = \text{BP} \cdot \exp \left( \frac{1}{N} \sum_{n=1}^{N} \log p_n \right),
\end{equation}
where $p_n$ denotes the modified $n$-gram precision and 
$\text{BP} = \min\big(1, e^{1 - |R|/|C|}\big)$ is the brevity penalty for candidate length $|C|$ against reference length $|R|$.
\gls{BLEU} remains widely used in text generation benchmarks, although its sensitivity to exact $n$-gram overlap has motivated several adaptations \cite{yang2018adaptationsrougebleubetter}.

\paragraph{\gls{ROUGE-L}.}  
\gls{ROUGE-L} evaluates recall-oriented overlap using the longest common subsequence (LCS):
\begin{equation}
\text{\gls{ROUGE-L}} = \frac{(1+\gamma^2) \cdot P_{\text{LCS}} \cdot R_{\text{LCS}}}{R_{\text{LCS}} + \gamma^2 P_{\text{LCS}}},
\end{equation}
where $P_{\text{LCS}} = \tfrac{\text{LCS}(C,R)}{|C|}$ and $R_{\text{LCS}} = \tfrac{\text{LCS}(C,R)}{|R|}$.
\gls{ROUGE-L} has been shown effective for summarization and generative evaluation where ordering matters\cite{yang2018adaptationsrougebleubetter}.

\paragraph{F1 Score.}  
Precision and recall can be applied at the token level to capture overlap between candidate tokens $C$ and reference tokens $R$:
\begin{equation}
\text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}},
\end{equation}
with $\text{Precision} = \tfrac{|C \cap R|}{|C|}$ and $\text{Recall} = \tfrac{|C \cap R|}{|R|}$.
Probabilistic extensions generalize this formulation by weighting overlaps with confidence values, making it suitable for stochastic generation settings \cite{yacouby-axman-2020-probabilistic}.

\paragraph{SBERT Similarity.}
Semantic textual similarity was evaluated using SBERT, a transformer-based model that maps sentences into a shared embedding space optimized through contrastive and siamese training objectives \cite{reimers2019sentencebertsentenceembeddingsusing}. Unlike lexical metrics such as BLEU or ROUGE that rely on $n$-gram overlap, SBERT computes cosine similarity between dense vector representations:
\begin{equation}
\text{SBERT}(C,R) = 
\frac{\langle \mathbf{e}_C, \mathbf{e}_R \rangle}
{\|\mathbf{e}_C\| \cdot \|\mathbf{e}_R\|},
\end{equation}
where $\mathbf{e}_C$ and $\mathbf{e}_R$ denote the embeddings of candidate and reference sentences, respectively. Values range from $-1$ to $1$, but well-aligned natural language pairs typically fall in the interval $[0,1]$, making the metric effective for capturing paraphrastic and semantic equivalence even in the presence of substantial lexical variation. This makes SBERT particularly suitable for evaluating approximated models, whose outputs may deviate at the token level while maintaining semantic fidelity.

\paragraph{Perplexity.}  
Perplexity measures the cross-entropy between the model distribution $P$ and the empirical sequence $x_1^T$:
\begin{equation}
\text{PPL} = \exp \left( -\frac{1}{T} \sum_{t=1}^T \log P(x_t \mid x_{<t}) \right).
\end{equation}
Lower perplexity indicates better predictive power. It is also used for anomaly detection and robustness assessment in LLMs \cite{alon2023detectinglanguagemodelattacks}.

\subsection{Efficiency Measures}
\label{sec:efficiency_measures}

Evaluating approximation techniques requires not only accuracy-based metrics but also measurements of computational and memory efficiency. In this work, efficiency is analyzed in terms of inference latency and model size. These measures provide complementary views of runtime cost and resource utilization.

\paragraph{Inference Latency.}
Latency represents the total time required for model inference and serves as a direct measure of computational efficiency. 
Inference time $T$ is defined as the wall-clock difference between the start and end of a single evaluation loop:
\begin{equation}
T = t_{\text{end}} - t_{\text{start}},
\end{equation}
where $t_{\text{start}}$ and $t_{\text{end}}$ correspond to timestamps collected immediately before and after the forward pass of the inference, respectively. 
In practice, the evaluation loop measures this duration using \texttt{time.time()} surrounding the \texttt{model.generate()} call. 
The forward generation includes token sampling with top-$p$ nucleus sampling and temperature scaling. This timing reflects end-to-end inference, including decoding and post-processing.

\paragraph{Output File Size.}
To assess the effect of approximation on textual generation volume, all model predictions, references, and prompts were stored in serialized JSONL format. The file size in kilobytes (KB) is computed as:
\begin{equation}
S_{\text{out}} = \frac{\text{Bytes(file)}}{1024}.
\end{equation}
This measure indirectly reflects verbosity and compression effects induced by quantization and stochastic methods. 

\paragraph{Model Size.}
To quantify memory efficiency, the effective model size is computed as the sum of all parameter and buffer tensors, weighted by their numerical precision. Let the model contain $K$ tensors with element counts $n_k$ and bit-widths $b_k$. 
The total memory footprint is:
\begin{equation}
S_{\text{model}} = \sum_{k=1}^{K} \frac{n_k \cdot b_k}{8 \cdot 1024^2} \quad \text{(MB)}.
\end{equation}
Full-precision models (e.g., BF16 or FP16) use uniform bit-widths, while approximation techniques modify $b_k$ through quantization (INT8, INT4, NF4), low-rank adaptation, or structural pruning. Quantization reduces $b_k$, shrinking each tensor, whereas pruning reduces $n_k$ by removing weights. \gls{LoRA} introduces small rank-decomposition matrices whose contribution is negligible compared to the full dense layers. Thus, the final model size captures the cumulative effect of compression,  precision scaling, and structural sparsity, providing a hardware-agnostic indicator of memory efficiency for deployment-oriented model design.

\paginavuota