\chapter{Experiments and Results}
\label{chap:results}

\section{Experimental Setup}
\label{sec:experimental_setup}

We evaluate the approximation techniques introduced in Chapter~\ref{sec:approximation} on two instruction-tuned transformer models that are LLaMA-3.2-1B-Instruct and Gemma-3-1B-Instruct. Both models were chosen for their compact size, open availability, and compatibility with approximate computing methods.

\paragraph{Datasets.}
Three instruction-following datasets were used in our experiments: Alpaca, Databricks-Dolly-15k, and AgentInstruct. To reduce overfitting and better evaluate the model’s ability to generalize across different instruction styles, we adopted a cross-dataset evaluation strategy. In this setting, the model is fine-tuned on one dataset and evaluated on a different one. For example, in the Alpaca→Agent configuration, the model is trained on a 5,000-sample subset of Alpaca and validated on the AgentInstruct dataset. Similarly, Alpaca→Dolly and Agent→Dolly configurations were used to introduce domain shift between training and evaluation. These cross-dataset evaluations allow us to assess whether the model learns transferable instruction-following behavior rather than memorizing dataset-specific patterns.

\paragraph{Baselines.}
Baseline inference results were taken using the pretrained checkpoints of both models. This serves as reference points for comparing relative improvements achieved through the application of approximate computing techniques.

\paragraph{Training Configuration.}
Fine-tuning was carried out using the AdamW optimizer, applied only to parameters marked as trainable (for example, LoRA update matrices when LoRA was enabled). A cosine learning rate schedule with a warmup ratio of 20\% of the total training steps was used. Training was performed for 3-5 epochs depending on the experiment, with gradient accumulation (typically 2 steps) employed to accommodate GPU memory limitations. The loss function was the standard cross-entropy loss with masking applied to padding or ignored labels. Perplexity was computed at the end of each epoch to monitor the progression of training.

\paragraph{Evaluation Protocol.}
The effectiveness of each approximation method was evaluated using metrics that quantify both linguistic accuracy and model efficiency. Linguistic performance was measured using BLEU, ROUGE-L, F1, and SBERT scores. BLEU captures $n$-gram precision between generated responses and reference outputs. ROUGE-L measures the longest common subsequence, reflecting semantic coverage and structural coherence. The F1 score balances precision and recall at the token level,
\begin{equation}
    F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}},
\end{equation}
ensuring robustness to differences in response length. In addition to these token-overlap metrics, we report a SBERT-based similarity score, obtained by computing the cosine similarity between sentence embeddings of the generated output and the reference response. SBERT values lie in $[0,1]$ and provide a semantic measure of alignment that is more tolerant to paraphrasing and variation in surface form, complementing BLEU and ROUGE-L.

To characterize model efficiency, we tracked three complementary indicators. Firstly, model size was estimated by accounting for parameter and buffer representations, including reduced-precision formats introduced by quantization. Secondly, the size of the generated outputs was recorded as an indicator of response verbosity and determinism. Thirdly, the average inference time was measured to capture the latency impact of each approximation technique and quantify the trade-off between computational efficiency and linguistic performance.

\clearpage
\section{Evaluation Results}

\subsection{Inference Results for the Dolly-15k Dataset(Fine-tuned with Alpaca Dataset)}
\label{sec:alpaca_results}

\begin{longtable}{|l l c c c c c c|}
\hline
\textbf{Model} & \makecell{\textbf{Approximation} \\ \textbf{Technique}} & \textbf{\gls{BLEU}} & \textbf{\gls{ROUGE-L}} & \textbf{SBERT} & \makecell{\textbf{Inf.} \\ \textbf{Time} \\ \textbf{(s)}} & \makecell{\textbf{Out.} \\ \textbf{Size} \\ \textbf{(KB)}} & \makecell{\textbf{Model} \\ \textbf{Size} \\ \textbf{(MB)}} \\
\hline
\endfirsthead
\hline
\textbf{Model} & \makecell{\textbf{Approximation} \\ \textbf{Technique}} & \textbf{\gls{BLEU}} & \textbf{\gls{ROUGE-L}} & \textbf{SBERT} & \makecell{\textbf{Inf.} \\ \textbf{Time} \\ \textbf{(s)}} & \makecell{\textbf{Out.} \\ \textbf{Size} \\ \textbf{(KB)}} & \makecell{\textbf{Model} \\ \textbf{Size} \\ \textbf{(MB)}} \\
\hline
\endhead
\hline
\multicolumn{8}{r}{\textit{Continued on next page}} \\
\hline
\endfoot
\hline
\caption{Inference results for LLaMA-3.2-1B-Instruct and Gemma-3-1B-Instruct on the Dolly-15k dataset (Fine-tuned with Alpaca dataset).}
\endlastfoot
\multicolumn{8}{|l|}{\textbf{Baseline}} \\
\scriptsize{LLaMA-3.2-1B} & -----             & 0.58 & 0.67 & 0.88 & 2.2 & 163.8 & 2,858\\
\scriptsize{Gemma-3-1B}   & -----             & 0.25 & 0.46 & 0.68 & 7.9 & 121.3 & 2,483\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Pruned}        & 0.62 & 0.69 & 0.90 & 2.0 & 164.9 & 3,370\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Pruned}        & 0.26 & 0.46 & 0.69 & 8.0 & 122.4 & 2,878\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8}          & 0.86 & 0.90 & 0.94 & 1.3 & 154.3 & 1,681\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8}          & 0.25 & 0.46 & 0.68 & 15.3 & 125.5 & 1,532\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4}          & 0.36 & 0.48 & 0.81 & 7.6 & 184.9 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4}          & 0.24 & 0.42 & 0.77 & 15.0 & 199.4 & 1,055\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{NF4}           & 0.47 & 0.56 & 0.83 & 8.1 & 187.7 & 1,164\\
\scriptsize{Gemma-3-1B}   & \scriptsize{NF4}           & 0.23 & 0.46 & 0.73 & 19.5 & 163.5 & 1,112\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{\gls{LoRA}}    & 0.55 & 0.65 & 0.89 & 2.7 & 162.8 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{\gls{LoRA}}    & 0.25 & 0.45 & 0.66 & 8.9 & 116.2 & 2,484\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Perturbed}     & 0.60 & 0.70 & 0.89 & 1.9 & 159.8 & 2,858\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Perturbed}     & 0.21 & 0.40 & 0.65 & 8.1 & 119.4 & 2,483\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Mem-masked}    & 0.58 & 0.66 & 0.87 & 2.2 & 158.0 & 2,858\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Mem-masked}    & 0.22 & 0.43 & 0.67 & 8.1 & 129.8 & 2,483\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Pruned + \gls{LoRA}}        & 0.57 & 0.66 & 0.87 & 2.7 & 165.2 & 3,371\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Pruned + \gls{LoRA}}        & 0.23 & 0.41 & 0.63 & 9.2 & 163.7 & 2,879\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8 + \gls{LoRA}}          & 0.75 & 0.80 & 0.91 & 4.1 & 141.5 & 1,682\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8 + \gls{LoRA}}          & 0.21 & 0.37 & 0.73 & 20.2 & 187.9 & 1,533\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4 + \gls{LoRA}}          & 0.32 & 0.44 & 0.83 & 10.5 & 169.6 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4 + \gls{LoRA}}          & 0.32 & 0.53 & 0.84 & 20.1 & 222.8 & 1,056\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{NF4 + \gls{LoRA}}           & 0.46 & 0.56 & 0.83 & 12.1 & 173.4 & 1,164\\
\scriptsize{Gemma-3-1B}   & \scriptsize{NF4 + \gls{LoRA}}           & 0.23 & 0.42 & 0.75 & 20.6 & 172.5 & 1,113\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Perturbed + \gls{LoRA}}     & 0.55 & 0.65 & 0.87 & 2.7 & 165.9 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Perturbed + \gls{LoRA}}     & 0.23 & 0.40 & 0.61 & 9.3 & 164.9 & 2,484\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Masked + \gls{LoRA}}        & 0.49 & 0.61 & 0.85 & 3.1 & 171.7 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Masked + \gls{LoRA}}        & 0.21 & 0.39 & 0.61 & 9.4 & 173.9 & 2,484\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8+Pruned+\gls{LoRA}}     & 0.71 & 0.77 & 0.88 & 4.1 & 169.9 & 1,682\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8+Pruned+\gls{LoRA}}     & 0.31 & 0.53 & 0.72 & 20.3 & 151.9 & 1,533\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4+Pruned+\gls{LoRA}}     & 0.35 & 0.46 & 0.82 & 10.4 & 183.6 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4+Pruned+\gls{LoRA}}     & 0.26 & 0.45 & 0.82 & 20.2 & 193.3 & 1,056\\
\end{longtable}

\paragraph{Accuracy.}
The accuracy characteristics of LLaMA-3.2-1B and Gemma-3-1B differ substantially across approximation methods. LLaMA-3.2-1B exhibits strong improvements from quantization, with \texttt{INT8} achieving the highest scores (\gls{BLEU} = 0.86, \gls{ROUGE-L} = 0.90, SBERT = 0.94). These results confirm LLaMA’s robustness to 8-bit quantization, where weights are stored in INT8 but 
dequantized back to FP16 before matrix multiplications, resulting in minimal loss of numerical fidelity. Other approximation methods such as pruning, memory masking, and perturbation remain close to the baseline, while more aggressive quantization (\texttt{INT4}, \texttt{NF4}) leads to visible degradation.

Gemma-3-1B shows a markedly different behavior. Neither \texttt{INT8} nor \texttt{INT8+LoRA} yields improvements over the baseline, and in several cases performance degrades significantly. Surprisingly, Gemma’s best-performing configurations are 4-bit variants (\texttt{INT4} and \texttt{INT4+LoRA}), which reach SBERT similarities of 0.84--0.85, outperforming all INT8 results. This effect is consistent with prior observations that small transformer models with sharp activation distributions may be more sensitive to uniform INT8 quantization, while structured 4-bit formats (e.g., INT4, NF4) preserve semantic similarity more effectively. 

Overall, LLaMA-3.2-1B consistently benefits from INT8 quantization, whereas Gemma-3-1B achieves its strongest semantic fidelity under 4-bit approximations.

\paragraph{Efficiency.}
LLaMA-3.2-1B also demonstrates favorable efficiency properties under INT8 quantization. The \texttt{INT8} model reduces the model size from $2{,}858$\,MB to $1{,}681$\,MB and achieves the lowest inference latency ($1.3$\,s), confirming the presence of efficient 8-bit kernel support. Although INT4 and NF4 provide higher compression ratios, they introduce substantial latency increases (7.6--12.1\,s), largely due to the lack of optimized low-bit kernels for these formats.

Gemma-3-1B displays the opposite latency trend. While INT8 reduces the model size to $1{,}532$\,MB, the inference latency increases sharply (15--20\,s), and NF4 results in even slower execution (up to 20.6\,s). These results indicate that Gemma’s architecture interacts less favorably with available INT8/INT4 kernels, leading to severely degraded speed despite reduced model size. Thus, approximation efficiency in Gemma is constrained more by kernel availability and activation-range sensitivity 
than by parameter precision.

\begin{table}[t]
\centering
\begin{tabular}{|l l c c c c|}
\hline
\textbf{Model} &
\makecell{\textbf{Approximation} \\ \textbf{Technique}} &
\textbf{ROUGE-L} &
\textbf{SBERT} &
\makecell{\textbf{Inference} \\ \textbf{Time(s)}} &
\makecell{\textbf{Model} \\ \textbf{Size(MB)}} \\
\hline
\scriptsize{LLaMA-3.2-1B} & INT8              & 0.90 & 0.94 & 1.3  & 1,681 \\
\scriptsize{LLaMA-3.2-1B} & INT8 + LoRA       & 0.80 & 0.91 & 4.1  & 1,682 \\
\scriptsize{LLaMA-3.2-1B} & Pruned            & 0.69 & 0.90 & 2.0  & 3,370 \\
\hline
\scriptsize{Gemma-3-1B}   & INT4 + LoRA       & 0.53 & 0.84 & 20.1 & 1,056 \\
\scriptsize{Gemma-3-1B}   & INT4+Pruned+\gls{LoRA} & 0.45 & 0.82 & 20.2 & 1,056 \\
\scriptsize{Gemma-3-1B}   & INT4              & 0.42 & 0.77 & 15.0 & 1,055 \\
\hline
\end{tabular}
\caption{Top-performing approximation techniques for LLaMA-3.2-1B and Gemma-3-1B on the Dolly-15k dataset (Fine-tuned with Alpaca dataset).}
\label{tab:top3_dolly15k}
\end{table}

\paragraph{Summary.}
Across the evaluated approximation techniques, INT8 quantization provides the most favorable accuracy--efficiency trade-off for LLaMA-3.2-1B. It achieves the highest semantic fidelity (SBERT = 0.94), strong n-gram overlap (\gls{BLEU} = 0.86; \gls{ROUGE-L} = 0.90), and the lowest inference latency, while also reducing the memory footprint. More aggressive 4-bit formats further compress the model but yield slower inference and reduced accuracy.

Gemma-3-1B exhibits a qualitatively different profile: INT8 offers no measurable accuracy improvement and significantly increases latency. Instead, INT4-based methods, in particular \texttt{INT4+LoRA} achievse the strongest semantic alignment 
and outperform all INT8 variants in terms of accuracy. These findings align with prior reports that small Gemma models are more sensitive to INT8 quantization, while 4-bit structured quantization preserves relative performance more effectively.

The results collectively show that approximation effectiveness is architecture-dependent. LLaMA-3.2-1B is best approximated through INT8 quantization, whereas Gemma-3-1B achieves its best balance of accuracy under INT4+LoRA, albeit with a significant inference-time penalty.

\clearpage
\subsection{Inference Results for the Dolly-15k Dataset(Fine-tuned with Agent Dataset)}
\label{sec:dolly_results}

\begin{longtable}{|l l c c c c c c|}
\hline
\textbf{Model} & \makecell{\textbf{Approximation} \\ \textbf{Technique}} & \textbf{\gls{BLEU}} & \textbf{\gls{ROUGE-L}} & \textbf{SBERT} & \makecell{\textbf{Inf.} \\ \textbf{Time} \\ \textbf{(s)}} & \makecell{\textbf{Out.} \\ \textbf{Size} \\ \textbf{(KB)}} & \makecell{\textbf{Model} \\ \textbf{Size} \\ \textbf{(MB)}} \\
\hline
\endfirsthead
\hline
\textbf{Model} & \makecell{\textbf{Approximation} \\ \textbf{Technique}} & \textbf{\gls{BLEU}} & \textbf{\gls{ROUGE-L}} & \textbf{SBERT} & \makecell{\textbf{Inf.} \\ \textbf{Time} \\ \textbf{(s)}} & \makecell{\textbf{Out.} \\ \textbf{Size} \\ \textbf{(KB)}} & \makecell{\textbf{Model} \\ \textbf{Size} \\ \textbf{(MB)}} \\
\hline
\endhead
\hline
\multicolumn{8}{r}{\textit{Continued on next page}} \\
\hline
\endfoot
\hline
\caption{Inference results for LLaMA-3.2-1B-Instruct and Gemma-3-1B-Instruct on the Dolly-15k dataset (Fine-tuned with Agent dataset).}
\endlastfoot
\multicolumn{8}{|l|}{\textbf{Baseline}} \\
\scriptsize{LLaMA-3.2-1B} & -----             & 0.54 & 0.64 & 0.86 & 2.2 & 152.4 & 2,858\\
\scriptsize{Gemma-3-1B}   & -----             & 0.29 & 0.49 & 0.75 & 7.7 & 133.6 & 2,483\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Pruned}        & 0.59 & 0.68 & 0.88 & 2.5 & 157.4 & 3,370\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Pruned}        & 0.27 & 0.48 & 0.73 & 7.7 & 131.3 & 2,878\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8}          & 0.53 & 0.62 & 0.83 & 5.5 & 174.5 & 1,681\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8}          & 0.27 & 0.47 & 0.74 & 14.7 & 129.7 & 1,532\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4}          & 0.34 & 0.44 & 0.77 & 7.7 & 190.9 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4}          & 0.28 & 0.47 & 0.85 & 14.8 & 187.9 & 1,055\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{NF4}           & 0.42 & 0.53 & 0.83 & 9.4 & 170.4 & 1,164\\
\scriptsize{Gemma-3-1B}   & \scriptsize{NF4}           & 0.22 & 0.44 & 0.69 & 18.6 & 176.9 & 1,112\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{\gls{LoRA}}    & 0.56 & 0.66 & 0.86 & 2.7 & 160.5 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{\gls{LoRA}}    & 0.26 & 0.45 & 0.68 & 8.5 & 129.0 & 2,484\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Perturbed}     & 0.56 & 0.66 & 0.89 & 2.5 & 158.3 & 2,858\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Perturbed}     & 0.26 & 0.46 & 0.73 & 7.7 & 130.5 & 2,483\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Mem-masked}    & 0.54 & 0.63 & 0.87 & 2.8 & 161.2 & 2,858\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Mem-masked}    & 0.26 & 0.45 & 0.74 & 8.3 & 139.5 & 2,483\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Pruned + \gls{LoRA}}        & 0.54 & 0.63 & 0.83 & 2.8 & 175.6 & 3,371\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Pruned + \gls{LoRA}}        & 0.29 & 0.46 & 0.68 & 8.8 & 154.0 & 2,879\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8 + \gls{LoRA}}          & 0.55 & 0.63 & 0.85 & 8.0 & 175.9 & 1,682\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8 + \gls{LoRA}}          & 0.30 & 0.47 & 0.72 & 19.6 & 146.1 & 1,533\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4 + \gls{LoRA}}          & 0.30 & 0.42 & 0.77 & 10.8 & 193.3 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4 + \gls{LoRA}}          & 0.26 & 0.45 & 0.83 & 19.7 & 176.6 & 1,056\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{NF4 + \gls{LoRA}}           & 0.43 & 0.54 & 0.85 & 12.0 & 153.9 & 1,164\\
\scriptsize{Gemma-3-1B}   & \scriptsize{NF4 + \gls{LoRA}}           & 0.25 & 0.43 & 0.73 & 26.4 & 161.5 & 1,113\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Perturbed + \gls{LoRA}}     & 0.50 & 0.58 & 0.82 & 3.4 & 182.3 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Perturbed + \gls{LoRA}}     & 0.32 & 0.52 & 0.74 & 8.8 & 157.8 & 2,484\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Masked + \gls{LoRA}}        & 0.49 & 0.58 & 0.82 & 3.3 & 178.8 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Masked + \gls{LoRA}}        & 0.24 & 0.47 & 0.73 & 8.9 & 163.2 & 2,484\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8+Pruned+\gls{LoRA}}     & 0.59 & 0.67 & 0.86 & 7.5 & 159.5 & 1,682\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8+Pruned+\gls{LoRA}}     & 0.25 & 0.43 & 0.69 & 19.3 & 164.4 & 1,533\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4+Pruned+\gls{LoRA}}     & 0.31 & 0.42 & 0.84 & 10.6 & 165.8 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4+Pruned+\gls{LoRA}}     & 0.31 & 0.48 & 0.85 & 19.6 & 220.4 & 1,056\\
\end{longtable}

\paragraph{Accuracy.}
The accuracy trends for the Agent-fine-tuned Dolly-15k dataset differ markedly from those obtained using the Alpaca dataset. 
LLaMA-3.2-1B shows only modest accuracy gains across most approximation methods. Perturbed remains the strongest performer among the simple approximations (\gls{BLEU} = 0.56, \gls{ROUGE-L} = 0.66, SBERT = 0.89), while the Pruning method preserves semantic similarity close to the baseline (SBERT = 0.88). However, in contrast to the Alpaca-trained setting, the \texttt{INT8} configuration does not improve accuracy on LLaMA. Its scores (\gls{BLEU} = 0.53, SBERT = 0.83) fall below the baseline, indicating that quantization interacts less favorably fine-tuning with the Agent dataset.

Gemma-3-1B displays a completely different behavior. As in the Alpaca case, 8-bit quantization is ineffective. However, 4-bit variants significantly outperform all other approximations. \texttt{INT4} and \texttt{INT4+Pruned+LoRA} achieve the highest semantic similarity (SBERT = 0.85), and \texttt{INT4+LoRA} also performs strongly (SBERT = 0.83). Thus, Gemma again shows a preference for structured 4-bit quantization, maintaining or improving semantic 
quality while 8-bit methods remain suboptimal.

\paragraph{Efficiency.}
From an efficiency perspective, LLaMA and Gemma behave very differently. LLaMA’s \texttt{INT8} and \texttt{INT8+LoRA} methods produce substantial increases in inference latency (5.5–8.0\,s), reflecting the absence of kernel-level optimizations for 8-bit execution in the Agent-trained setting. Meanwhile, pruning and memory masking produce latencies similar to the baseline (2.5–3.0\,s) while providing moderate accuracy benefits.

Gemma-3-1B experiences severe latency slowdowns across nearly all approximation methods. All INT8 and INT4 variants exceed 14–20\,s per inference, and NF4-based methods are even slower (up to 26.4\,s). Only pruning and memory masking maintain baseline-level speed (7–9\,s), but these provide only mild accuracy gains. Thus, Gemma lacks an approximation technique that simultaneously improves accuracy and reduces latency.

\begin{table}[t]
\centering
\begin{tabular}{|l l c c c c|}
\hline
\textbf{Model} &
\makecell{\textbf{Approximation} \\ \textbf{Technique}} &
\textbf{ROUGE-L} &
\textbf{SBERT} &
\makecell{\textbf{Inference} \\ \textbf{Time(s)}} &
\makecell{\textbf{Model} \\ \textbf{Size(MB)}} \\
\hline
\scriptsize{LLaMA-3.2-1B} & Perturbed                    & 0.66 & 0.89 & 2.5 & 2,858 \\
\scriptsize{LLaMA-3.2-1B} & Pruned                       & 0.68 & 0.88 & 2.5 & 3,370 \\
\scriptsize{LLaMA-3.2-1B} & INT8 + Pruned + LoRA         & 0.67 & 0.86 & 7.5 & 1,682 \\
\hline
\scriptsize{Gemma-3-1B}   & INT4                         & 0.47 & 0.85 & 14.8 & 1,055 \\
\scriptsize{Gemma-3-1B}   & INT4 + Pruned + LoRA         & 0.48 & 0.85 & 19.6 & 1,056 \\
\scriptsize{Gemma-3-1B}   & INT4 + LoRA                  & 0.45 & 0.83 & 19.7 & 1,056 \\
\hline
\end{tabular}
\caption{Top-performing approximation techniques for LLaMA-3.2-1B and Gemma-3-1B on the Dolly-15k dataset (Fine-tuned with Agent dataset).}
\label{tab:top3_agent_dolly15k}
\end{table}

\paragraph{Summary.}
The Agent-fine-tuned results reveal two distinct approximation profiles. For LLaMA-3.2-1B, pruning and hybrid variants such as \texttt{INT8+Pruned+LoRA} provide the best accuracy while maintaining reasonable inference times. Unlike the Alpaca dataset, INT8 quantization does not enhance accuracy and frequently increases latency. In contrast, Gemma-3-1B achieves its strongest accuracy under 4-bit approximations, despite their substantial computational overhead. INT4- and INT4+LoRA-based methods consistently reach the highest SBERT similarity (0.85) among all Gemma variants, confirming that Gemma's architecture is more compatible with structured 4-bit quantization than with 8-bit precision.

\subsection{Inference Results for the AgentInstruct Dataset(Fine-tuned with Alpaca Dataset)}
\label{sec:agentinstruct_results}

\begin{longtable}{|l l c c c c c c|}
\hline
\textbf{Model} & \makecell{\textbf{Approximation} \\ \textbf{Technique}} & \textbf{\gls{BLEU}} & \textbf{\gls{ROUGE-L}} & \textbf{SBERT} & \makecell{\textbf{Inf.} \\ \textbf{Time} \\ \textbf{(s)}} & \makecell{\textbf{Out.} \\ \textbf{Size} \\ \textbf{(KB)}} & \makecell{\textbf{Model} \\ \textbf{Size} \\ \textbf{(MB)}} \\
\hline
\endfirsthead
\hline
\textbf{Model} & \makecell{\textbf{Approximation} \\ \textbf{Technique}} & \textbf{\gls{BLEU}} & \textbf{\gls{ROUGE-L}} & \textbf{SBERT} & \makecell{\textbf{Inf.} \\ \textbf{Time} \\ \textbf{(s)}} & \makecell{\textbf{Out.} \\ \textbf{Size} \\ \textbf{(KB)}} & \makecell{\textbf{Model} \\ \textbf{Size} \\ \textbf{(MB)}} \\
\hline
\endhead
\hline
\multicolumn{8}{r}{\textit{Continued on next page}} \\
\hline
\endfoot
\hline
\caption{Inference results for LLaMA-3.2-1B-Instruct and Gemma-3-1B-Instruct on the AgentInstruct dataset (Fine-tuned with Alpaca dataset).}
\endlastfoot
\multicolumn{8}{|l|}{\textbf{Baseline}} \\
\scriptsize{LLaMA-3.2-1B} & -----             & 0.59 & 0.68 & 0.85 & 2.7 & 34.3 & 2,858\\
\scriptsize{Gemma-3-1B}   & -----             & 0.26 & 0.63 & 0.75 & 8.1 & 36.7 & 2,483\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Pruned}        & 0.59 & 0.67 & 0.86 & 2.4 & 34.6 & 3,370\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Pruned}        & 0.22 & 0.54 & 0.71 & 8.3 & 35.4 & 2,878\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8}          & 0.93 & 0.95 & 0.98 & 0.57 & 29.8 & 1,681\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8}          & 0.30 & 0.58 & 0.82 & 15.4 & 41.9 & 1,532\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4}          & 0.23 & 0.37 & 0.71 & 7.5 & 39.8 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4}          & 0.19 & 0.46 & 0.87 & 15.3 & 41.5 & 1,055\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{NF4}           & 0.49 & 0.61 & 0.75 & 8.2 & 37.3 & 1,164\\
\scriptsize{Gemma-3-1B}   & \scriptsize{NF4}           & 0.17 & 0.39 & 0.71 & 19.2 & 37.9 & 1,112\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{\gls{LoRA}}    & 0.57 & 0.68 & 0.82 & 3.5 & 33.2 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{\gls{LoRA}}    & 0.25 & 0.56 & 0.70 & 9.2 & 35.3 & 2,484\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Perturbed}     & 0.48 & 0.58 & 0.75 & 2.8 & 34.8 & 2,858\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Perturbed}     & 0.22 & 0.53 & 0.73 & 8.4 & 36.6 & 2,483\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Mem-masked}    & 0.48 & 0.60 & 0.75 & 3.3 & 33.4 & 2,858\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Mem-masked}    & 0.22 & 0.48 & 0.72 & 8.3 & 38.8 & 2,483\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Pruned + \gls{LoRA}}        & 0.44 & 0.57 & 0.75 & 3.8 & 34.9 & 3,371\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Pruned + \gls{LoRA}}        & 0.25 & 0.58 & 0.73 & 9.3 & 35.2 & 2,879\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8 + \gls{LoRA}}          & 0.86 & 0.88 & 0.95 & 1.8 & 31.6 & 1,682\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8 + \gls{LoRA}}          & 0.26 & 0.55 & 0.82 & 20.3 & 38.5 & 1,533\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4 + \gls{LoRA}}          & 0.33 & 0.47 & 0.83 & 10.0 & 42.3 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4 + \gls{LoRA}}          & 0.19 & 0.41 & 0.84 & 20.3 & 40.2 & 1,056\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{NF4 + \gls{LoRA}}           & 0.51 & 0.59 & 0.83 & 11.5 & 39.3 & 1,164\\
\scriptsize{Gemma-3-1B}   & \scriptsize{NF4 + \gls{LoRA}}           & 0.20 & 0.57 & 0.80 & 26.2 & 41.4 & 1,113\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Perturbed + \gls{LoRA}}     & 0.44 & 0.57 & 0.82 & 3.5 & 33.8 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Perturbed + \gls{LoRA}}     & 0.31 & 0.64 & 0.73 & 9.3 & 34.9 & 2,484\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{Masked + \gls{LoRA}}        & 0.47 & 0.61 & 0.81 & 3.5 & 32.8 & 2,859\\
\scriptsize{Gemma-3-1B}   & \scriptsize{Masked + \gls{LoRA}}        & 0.23 & 0.53 & 0.75 & 9.3 & 39.5 & 2,484\\
\hline
\multicolumn{8}{|l|}{\textbf{Fine-tuned}} \\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT8+Pruned+\gls{LoRA}}     & 0.72 & 0.77 & 0.88 & 4.9 & 35.3 & 1,682\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT8+Pruned+\gls{LoRA}}     & 0.21 & 0.54 & 0.73 & 20.0 & 37.4 & 1,533\\
\scriptsize{LLaMA-3.2-1B} & \scriptsize{INT4+Pruned+\gls{LoRA}}     & 0.37 & 0.48 & 0.80 & 9.8 & 44.0 & 1,092\\
\scriptsize{Gemma-3-1B}   & \scriptsize{INT4+Pruned+\gls{LoRA}}     & 0.22 & 0.40 & 0.87 & 20.0 & 44.4 & 1,056\\
\end{longtable}


\paragraph{Accuracy.}
For the AgentInstruct dataset, the strongest pattern across all evaluations is the dominant performance of INT8-based methods for LLaMA-3.2-1B. The plain \texttt{INT8} configuration achieves exceptionally high accuracy (\gls{BLEU} = 0.93, \gls{ROUGE-L} = 0.95, SBERT = 0.98), representing the best scores among all tested techniques and datasets. This behavior mirrors the Alpaca-trained results and confirms that LLaMA’s architecture benefits substantially from 8-bit quantization, preserving semantic similarity while reducing numerical noise through FP16 dequantized computation.

LLaMA’s \texttt{INT8+LoRA} variant also performs well (\gls{BLEU} = 0.86, SBERT = 0.95), although it remains slightly below the pure INT8 model. Other approximation techniques including pruning, perturbation, memory masking, and all 4-bit or NF4 methods provide only limited accuracy improvements and often degrade performance. The weakest methods for LLaMA are the INT4-based variants, which fall significantly in both token-level and semantic metrics (\gls{BLEU} = 0.23--0.37).

Gemma-3-1B again shows a contrasting behavior.  As in previous datasets, INT8 remains ineffective, providing only moderate improvements (SBERT = 0.82) and substantially lower BLEU/ROUGE-L than LLaMA. Both \texttt{INT4} and \texttt{INT4+Pruned+LoRA} achieve SBERT values of 0.86 and 0.87, respectively the highest semantic similarity across all Gemma configurations. This reinforces the observation that Gemma’s small-parameter architecture responds better to 4-bit structured quantization than to 8-bit uniform quantization.

\paragraph{Efficiency.}
The efficiency profile sharply distinguishes the two architectures. For LLaMA-3.2-1B, INT8 delivers substantial latency reductions; the \texttt{INT8} model achieves the lowest inference time in all experiments (0.57\,s), representing more than a $4\times$ improvement over the baseline. Even with LoRA applied, INT8 remains efficient (1.8\,s), significantly outperforming heavier methods such as INT4 (7–10\,s) or NF4 (8–12\,s). Pruning and perturbation remain lightweight, maintaining near-baseline runtimes (2.4–3.5\,s).

Gemma-3-1B again lacks an efficient approximation method. All quantized variants (INT8, INT4, NF4) suffer from severe latency penalties (15–26\,s), consistent with the lack of optimized kernels. Only pruning and memory masking preserve baseline-level latency, but they do not offer meaningful accuracy gains in this dataset. Thus, no approximation technique simultaneously improves Gemma’s accuracy and efficiency.

\begin{table}[t]
\centering
\begin{tabular}{|l l c c c c|}
\hline
\textbf{Model} &
\makecell{\textbf{Approximation} \\ \textbf{Technique}} &
\textbf{ROUGE-L} &
\textbf{SBERT} &
\makecell{\textbf{Inference} \\ \textbf{Time(s)}} &
\makecell{\textbf{Model} \\ \textbf{Size(MB)}} \\
\hline
\scriptsize{LLaMA-3.2-1B} & INT8                        & 0.95 & 0.98 & 0.57 & 1,681 \\
\scriptsize{LLaMA-3.2-1B} & INT8 + LoRA                 & 0.88 & 0.95 & 1.8  & 1,682 \\
\scriptsize{LLaMA-3.2-1B} & INT8 + Pruned + LoRA        & 0.77 & 0.88 & 4.9  & 1,682 \\
\hline
\scriptsize{Gemma-3-1B}   & INT4                        & 0.46 & 0.87 & 15.3 & 1,055 \\
\scriptsize{Gemma-3-1B}   & INT4 + Pruned + LoRA        & 0.40 & 0.87 & 20.0 & 1,056 \\
\scriptsize{Gemma-3-1B}   & INT4 + LoRA                 & 0.41 & 0.84 & 20.3 & 1,056 \\
\hline
\end{tabular}
\caption{Top-performing approximation techniques for LLaMA-3.2-1B and Gemma-3-1B on the AgentInstruct dataset (Fine-tuned with Alpaca dataset).}
\label{tab:top3_agentinstruct}
\end{table}

\paragraph{Summary.}
The AgentInstruct evaluations confirm a strong dataset- and model-dependent divide between approximation behaviors. For LLaMA-3.2-1B, INT8 is clearly the optimal choice. It simultaneously achieves the highest accuracy (SBERT = 0.98) and lowest inference latency (0.57\,s), providing the best performance–efficiency trade-off among all tested methods. Combining INT8 with LoRA produces slightly lower accuracy but remains highly competitive. Other approximations provide only modest improvements or degrade performance.

Gemma-3-1B, in contrast, achieves its best semantic similarity under 4-bit structured quantization (\texttt{INT4} and \texttt{INT4+Pruned+LoRA}, SBERT = 0.86--0.87), confirming the model's preference for lower-precision quantization over 8-bit methods. However, these accuracy gains come with significant inference slowdowns, making them impractical for efficiency-oriented scenarios. Overall, the results highlight the importance of architectural compatibility: INT8 is ideal for LLaMA, while Gemma benefits more from 4-bit quantization, albeit at high computational cost.

\paginavuota