Entering the sub-nanometer era of transistors, energy efficiency has become essential for high-performance hardware architectures, especially in high-end data center accelerators tasked with complex workloads like Convolutional Neural Networks (CNNs) and other Deep Learning (DL) applications. Following this trend, new architectures are emerging. One of these is Soft-SIMD Functional Units, a specific type of SIMD functional unit. This architecture supports the flexible use of low bit-width data types (as low as 3 bits), improving parallel performance in both uniformly and heterogeneously quantized (UQ and HQ) CNNs compared to hardware-based counterparts (hard-SIMD). This design also utilizes a shift-add-based Canonical Signed Digit (CSD) multiplication, which further reduces area (59.9%), and energy consumption (50.1%) compared to hard-SIMD, with only minor performance degradation.