Approximate Computing for Softmax and Squash functions in Capsule Networks

This work presents a spectrum of approximations of the softmax and squash functions used in Capsule Neural Networks, building upon previous related research as well as introducing new ideas and solutions. The main focus of the work is to explore approximate computing techniques for softmax and squash at the algorithmic level, in order to demonstrate a trade-off between complexity cost of the hardware implementation of the functions and inference accuracy of the Capsule Network. In particular, area usage and power consumption are considered as hardware complexity metrics and the inference pass is performed with different Capsule Network models on different benchmark image datasets. Three approximate softmax and three approximate squash architectures are proposed with the ultimate goal of making comparisons between multiple instances of the same function type, in terms of area/energy costs and accuracy of the overall Capsule Network. The motivational idea behind the application of the approximate computing paradigm to the softmax and squash functions is to reduce hardware costs at the expense of lower quality classification results, in the context of error-tolerant applications. As a matter of fact, a wide variety of applications exists for which a circuit that produces some incorrect outputs may be still acceptable, provided that the error rate is kept within application-specific thresholds.