To date, neural networks are fundamental and are applied in numerous fields. Nevertheless, in the literature hardware acceleration focuses almost exclusively on matrix and convolutional operations, while activation functions are often neglected and systematic comparisons among alternative implementation techniques are rare. The optimization of activation functions is particularly relevant in recurrent neural networks, such as LSTMs, in which sigmoid and tanh are executed at every cycle and significantly affect latency, power consumption, and area occupation. This work aims to carry out a comparative study of optimization techniques for sigmoid and tanh by analyzing four approximation algorithms selected from the literature: 2B (base-2), PWL (Piecewise Linear), IBERT, and CORDIC.