With the dawn of powerful and energy-efficient hardware in the low-power computing space, there is the need to decide to go either for performance or for efficiency. Although not completely exclusive to each other there is usually a winner during practical applications. However, the chosen solution might not really work due to energy constraints set by the limited power sources of mobile devices. The performance capabilities of these edge devices are severely limited to thermal and energy constraints. For the sake of improving efficiency and accuracy in machine learning and running neural networks onboard such portable devices we are faced with the following simple yet brutal choice.