This work presents a framework for Deep Neural Networks compression targeting on-edge inference on low power embedded systems. The work first shows the limitation of state-of-the-art compression techniques, which do not consider the target hardware in the optimization loop. Then, it presents an hardware-aware compression tool based on network pruning and knowledge distillation. As key features, the proposed technique (i) provides an optimized model which satisfies the hardware constraints while maximizing the target accuracy (ii) it does not need a labelled dataset. Experimental results highlight the necessity of a two-step approach, in which both model compression and model fine-tuning have a critical relevance. Results also show that the presented framework offers better results with respect to many SoA proposals.