Today in many applications the edge devices are used from cloud computing, internet of things (IoT) to manufacturing sectors to monitor, analyse processes through applying machine learning and other algorithms. The edge devices usually run light software’s like quantized and small machine learning algorithms due to their limited performance, energy consumption and memory. The algorithms containing the matrix to vector multiplication operation, especially, the quantized fully connected stage of Neural Networks have weighted matrices which are usually consists of high percentage of zero elements. For the sake of reduction of resource usage, energy consumption and increasing the performance, the only non-zero elements can be used in these operations.