Accelerating Transformer Deep Learning Models on FPGAs using High-Level Synthesis

Accelerating Transformer Deep Learning Models on FPGAs using High-Level Synthesis In the current electronic industry, logic synthesis that starts from RTL description has been the superior method to imlong-termigital systems on both FPGAs and application-specific chips. But recently, High-Level Synthesis (HLS) has grown and now is the choice of hardware engineers and designers for the implementation of complex digital systems. High-Level Synthesis or HLS is an automatic process that accepts synthesizable code written using high-level languages such as C, SystemC, OpenCL (Open Computing Language), and C++ and then transforming them into an RTL design. Finally, This design is then implemented on hardware devices such as FPGAs. FPGA has limited resources of hardware in terms of the logic cell, interconnection which contains wires that are routed to the power supply, clock, and signal nets. In terms of language translation (Italian to English or vice versa) natural language processing, RNN (Recurrent Neural Networks) can be used but this method severely suffers from two issues: incapable of capturing very long term dependencies and also unable in order to parallelizing sequential computation flow. Consider that, models with multi-head attention such as Transformer have extreme effectiveness in order to capture the long-term dependencies in a variety of sequence modeling tasks. Here in this project Transformers applied on FPGA in terms of performing and analyzing time, area, and power. The network designed with C++ and applied through the Vivado HLS tools on the FPGA board. this work has been depicted by designing a customized hardware accelerator for the Transformer by using a High-Level Synthesis. The tool is provided by Xilinx which is called Vivado HLS. This accelerator needs to be implemented on the board. For this, the PYNQ board has been chosen. It has a dual-core Cortex A9 processor.