KR102600086B1 - A Resource Efficient Integer-Arithmetic-Only FPGA-Based CNN Accelerator for Real-Time Facial Emotion Recognition - Google Patents

Abstract

A resource-efficient integer-arithmetic-only FPGA-based CNN accelerator and its operation method for real-time facial emotion recognition are presented. The operating method of the FPGA-based CNN accelerator implemented on a heterogeneous SoC platform including a CPU (Processing System; PS) and an FPGA (Programmable Logic, PL) proposed in the present invention is to use a plurality of CNN accelerators in the CPU to improve CNN facial expression recognition performance. Creating a training dataset that combines image processing algorithms and converting parameters through floating-point training for the generated training dataset. Regarding the parameters converted through floating-point training on the CPU. Extracting quantized parameters through quantization recognition training, performing Integer-Arithmetic-Only CNN reconstruction, and FPGA (Programmable Logic; PL) and CPU (Processing) on heterogeneous SoC (System on Chip) platforms and performing real-time facial emotion recognition using the integer-arithmetic-only CNN in the System; PS) domain.

Description

{A Resource Efficient Integer-Arithmetic-Only FPGA-Based CNN Accelerator for Real-Time Facial Emotion Recognition}

The present invention relates to a resource-efficient integer-arithmetic dedicated FPGA-based CNN accelerator for real-time facial emotion recognition and a method of operating the same.

Computers play a central role in industry and society, and are quickly becoming a part of everyday life. Accordingly, the need for research on interaction between humans and computers is increasing. For smooth interaction between humans and computers, computers must be able to analyze human intentions and react accordingly. Emotions expressed in facial expressions are a universal and effective way to express human intentions. Analyzing a person's intentions through facial expressions is called facial expression recognition technology, and is used in various fields, including the automobile and robot industries. In order to accurately understand the expressions on a person's face, a computer must recognize the face and automatically classify the expressions according to specific expression groups.

In the prior art, seven basic emotions are defined: happiness, anger, fear, surprise, disgust, sadness, and neutrality, and it can be confirmed that the basic emotions are recognized in the same way regardless of human culture. In the field of computer vision, FACS (Facial Action Coding System), a facial expression analysis method proposed in the prior art, was used as a representative facial emotion recognition model. FACS transforms the basic movements of facial muscles into AU (Action Units) and then uses a combination of AUs to recognize facial expressions. However, the development of real-time automation systems has problems due to inadequate computational power and inefficient preprocessing algorithms.

With the development of image processing algorithms and improvements in computing power, facial expression recognition technology has developed into a method of extracting manually created features through a three-phase pipeline. This approach consists of image preprocessing, feature extraction, and expression classification steps. The image preprocessing step uses filtering and histogram equalization to remove irrelevant information and enhance information related to facial expressions. The feature extraction step extracts facial features from the image using feature extractors such as Gabor Wavelet Kernel, Local Binary Pattern (LBP), Active Shape Model (ASM), and Harr-Like Feature Template. Extract . The expression classification step uses a classifier such as Support Vector Machine (SVM), KNN (K-Nearest Neighbor), or AdaBoost to classify the extracted features into expression groups. Methods using hand-crafted features require designing appropriate feature extractors and expression classifiers separately, so the two steps cannot be optimized simultaneously. Additionally, external factors (e.g., posture changes, occlusions, lighting) can cause significant performance degradation.

Due to recent breakthroughs in hardware technology and big data technology, deep learning-based methods are being applied to a variety of applications because they show excellent performance in complex and complex problems. In particular, in computer vision such as image classification and object detection, methods using Convolutional Neural Networks (CNN) have shown excellent performance. Unlike existing methods, CNN-based approaches are more robust in noisy environments, significantly reducing reliance on image preprocessing and feature extraction. Additionally, this method can optimize parameters at once through end-to-end training. In facial expression recognition research, CNN-based methods have shown higher performance than existing methods, but use many convolutional layers when building the network. A network like this requires a lot of work and memory footprint. Therefore, its use in applications that require real-time processing, such as advanced driver assistance systems (ADAS), is limited. Additionally, it is not suitable for environments with limited hardware resources, such as virtual reality (VR).

[1] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, "Learned step size quantization," 2019, arXiv:1902.08153. [Online]. Available: https://arxiv.org/abs/1902.08153 [2] S. R. Jain, A. Gural, M. Wu, and C. H. Dick, “Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks,” in Proc. Mach. Learn. Syst., Mar. 2020, pp. 112-128. [3] H. Phan-Xuan, T. Le-Tien, and S. Nguyen-Tan, “FPGA platform applied for facial expression recognition system using convolutional neural networks,” Procedia Comput. Sci., vol. 151, pp. 651-658, Jan. 2019. [4] T. N. D. Phuc, N. N. Nhut, N. Trinh, and L. Tran, “Facial expression recognition system using FPGA-based convolutional neural network,” in Research in Intelligent and Computing in Engineering, vol. 1254. Singapore: Springer, Jan. 2021, pp. 341-351. [5] P. T. Vinh and T. Q. Vinh, “Facial expression recognition system on SoC FPGA,” in Proc. Int. Symp. Electr. Electron. Eng. (ISEE), Ho Chi Minh City, Vietnam, Oct. 2019, pp. 1-4. [6] R. Ding, G. Su, G. Bai, W. Xu, N. Su, and X. Wu, “A FPGA-based accelerator of convolutional neural network for face feature extraction,” in Proc. IEEE Int. Conf. Electron Devices Solid-State Circuits (EDSSC), Xi'an, China, Jun. 2019, pp. 1-3. [7] R. Ding, X. Tian, G. Bai, G. Su, and X. Wu, “Hardware implementation of convolutional neural network for face feature extraction,” in Proc. IEEE 13th Int. Conf. ASIC (ASICON), Chongqing, China, Oct. 2019, pp. 1-4. [8] Y. Bengio, N. Leonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” 2013, arXiv:1308.3432. [Online]. Available: https://arxiv.org/abs/1308.3432 [9] K. Abdelouahab, M. Pelcat, J. Serot, and F. Berry, “Accelerating CNN inference on FPGAs: A survey,” 2018, arXiv:1806.01683. [Online]. Available: https://arxiv.org/abs/1806.01683 [10] L. Zhu. THOP: PyTorch-OpCounter. Accessed: Apr. 26, 2021. [Online]. Available: https://github.com/Lyken17/pytorch-OpCounter [11] J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan, ''PACT: Parameterized clipping activation for quantized neural networks,'' 2018, arXiv:1805.06085. [Online]. Available: https://arxiv.org/abs/1805.06085 [12] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, ''Quantization and training of neural networks for efficient integer-arithmetic- only inference,'' in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition., Jun. 2018, pp. 2704-2713. [13] S. Jung, C. Son, S. Lee, J. Son, J. J. Han, Y. Kwak, and C. Choi, ''Learning to quantize deep networks by optimizing quantization intervals with task loss,'' in Proc . IEEE/CVF Conf. Comput. Vis. Pattern Recognition., Jun. 2019, pp. 4350-4359. [14] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, ''Learned step size quantization,'' 2019, arXiv:1902.08153. [Online]. Available: https://arxiv.org/abs/1902.08153 [15] S. R. Jain, A. Gural, M. Wu, and C. H. Dick, ''Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks,'' in Proc. Mach. Learn. Syst., Mar. 2020, pp. 112-128. [16] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, ''SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and

The technical problem to be achieved by the present invention is to provide a resource-efficient integer-arithmetic dedicated FPGA-based CNN accelerator and its operating method for real-time facial emotion recognition to reduce memory installation space and computational complexity. In addition, we propose a new hardware-friendly quantization method using only integer arithmetic through the CNN accelerator according to an embodiment of the present invention, and a new training dataset created using various image processing algorithms to improve the generalization and classification performance of CNN. Provides a method and device for generating a FERPlus-A dataset and performing quantization after training.

In one aspect, the operating method of the FPGA-based CNN accelerator implemented on a heterogeneous SoC (System on Chip) platform including a CPU (Processing System; PS) and an FPGA (Programmable Logic, PL) proposed in the present invention includes the CNN accelerator in the CPU. Creating a training dataset combining multiple image processing algorithms to improve facial expression recognition performance, and converting parameters through floating-point training for the generated training dataset, floating on the CPU Extracting quantized parameters through quantization recognition training for parameters converted through point training and performing Integer-Arithmetic-Only CNN reconstruction, and performing the above in the FPGA and CPU areas of heterogeneous SoC platforms It includes performing real-time facial emotion recognition using an integer-arithmetic-only CNN.

The step of generating a training dataset combining a plurality of image processing algorithms in the CPU to improve CNN facial expression recognition performance and converting parameters through floating-point training for the generated training dataset includes the first calculation module ( The operation is performed through a basic calculation block including FireA) and a second calculation module (FireB), and each of the first calculation module (FireA) and the second calculation module (FireB) reduces the number of channels of the feature map, It includes a squeeze layer to reduce the number of operations in the subsequent expansion layer and an expansion layer to expand the channel again, and a squeeze layer of the first calculation module and a squeeze layer of the second calculation module to minimize accuracy degradation by maintaining the reception field. Squeeze layers have different kernel sizes.

In order to halve the resolution of the feature map and replace the max pooling layer, the stride of the basic calculation block is set to 2, and the features extracted at the end of the CNN are classified and a fully connected layer containing multiple parameters is global averaged. The above parameters are reduced by replacing it with pooling and a convolution layer with a kernel size of 1, and the CNN classifier minimizes the number of channels to match the number of classes through the convolution layer and then reduces the resolution of the feature map to one pixel. It is compressed, and a batch normalization layer is inserted between the convolution layer and the activation function to improve the convergence stability and performance of CNN.
The step of extracting quantized parameters through quantization recognition training for parameters converted through floating-point training in the CPU and performing integer-arithmetic-only CNN reconstruction is performed using the Learned Step Size Quantization (LSQ) method. ) is used to set a trainable parameter as a reference factor and train the reference factor together with other parameters, thereby eliminating the need for a pre-correction process for the reference factor in the quantization process and creating a trained quantization threshold (Trained Quantization). By using the Threshold (TQT) method to set the trainable parameter to a logarithmic threshold (log ₂ t) and mapping the standardization factor to a power of 2 term, multiplication and division using the standardization factor can be performed as a shift operation ( shift operations) and the computational overhead for zeros that occur during uniform quantization by using symmetric quantization in the process of performing Quantization-Aware Training (QAT) using pre-trained floating-point parameters. is removed, and the Log Level Threshold Quantization (LLTQ) method is used, which performs quantization with one standardization factor for all elements of a given parameter or input tensor by introducing scaling for each layer.

delete

In another aspect, in the FPGA-based CNN accelerator implemented on a heterogeneous SoC (System on Chip) platform including a CPU (Processing System; PS) and an FPGA (Programmable Logic, PL) proposed in the present invention, the CPU To improve CNN facial expression recognition performance, a training dataset is created by combining multiple image processing algorithms, and parameters are converted through floating-point training for the generated training dataset, and floating-point For parameters converted through training, quantized parameters are extracted through quantization recognition training, Integer-Arithmetic-Only CNN reconstruction is performed, and the integer-arithmetic-only CNN is reconstructed in the FPGA and CPU areas of heterogeneous SoC platforms. Real-time facial emotion recognition is performed using an arithmetic-only CNN.

Memory installation space and computational complexity can be reduced through a resource-efficient integer-arithmetic dedicated FPGA-based CNN accelerator and its operation method for real-time facial emotion recognition according to embodiments of the present invention. In addition, we propose a new hardware-friendly quantization method using only integer arithmetic through a CNN accelerator according to an embodiment of the present invention, and generate and train the FERPlus-A dataset, a new training dataset created using various image processing algorithms. By performing post-quantization, the generalization and classification performance of CNN can be improved.

1 is a diagram illustrating the operation method of a resource-efficient integer-arithmetic-only FPGA-based CNN accelerator for real-time facial emotion recognition according to an embodiment of the present invention.
Figure 2 is a diagram illustrating a process for optimizing memory usage and computational complexity of a lightweight CNN architecture according to an embodiment of the present invention.
Figure 3 is a diagram for explaining the uniform quantization process using the quantization method LLTQ according to an embodiment of the present invention and the LSQ and TQT quantization processes of the prior art.
Figure 4 is a diagram for explaining an integer scale conversion method according to an embodiment of the present invention.
Figure 5 shows the architecture of a resource-efficient integer-arithmetic dedicated FPGA-based CNN accelerator for real-time facial emotion recognition according to an embodiment of the present invention.
Figure 6 is a diagram for explaining the entire task scheduling process for an integer-arithmetic dedicated FPGA-based CNN accelerator according to an embodiment of the present invention.

Recently, much research has been conducted on facial expression recognition using Convolutional Neural Networks (CNN), which show excellent performance in computer vision. To achieve high classification accuracy, a CNN architecture with many parameters and high computational complexity is required. However, it is not suitable for embedded systems with limited hardware resources.

The present invention proposes a lightweight CNN architecture optimized for embedded systems. The proposed CNN architecture has a small memory footprint and low computational complexity. Additionally, we propose a new hardware-friendly quantization method that uses only integer arithmetic. The proposed hardware-friendly quantization method maps the scaling factor to a power of 2 term and replaces multiplication and division operations using the scaling factor with shift operations.

Additionally, the FERPlus-A dataset is created to improve the generalization and classification performance of CNN. This is a new training dataset created using various image processing algorithms. After training the FERPlus-A dataset with FERPlus-A, quantization is performed.

According to an embodiment of the present invention, the size of the quantized CNN parameters is about 0.39 MB, and the number of operations is about 28M Integer Operations (IOP). The performance of the quantized CNN was evaluated using only integer arithmetic on the FERPlus test dataset, and the classification accuracy was approximately 86.58%, which was higher than that of other lightweight CNNs in the prior art. The proposed CNN architecture using only integer arithmetic is implemented for real-time facial expression recognition by applying parallelization strategy and efficient data caching strategy on Xilinx ZC706 SoC platform. An FPGA-based CNN accelerator implemented for real-time facial expression recognition achieves approximately 10 frames per second (FPS) at 250MHz and consumes 2.3W.

In several facial expression recognition techniques using lightweight CNNs, the prior art used crowdsourcing and label distribution methods to construct the FERPlus dataset, which solved the problems of the FER2013 dataset, and four different training techniques using lightweight VGG were proposed. It has been done. In another prior art, identification features and facial expression features were extracted through DeepID2 and ResNet, respectively, and the features of each CNN output were combined and classified. Another prior art proposed a transfer learning and ensemble method that uses pre-trained parameters with a dataset frequently used for image classification. Another prior art proposed an approach to train MobileNetv2 with a dataset that integrates the Sobel gradient and Laplacian of the original image dataset. Another prior art used a training method that adds center loss to the global loss to improve the discrimination power of CNN.

In another prior art, two custom CNNs were constructed using Gradient Weighted Class Activation Mapping (Grad-CAM). They also proposed a dual-integration CNN that combines features extracted from two CNNs and uses them for facial expression recognition. Another prior art proposed SHCNN, which can alleviate the overfitting problem of relatively small datasets and recognize static and micro expressions simultaneously. Another prior art introduced a method of quantizing the contributions of various facial regions using Class Activation Mapping (CAM) in features extracted through DenseNet. In another prior art, a new dataset was created that integrates two similar facial expression groups into one. Here, we proposed a method to train DenseNet by applying face detection and face alignment methods.

However, other prior technologies took up a large amount of memory space and had the problem of high computational complexity. In some prior art, CNNs were used with smaller parameters, but because two CNNs were used for inference, they still occupied a large amount of memory space and remained unsuitable for real-time processing. In some prior technologies, CNNs have small parameters and low computational complexity, but their facial expression recognition accuracy is very low. Even the prior art using a small number of parameters was not suitable for real-time processing because it required too much memory space due to dense connections. This is the characteristic of DenseNet. To solve these problems, techniques are needed to design a lightweight CNN architecture by optimizing memory footprint and computational complexity while maintaining the accuracy of facial expression recognition.

Quantization is a method of converting parameters trained as 32-bit floating-point (FP) to low-bit fixed-point (FX) or integer (INT). Applying quantization can optimize memory usage and reduce the number of calculations using simple hardware. Quantization is divided into two methods. Post-Training Quantization (PTQ) methods minimize quantization errors by applying corrections at the time of meeting using pre-trained parameters. The Quantization-Aware Training (QAT) method retrains parameters by considering the effect of quantization.

In the prior art, bias correction and cross-layer equalization methods were proposed to resolve biased weight errors and imbalances during the quantization process. Another prior art introduced a method of minimizing the quantization error using the minimum mean square error between the floating-point value and the quantization error. In another prior art, quantization error was minimized by calculating the optimal clipping value using Analytical Clipping for Integer Quantization (ACIQ). Another prior art proposed a distillation dataset using the mean and variance of a batch normalization (BN) layer to obtain the optimal quantization range.

The PTQ method does not require a retraining process, saving computing resources and optimization time and enabling rapid deployment. However, in bias correction and cross-layer equalization methods and methods that minimize quantization error using the minimum mean square error between floating-point values and quantization error, fine-tuning values are required to minimize accuracy loss. In the ACIQ method and the BN method, a different number of bits must be allocated to each channel to minimize accuracy degradation. In other prior art, a separate standardization factor must be obtained for each channel to maintain accuracy. Additionally, since all PTQ methods require floating-point parameters, dedicated operators are needed when implementing hardware.

Another prior art proposed a method of minimizing quantization error by defining a Parameterized Clipping Activation (PACT) function to find an appropriate standardization factor. Another prior art defined the operation of a quantizer that performs matrix multiplication using only integer-arithmetic and proposed a hierarchical fusion method. Another prior art proposed a method of minimizing work loss using a parameterized quantization interval. Another prior art proposed a method of learning optimal quantization mapping by making the standardization argument into a learnable parameter. Another prior art proposed a method of mapping a scaling factor to a power of 2 term using a scaling factor conversion equation.

The QAT method can achieve higher accuracy than the PTQ method by training the quantization parameter with other parameters. However, in some prior technologies, the BN layer was not quantized. In some prior technologies, the parameters required for quantization are floating-points, and dedicated operators are required when implementing hardware. Other approaches require calibration of the activation criterion factor before starting QAT. To compensate for this problem, a hardware-friendly quantization method that does not require a pre-compensation step is needed.

Performing facial expression recognition using GPUs provides the best performance in terms of throughput and speed. However, using GPUs in embedded systems is difficult. These systems require solutions that reduce energy consumption and hardware resource usage. Field Programmable Gate Arrays (FPGAs) have the potential to allow designers to tailor their designs to their applications. It is suitable for real-time processing of embedded systems because it can be programmed for optimal operating speed and reasonable power consumption.

Because of these advantages, several FPGA-based CNN accelerators have been proposed for facial expression recognition. In the prior art, a CNN accelerator was implemented on the Xilinx Zynq-XC7Z020 FPGA using High Level-Synthesis (HLS).

To reduce Block RAM (BRAM) usage, we designed a CNN accelerator that stores layer results and inputs in DRAM and used VDMA for high-speed DRAM access. In another prior art, a CNN accelerator was designed by applying an efficient structure and data preprocessing method for facial emotion recognition and implemented on Altera DE-10 FPGA using Verilog HDL. In another prior art, an accelerator comprised the processing engine core of an Altera DE-10 FPGA to accelerate convolution operations. Another prior art proposed an Altera Cyclone-V FPGA accelerator using Verilog HDL with different parallel processing for the various convolutional layers that make up the CNN. In another prior art, a CNN accelerator was implemented on an Altera Cyclone-V FPGA using Verilog HDL. They designed a configurable convolutional computing array to maximize DSP usage in FPGAs.

Some prior art implementations of simple CNN architectures include fully connected, convolutional, and pooling layers. Although the number of parameters and computational complexity were low, the accuracy was also low, making it unsuitable for applications that require highly accurate facial expression recognition. In some prior technologies, the DeepID architecture is implemented in FPGAs, which require a lot of hardware resources and cannot be used in embedded systems. Therefore, it is necessary to design a low-power, low-cost FPGA-based CNN accelerator for real-time facial expression recognition and maintain high facial expression recognition performance.

1 is a diagram illustrating the operation method of a resource-efficient integer-arithmetic-only FPGA-based CNN accelerator for real-time facial emotion recognition according to an embodiment of the present invention.

The operation method of a resource-efficient integer-arithmetic-only FPGA-based CNN accelerator for real-time facial emotion recognition is achieved by optimizing the memory footprint and computational complexity while maintaining the accuracy of facial expression recognition. Additionally, Log Level Threshold Quantization (LLTQ) is proposed to solve the shortcomings of the prior art [1], [2]. Additionally, a low-power, low-cost FPGA-based CNN accelerator is proposed for real-time facial expression recognition.

The operating method of the proposed FPGA-based CNN accelerator implemented on a heterogeneous SoC platform including CPU (Processing System; PS) and FPGA (Programmable Logic, PL) is to use multiple image processing algorithms to improve CNN facial expression recognition performance in the CPU. A step of generating a training dataset that combines and converting parameters through floating-point training for the generated training dataset (110), with respect to the parameters converted through floating-point training in the CPU. Step 120 of extracting quantized parameters through quantization recognition training and performing Integer-Arithmetic-Only CNN reconstruction, and FPGA (Programmable Logic; PL) of heterogeneous System on Chip (SoC) platforms and It includes a step 130 of performing real-time facial emotion recognition using the integer-arithmetic dedicated CNN in the CPU (Processing System; PS) domain.

In step 110, the CPU generates a training dataset combining a plurality of image processing algorithms to improve CNN facial expression recognition performance (111), and uses a new lightweight CNN architecture (112) according to an embodiment of the present invention. Thus, parameters are converted through floating-point training for the generated training dataset (113), and a PTH File is created (114). PTH File according to an embodiment of the present invention is floating-point trained parameters.

In step 110, an operation may be performed through a basic calculation block including a first calculation module (FireA) and a second calculation module (FireB). The first calculation module (FireA) and the second calculation module (FireB) each include a squeeze layer to reduce the number of channels in the feature map and the number of operations in the subsequent expansion layer, and an expansion layer to expand the channels again. Includes. At this time, in order to maintain the reception field and minimize accuracy degradation, the squeeze layer of the first calculation module and the squeeze layer of the second calculation module have different kernel sizes.

According to an embodiment of the present invention, the resolution of the feature map can be reduced by half and the stride of the basic calculation block can be set to 2 to replace the maximum pooling layer. In addition, the parameters can be reduced by classifying the features extracted at the end of the CNN, replacing the fully connected layer containing multiple parameters with global average pooling, and replacing the convolution layer with a kernel size of 1.

According to an embodiment of the present invention, the CNN classifier minimizes the number of channels to match the number of classes through a convolution layer and then compresses the resolution of the feature map to one pixel, in order to improve the convergence stability and performance of the CNN. A batch normalization layer can be inserted between the convolution layer and the activation function.

In step 120, quantized parameters are extracted through quantization recognition training (121) for the parameters converted through floating-point training in the CPU (122), and an Integer-Arithmetic-Only CNN is used. Perform reconstruction (123) and create a BIN File (124). BIN File according to an embodiment of the present invention is serialized integer parameters.

The log-level threshold quantification (LLTQ) method according to an embodiment of the present invention uses a learned step size quantization (LSQ) method to set a trainable parameter as a standardization factor and set the standard together with other parameters. By training the quantization factor, the quantization process does not require a pre-calibration process for the standardization factor, and uses the Trained Quantization Threshold (TQT) method to set the trainable parameter to a logarithmic threshold (log ₂ t ) and mapping the scaling factor to a power-of-2 term, replacing multiplication and division using the scaling factor with shift operations, and training quantization recognition using pre-trained floating-point parameters. By using symmetric quantization in the process of performing Quantization-Aware Training (QAT), the computational overhead for zero points that occurs during uniform quantization is eliminated, and by introducing layer-wise scaling, for all elements of a given parameter or input tensor. Quantization is performed with one standardization argument.

In step 130, real-time face analysis is performed using the integer-arithmetic-only CNN in the FPGA (Programmable Logic; PL) area 131 and the CPU (Processing System; PS) area 132 of the heterogeneous System on Chip (SoC) platform. Perform emotion recognition.

The XSA File (143) can be used in the FPGA area (131) according to the embodiment of the present invention. The XSA (Xilinx support Ardhive) file creation process 140 can be created through the HLS compiler 141 and CNN accelerator logic 142.

The ELF File (153) can be used in the CPU area (132) according to the embodiment of the present invention. The ELF (Executable and linkable) File creation process 150 can be created through a standard C/C++ compiler 151 and a device driver 152.

As such, the CNN accelerator logic 142 and device driver 152 according to an embodiment of the present invention were implemented in the PL and PS areas of heterogeneous SoC platforms for real-time facial expression recognition applications.

Figure 2 is a diagram illustrating a process for optimizing memory usage and computational complexity of a lightweight CNN architecture according to an embodiment of the present invention.

Referring to Figure 2, it shows a lightweight CNN architecture according to an embodiment of the present invention consisting of 21 convolutional layers and one global average pooling layer. The computation block contains two basic computation modules, FireA and FireB. The parameters of the next calculation block, the number of filters in the squeeze layer (s) and the number of filters in the expansion layer (e), are doubled in the next calculation block. A CNN according to an embodiment of the present invention was designed by applying the following techniques: basic computational modules, all convolutional and regular networks, and power-of-two resolution and quadruple output channels.

First, a basic operation module according to an embodiment of the present invention will be described.

The lightweight CNN according to an embodiment of the present invention applies the fire module proposed in SqueezeNet. The fire module consists of two types of layers: The Squeeze Layer (SQ) can reduce the number of channels in the feature map and reduce the number of operations in the subsequent expansion layer. The expansion layer plays the role of expanding the channel again and consists of two layers: a layer with a kernel size of 3 and a layer with a kernel size of 1. The output feature map of SQ is divided into two paths and input as EX1 and EX3, respectively. Additionally, the output feature maps of EX1 and EX3 are merged through channel-specific connection and then entered the next layer.

The lightweight CNN according to an embodiment of the present invention uses a modified fire module as the basic calculation module and is composed of two types (FireA and FireB). The expansion layer configuration of FireA and FireB is the same. However, the kernel size of SQ is set to 3 in FireA and 1 in FireB. The reason for setting the kernel size to 3 in FireA is to minimize accuracy degradation by maintaining appropriate reception fields.

Next, all convolutional and regular networks according to embodiments of the present invention will be described.

Since the lightweight CNN according to an embodiment of the present invention uses only convolutional layers with a kernel size of 3 or 1, there is no need to consider acceleration methods for other kernel sizes when implemented in hardware. The max pooling layer reduces the resolution of the feature map by half and compresses the information. Because the maximum value of the feature map must be found, a bottleneck may occur when implemented in hardware. The proposed CNN sets the stride of FireA to 2. This halves the resolution of the feature maps and replaces the max pooling layer.

The Fully-Connected (FC) layer according to an embodiment of the present invention classifies the features extracted at the end of the CNN and includes many parameters. The number of parameters can be reduced by replacing the FC layer with global average pooling and a convolutional layer with a kernel size of 1. The constructed classifier minimizes the number of channels to match the number of classes through a convolutional layer, and then compresses the resolution of the feature map to one pixel. The activation function of every layer is labeled ReLU. When implemented in hardware, the ReLU function is very efficient because it uses only sign bit comparisons. In this way, all layers of CNN were made of convolutional layers and designed as a very general network. Additionally, a BN layer was inserted between the convolution layer and the activation function to improve CNN convergence stability and performance. Since the BN layer can be combined with the convolutional layer through BN layer fusion, the number of additional operations and parameters generated by the BN layer can be neglected when implemented in hardware.

Next, power-of-2 resolution and quadruple output channels according to an embodiment of the present invention will be described.

According to an embodiment of the present invention, by making both the width and height of the feature map a power of 2 term, the multiplication and division operations in the part where the address of the feature map is calculated can be replaced with a move operation. Additionally, the output channels of the convolutional layer were all set to multiples of 4 for efficient parallel multi-accumulate (MAC) hardware operation.

Figure 3 is a diagram for explaining the uniform quantization process using the quantization method LLTQ according to an embodiment of the present invention and the LSQ and TQT quantization processes of the prior art.

A log-level threshold quantization (LLTQ) technique according to an embodiment of the present invention will be described.

The Learned Step Size Quantization (LSQ) method [1] and the Trained Quantization Threshold (TQT) method [2] use uniform quantization. A CNN with a uniform quantizer inserted performs quantization-aware training (QAT) using pre-trained floating-point parameters.

Figure 3 shows the structure of the convolutional layer 330 into which the uniform quantizer 320 is inserted and the four-step operation of the quantizer 310. The first step, the scale process, the quantizer maps the real value range to the integer value range of the specified bits. The second step, called the round process, converts all real values mapped to a range of integer values to integers. The third step is the clamp process. This process removes elements that exceed the quantization range to match the quantization level. When the quantization bit is n, the value after the clamp process is located inside the Bound, as shown in (1).

(One)

The fourth step, the de-quant process, converts values mapped to the integer range into floating-point values subject to quantization. The reason for converting quantized values from integers back to floating-points is that most existing deep learning frameworks using GPUs are optimized for floating-point training. The overall equation for the forward path can be described as follows.

(2)

Here, x _f is a floating-point value that is the input of the quantizer, and x _d is the output value of the quantizer that has passed the inverse-quantization process. The clamp(x, Bound) function is the second stage of the uniform quantizer and truncates the rounded value (x) to fit within the bounds. The round(·) function is the third stage of the uniform quantizer. Round the actual data to the nearest integer. Discontinuities occurring in the rounding process were resolved using the Straight-Through Estimator (STE) [8].

The LSQ [1] method sets a trainable parameter as a baseline argument and trains the baseline argument together with other parameters. Therefore, LSQ does not need to convert the reference factor in the quantization process. This is well known for use in the proposed lightweight CNN, but has the disadvantage of requiring dedicated operators when implementing it in hardware. This is because it is a floating-point standardization argument. The TQT [2] method sets the trainable parameters to a logarithmic threshold (log ₂ t) and maps the criterion factor to a power of 2 term as follows.

(3)

Here, q_level is a value that adds 1 to the maximum Bound value in equation (1).

Because TQT maps the standardization factor to a power of 2 term, it has the advantage of being able to replace multiplication and division using the standardization factor with shift operations. However, before starting retraining, the activation standardization factor needs to be calibrated. Therefore, the present invention proposes a new hardware-friendly quantization method that solves the problems of LSQ and TQT in previous studies.

The proposed LLTQ can map trainable parameters directly to power-of-2 factors. Also, unlike TQT, the LLTQ method uses the Learned Step Size Quantization (LSQ) method to set a trainable parameter as a standardization factor and train the standardization factor along with other parameters in the quantization process. Instead of requiring a pre-calibration process for the reference factor, the trainable parameter is set to a logarithmic threshold (log ₂ t) using the Trained Quantization Threshold (TQT) method and the reference factor to power-of-2 terms, replacing multiplication and division using scaling factors with shift operations, and Quantization-Aware Training (QAT) using pre-trained floating-point parameters. By using symmetric quantization in the process of performing, the computational overhead for zeros that occurs during uniform quantization is eliminated, and by introducing layer-by-layer scaling, quantization is performed with one baseline factor for a given parameter or all elements of the input tensor. Perform.

The performance of the LLTQ method according to an embodiment of the present invention is evaluated using only integer arithmetic through an integer scale transformation method. LLTQ sets the trainable parameter to a level threshold (level_th) and maps the criterion factor to a power-of-2 term using the following equation:

(4)

Here, q_level is the same as in equation (3). Calculating the standardization factor requires many operations. However, when performing inference using a CNN reconfigured to use only integer arithmetic, only the amount of movement of the scaling factor mapped to a power-of-2 term is needed.

Additionally, various hardware-friendly quantization methods were applied to LLTQ. By using symmetric quantization, the computational overhead for zeros that occurs during uniform quantization is eliminated. By introducing layer-wise scaling, quantization can be performed with one baseline factor for a given parameter or all elements of the input tensor. BN layer fusion, which combines the parameters of the convolution layer and the BN layer, was applied. This method does not require quantization in the BN layer. This can reduce the number of operations and inference time by eliminating the multiplication and division operations required for the BN layer. The fire module is structured so that the output of the squeeze layer goes into the input of two expansion layers. In floating-point, the inputs of both expansion layers are the same. However, in QAT, each convolutional layer requires a different normalization factor. Therefore, the squeeze layer must calculate the output of each expansion layer separately. This can cause a bottleneck when performing CNN inference in hardware. To solve this problem, QAT was performed by making the quantizers of both expansion layers into a single shared quantizer.

Log-level threshold quantization (LLTQ) according to an embodiment of the present invention uses the learned step size quantization (LSQ) method [1] and the trained quantization threshold (TQT) method [2] uses uniform quantization. A CNN with a uniform quantizer embedded performs quantization-aware training (QAT) using pre-trained floating-point parameters.

Figure 3 shows the structure of a convolutional layer with a uniform quantizer inserted and the four-step operation of the quantizer. The first step, the scale process, the quantizer maps the real value range to the integer value range of the specified bits. The second step, called the round process, converts all real values mapped to a range of integer values to integers. The third step is the clamp process. This process removes elements that exceed the quantization range to match the quantization level. When the quantization bit is n, the value after the clamp process is located inside the Bound as shown in equation (1).

The fourth step, the de-quant process, converts values mapped to the integer range into floating-point values subject to quantization. The reason for converting quantized values from integers back to floating-points is that most existing deep learning frameworks using GPUs are optimized for floating-point training. The overall equation for the forward path can be described as follows.

Here, xf is the floating-point value that is the input of the quantizer, and xd is the output value of the quantizer that passed the de-quant process. The clamp(x, Bound) function is the second stage of the uniform quantizer and truncates the rounded value (x) to fit within the bounds. The round(·) function is the third stage of the uniform quantizer. Round the actual data to the nearest integer. Discontinuities arising from the rounding process were resolved using the straight-through estimator (STE) [8].

The LSQ [1] method sets a trainable parameter as a baseline argument and trains the baseline argument together with other parameters. Therefore, LSQ does not need to convert the reference factor in the quantization process. Although it is well known for use in lightweight CNNs described in Section III-B, it has the disadvantage of requiring dedicated operators when implementing it in hardware. This is because it is a floating-point standardization argument. The TQT [2] method sets the trainable parameters to a logarithmic threshold (log ₂ t) and maps the criterion factor to a power of 2 term as follows.

Here, q_level is a value that adds 1 to the maximum Bound value of (1).

Because TQT maps the standardization factor to a power of 2 term, it has the advantage of being able to replace multiplication and division using the standardization factor with shift operations. However, before starting retraining, the activation standardization factor needs to be calibrated. Therefore, in this paper, we propose a new hardware-friendly quantization method that solves the problems of previous studies [1], [2]. The proposed LLTQ can map trainable parameters directly to power-of-two factors. Additionally, unlike TQT [2], the LLTQ method allows the activation criterion factor to be initialized with the activation data statistic values of the first batch of the first epoch, thus eliminating the need for a pre-calibration process. Additionally, by mapping the scaling factor to a power-of-2 term, the multiplication and division operations that occur in scale, as well as de-quant processes, can all be replaced by shift operations. The performance of the LLTQ method is evaluated using only integer arithmetic via the integer scale transformation method described in Section III-C-2. LLTQ sets the trainable parameter to a level threshold (level_th) and maps the criterion factor to a power-of-2 term using the following equation:

Here, q_level is the same as in (3). Calculating the standardization factor requires many operations. However, when performing inference using a CNN reconfigured to use only integer arithmetic, only the amount of movement of the scaling factor mapped to a power-of-2 term is needed.

Additionally, several hardware-friendly quantization methods have been applied to LLTQ. By using symmetric quantization, the computational overhead for zeros that occurs during uniform quantization is eliminated. By introducing layer-wise scaling, quantization can be performed with one baseline factor for a given parameter or all elements of the input tensor. BN layer fusion, which combines the parameters of the convolution layer and the BN layer, was applied. This method does not require quantization in the BN layer. This can reduce the number of operations and inference time by eliminating the multiplication and division operations required for the BN layer. The fire module is structured so that the output of the squeeze layer goes into the input of two expansion layers. In floating-point, the inputs of both expansion layers are the same. However, in QAT, each convolutional layer requires a different normalization factor. Therefore, the squeeze layer must calculate the output of each expansion layer separately. This can cause a bottleneck when performing CNN inference in hardware. To solve this problem, QAT was performed by making the quantizers of both expansion layers into a single shared quantizer.

Figure 4 is a diagram for explaining an integer scale conversion method according to an embodiment of the present invention.

A CNN using parameters created by the proposed quantization method was reconstructed using only integer arithmetic through the proposed integer scale transformation method. As shown in Figures 4(b) and 4(d), the arrows of W _k ^f , B _k ^f , W _k+1 ^f , B _k+1 ^f , W _k+2 ^f , and B _k+2 ^f are floating-points. It represents the flow of data, and the remaining arrows represent the flow of integer data. The superscript of each letter indicates the data type. f represents floating-point data, i represents an integer, and the subscripts k, k + 1, k + 2, k + 3 represent the number of each layer. Q _a is an Integer-Arithmetic-Only (IAO) activation quantizer, Q _b is an IAO bias quantizer, and Q _w is an IAO weight quantizer. S _w,k , S _w,k+1, S _w,k+2 are weight standardization factors, S _b,k , S _b,k+1, S _b,k+2 are bias standardization factors, S _{a ,k} , S _a,k+1 , S _a,k+2 , S _a,k+3 are the active standardization factors. Here, A, W, B, and C indicated on the data flow arrows represent layer input, weight, bias, and activation function output, respectively. The part marked offline in Figure 4 is the process of converting the floating-point parameter affected by quantization into an integer. This process is performed before inference, so it does not affect inference time. The process of converting floating-point parameters affected by quantization to integer parameters is shown as follows.

(5)

(6)

The input to the convolution layer was assumed to be an integer quantized with the bit precision specified by QAT. As shown in Figures 4(a) and 4(b), the general convolutional layer performs MAC operations on integer-scale weights and integer-scale activations and adds integer-scale bias to the MAC results. As shown in equation (6), the bias value must be converted to the same scale as the MAC operation result. This method is called bias upscaling, and without this process, errors will occur because the bias and MAC operation results have different bit precisions. After adding the bias, the data passes the activation function ReLU, and the result of ReLU can be expressed as equation (7).

(7)

Here, C _k ⁱ is the output value of ReLU and must be converted to the bit precision of the current layer. MAC operations are performed with the bit precision specified by QAT before being input to the next layer. This method is called activation downscaling, and cancels the influence of the current layer's baseline factor and scales it to the input precision of the next layer. Activation downscaling can be expressed as Equation (8).

(8)

The behavior of the squeeze layer in the fire module shown in Figures 4(c) and 4(d) is the same as the general convolution layer to which Equations (5), (6), and (7) are applied. Due to the introduction of a shared quantizer, a single normalization factor is sufficient. Therefore, the same input can be used without having to calculate the input for each expansion layer. The input of each expansion layer can be described as Equation (9).

(9)

After activation downscaling using equation (8), the output of each expansion layer must be connected for each channel. Channel-specific connections do not affect the value, so no expansion is required. This process is shown in equation (10).

(10)

Figure 5 shows the architecture of a resource-efficient integer-arithmetic dedicated FPGA-based CNN accelerator for real-time facial emotion recognition according to an embodiment of the present invention.

The CNN accelerator for real-time facial expression recognition according to an embodiment of the present invention was implemented on a heterogeneous SoC platform consisting of a CPU (Processing System; PS) 510 and an FPGA (Programmable Logic; PL) 520. The CPU 510 allocates a memory area and transmits each layer configuration information, such as kernel size, stride, padding, input channel, output channel, movement amount, and address, to the FPGA 520 accelerator IP through the AXI_LITE bus. Data for MAC operation, such as input images, feature maps, and parameters, are transferred from DRAM to the FPGA (520) accelerator through the AXI_MEM bus. Each layer that makes up the CNN shares a multiple accumulation unit (MACU) and a post-processing module (PPM) to perform operations. The calculation results are transferred from the FPGA (520) accelerator to DRAM using the AXI_MEM bus.

The general-purpose embedded CPU according to an embodiment of the present invention performs a convolution operation with six nested loops very slowly. To solve this problem, a parallel processing module was implemented in the PL region 520 of the SoC platform to accelerate the CNN inference process. To implement the parallel processing module, we find out where the parallel processing strategy can be applied to the six nested loops. Among several possible parallelization strategies, two have been used to efficiently implement loop-level parallelism in embedded devices [9].

First, a parallelization strategy within the convolution was implemented. Parallelization within the convolution was achieved by completely unrolling the loop corresponding to the internal operation of the MACU shown in Figure 5. The fully unrolled loop was implemented as a MACU consisting of 9 parallel multipliers and 8 adders for 3×3 convolution operations. Second, an inter-feature map parallelization strategy was applied. The MACU shown in Figure 5 was used to achieve inter-feature map parallelism by partially unblocking loops for other channels of the output feature map. Because the unrolling factor of the partially unrolled loop was 4, the accelerator can execute MAC operations on the other four channels of the output feature map in parallel. The 1×1 operation is performed by reusing a portion of the fully unrolled MACU for the 3×3 convolution operation.

When considering SoC platforms, more parallelization strategies are available. However, in the case of intra-feature map parallel processing, it was not free from data dependency because all results of MAC operations for each input channel targeting pixels of the same output channel had to be accumulated. Therefore, it is likely that many resources will be utilized. Parallelization between convolutions required large amounts of data to be preloaded into on-chip buffers, making it unsuitable for implementation in embedded devices.

In the data caching module according to an embodiment of the present invention, the process of accessing data from off-chip memory each time for MAC operation is costly in terms of waiting time and energy. Additionally, the on-chip memory inside the PL area 520 is not large enough to store all parameters and calculation results of the CNN. Therefore, there is a need to maximize data reuse in on-chip memory to minimize the number of accesses to off-chip memory. Three data caching modules (FMCM, PCM, OCAM) were used in the PL area 520.

In MACU, where the intra-convolutional parallel processing strategy is applied, the 3×3 convolutional filter requires an input feature map of 9 pixels for MAC operation. For efficient pre-reading of the input feature map, the accelerator requires a line buffer that can store at least three lines. To facilitate addressing, a line buffer was created that can store four lines of the input feature map. These line buffers were bundled together and called the Feature Map Caching Module (FMCM). To maximize parameter data reuse, an on-chip memory, called Parameter Caching Module (PCM), was created to store all parameters of the layer where the task is performed, and an inter-feature map parallelization strategy is used by partitioning the PCM of each processing module. did. The memory was created to accumulate the MAC operation results for the channel of the output feature map during MAC operation, and the memory is called the output channel accumulation module (OCAM).

In the post-processing module according to an embodiment of the present invention, all operations except the convolution operation are performed in PPM. PPM consists of a shift operation, an addition operation, and a sign bit comparison operation. PPM receives the final output value of OCAM as input. Also, like the parallel processing module, the feature-to-feature map parallelization strategy is applied with the unrolled argument set to 4. The final output of PPM is a writeback to DRAM to input the input of the next layer.

Figure 6 is a diagram for explaining the entire task scheduling process for an integer-arithmetic dedicated FPGA-based CNN accelerator according to an embodiment of the present invention.

Referring to Figure 6, first, before starting the CNN task, the CPU calculates the size and layer of the data caching module and the address of the data area using pre-calculated network constraints (610). The calculated address and layer configuration are transferred from PS to PL configuration registers via the AXI_LITE bus. Data caching modules (FMCM, PCM) pre-fetch data from DRAM via the AXI_MEM bus using the pre-transferred address and layer configuration (620). The pre-read data is loaded into the pixel buffer and the parameter buffer, and the MAC operation is performed in the parallel processing module (630). OCAM stores and accumulates intermediate results of MAC operations. After completion of accumulation, PPM executes the remaining operations (640). This data flow continues until all processes in the layer are terminated (650).

As previously explained, the present invention proposes an FPGA-based CNN accelerator for real-time facial expression recognition. The accelerator is optimized for embedded systems and applies a new hardware-friendly quantization method.

According to an embodiment of the present invention, FERPlus-A is proposed to improve facial expression recognition performance. This is a new training dataset created by applying various image processing algorithms such as interactive filtering, Contrast Limited Adaptive History Equalization (CLAHE), and edge enhancement.

According to an embodiment of the present invention, a lightweight CNN architecture is proposed for embedded devices. The proposed CNN uses SqueezeNet's modified fire module as the basic computational module to optimize the number of operations and memory footprint. The proposed CNN showed facial expression recognition accuracy of approximately 86.58% when evaluated using the FERPlus test dataset. This is the highest performance reported compared to previous work using lightweight CNNs.

According to an embodiment of the present invention, a Log Level Threshold Quantization (LLTQ) method is proposed, which is a new hardware-friendly quantization method. The proposed method can replace multiplication and division operations using standardization factors with shift operations. The size of the proposed quantized CNN parameters is approximately 0.39MB, and the number of operations is approximately 28M Integer Operations (IOP). Additionally, it was confirmed that the performance of CNN using only 8-bit integer arithmetic does not decrease accuracy compared to using 32-bit floating-point.

According to an embodiment of the present invention, the proposed CNN optimized for embedded devices is implemented on the Xilinx ZC706 evaluation version by applying a parallelization strategy and an efficient data caching strategy for real-time facial expression recognition. FPGA-based CNN accelerator, using only 8-bit integer arithmetic, utilized 5,090 LUTs, 7,588 flip-flops, 61 BRAMs, and 49 DSPs. The accelerator achieved a frame rate of approximately 10 frames per second (FPS) while consuming 2.3W of power.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

Claims (8)

In a method of operating an FPGA-based CNN accelerator implemented on a heterogeneous SoC (System on Chip) platform including a CPU (Processing System; PS) and FPGA (Programmable Logic, PL),
Generating a training dataset combining a plurality of image processing algorithms in the CPU to improve CNN facial expression recognition performance, and converting parameters of the generated training dataset through floating-point training;
Extracting quantized parameters through quantization recognition training for parameters converted through floating-point training in the CPU, and performing Integer-Arithmetic-Only CNN reconstruction; and
Performing real-time facial emotion recognition using the integer-arithmetic dedicated CNN in the FPGA and CPU areas of a heterogeneous SoC platform.
Including,
The step of extracting quantized parameters through quantization recognition training for parameters converted through floating-point training in the CPU and performing integer-arithmetic-only CNN reconstruction,
The Learned Step Size Quantization (LSQ) method is used to set a trainable parameter as a reference factor and train the reference factor along with other parameters to perform a pre-calibration process for the reference factor in the quantization process. without need,
By setting the trainable parameter to a logarithmic threshold (log ₂ t) using the Trained Quantization Threshold (TQT) method and mapping the quantization factor to a power of 2 term, the quantization factor is Replace the multiplication and division used with shift operations,
By using symmetric quantization in the process of performing quantization-aware training (QAT) using pre-trained floating-point parameters, the computational overhead for zero points that occurs during uniform quantization is eliminated, and layer-by-layer scaling is eliminated. By introducing a Log Level Threshold Quantization (LLTQ) method that performs quantization with one reference factor for a given parameter or all elements of the input tensor.
How an FPGA-based CNN accelerator works. According to paragraph 1,
The step of generating a training dataset combining a plurality of image processing algorithms in the CPU to improve CNN facial expression recognition performance, and converting parameters through floating-point training for the generated training dataset,
Perform calculations through a basic calculation block including a first calculation module (FireA) and a second calculation module (FireB),
Each of the first calculation module (FireA) and the second calculation module (FireB),
A squeeze layer to reduce the number of channels in the feature map and the number of operations in the subsequent expansion layer; and
Includes an expansion layer to receive the output of the squeeze layer as input and expand the channel again,
In order to minimize accuracy degradation by maintaining the reception field, the kernel sizes of the filters included in each of the squeeze layer of the first calculation module and the squeeze layer of the second calculation module have different kernel sizes.
How an FPGA-based CNN accelerator works. According to paragraph 2,
The step of generating a training dataset combining a plurality of image processing algorithms in the CPU to improve CNN facial expression recognition performance and converting parameters through floating-point training for the generated training dataset,
In order to reduce the resolution of the feature map by half, the stride, which is the filter sampling size of each of the squeeze layer and expansion layer of the basic calculation block, is set to 2,
A CNN classifier that includes multiple convolutional layers and one global average pooling layer reduces the output parameters by placing a global average pooling layer with a filter kernel size of 1 at the output stage,
CNN's classifier minimizes the number of channels to match the number of classes through a convolutional layer and then compresses the resolution of the feature map to one pixel.
The convergence stability and performance of CNN are improved through a batch normalization layer included between the convolution layer and the activation function.
How an FPGA-based CNN accelerator works. delete In the FPGA-based CNN accelerator implemented on a heterogeneous SoC (System on Chip) platform including a CPU (Processing System; PS) and FPGA (Programmable Logic, PL),
The CPU is,
To improve CNN facial expression recognition performance, a training dataset combining multiple image processing algorithms is created, and parameters are converted through floating-point training for the generated training dataset,
For parameters converted through floating-point training, quantized parameters are extracted through quantization recognition training, and Integer-Arithmetic-Only CNN reconstruction is performed.
Perform real-time facial emotion recognition using the integer-arithmetic-only CNN in the FPGA and CPU areas of heterogeneous SoC platforms,
The CPU is,
In the process of extracting quantized parameters through quantization recognition training for parameters converted through floating-point training and performing Integer-Arithmetic-Only CNN reconstruction,
The Learned Step Size Quantization (LSQ) method is used to set a trainable parameter as a reference factor and train the reference factor along with other parameters to perform a pre-calibration process for the reference factor in the quantization process. without need,
By setting the trainable parameter to a logarithmic threshold (log ₂ t) using the Trained Quantization Threshold (TQT) method and mapping the quantization factor to a power of 2 term, the quantization factor is Replace the multiplication and division used with shift operations,
By using symmetric quantization in the process of performing quantization-aware training (QAT) using pre-trained floating-point parameters, the computational overhead for zero points that occurs during uniform quantization is eliminated, and layer-by-layer scaling is eliminated. By introducing a Log Level Threshold Quantization (LLTQ) method that performs quantization with one reference factor for a given parameter or all elements of the input tensor.
FPGA-based CNN accelerator. According to clause 5,
The CPU is,
In the process of creating a training dataset combining multiple image processing algorithms to improve CNN facial expression recognition performance, and converting parameters through floating-point training for the generated training dataset,
Perform calculations through a basic calculation block including a first calculation module (FireA) and a second calculation module (FireB),
Each of the first calculation module (FireA) and the second calculation module (FireB),
A squeeze layer to reduce the number of channels in the feature map and the number of operations in the subsequent expansion layer; and
Contains an expansion layer to expand the channel again,
In order to minimize accuracy degradation by maintaining the reception field, the kernel sizes of the filters included in each of the squeeze layer of the first calculation module and the squeeze layer of the second calculation module have different kernel sizes.
FPGA-based CNN accelerator. According to clause 6,
The CPU is,
In order to reduce the resolution of the feature map by half, the stride, which is the filter sampling size of each of the squeeze layer and expansion layer of the basic calculation block, is set to 2,
A CNN classifier that includes multiple convolutional layers and one global average pooling layer reduces the output parameters by placing a global average pooling layer with a filter kernel size of 1 at the output stage,
CNN's classifier minimizes the number of channels to match the number of classes through a convolutional layer and then compresses the resolution of the feature map to one pixel.
The convergence stability and performance of CNN are improved through a batch normalization layer included between the convolution layer and the activation function.
FPGA-based CNN accelerator. delete

Images