KR102316528B1 - Hardware friendly neural architecture search(nas) based neural network quantization method - Google Patents

Abstract

Provided are a neural network data quantization method and system based on a new neural network architecture search technique called neural channel expansion. Disclosed is a hardware friendly neural architecture search (NAS)-based neural data network quantization method. The data quantization method in a neural network may include: a step of searching for a neural network architecture in which channel information is adjusted to derive a neural network model robust against quantization errors; and a step of deriving a quantized neural network model through training on the searched neural network architecture.

Description

HARDWARE FRIENDLY NEURAL ARCHITECTURE SEARCH (NAS) BASED NEURAL NETWORK QUANTIZATION METHOD

The following description relates to a Neural Network data quantization technique based on Neural Architecture Search.

Recently, with the development of artificial intelligence (AI) technology, various industries are applying and introducing artificial intelligence technology. According to this trend, the demand for real-time hardware implementation of artificial neural networks, such as convolutional neural networks, is also increasing.

A large amount of computation and memory is required in the learning and recognition process of a multi-layer deep neural network such as a convolutional neural network. As a method of reducing the amount of computation and memory of a multi-layered deep neural network, a bit quantization method of reducing the data expression size of a parameter used for computation of an artificial neural network in bits may be used. In the conventional bit quantization method, uniform bit quantization, which quantizes all parameters of the artificial neural network with the same number of bits, is used, but the conventional uniform bit quantization method uses the number of bits for each parameter used in the artificial neural network. The problem is that the change does not accurately reflect the impact on overall performance.

In Korean Patent Laid-Open Publication No. 10-2018-0120967 (published on November 07, 2018), a data quantization method and apparatus for a neural network have been disclosed.

It is possible to provide a method and system for quantizing neural network data based on a new neural network structure search technique called Neural Channel Expansion.

It is possible to provide a method and system for exploring a neural network structure with a more robust network for uniform precision quantization errors while maintaining hardware constraints.

The method for quantizing neural network data includes: searching for a neural network structure in which channel information is adjusted in order to derive a robust neural network model against quantization errors; and deriving a quantized neural network model through training on the searched neural network structure.

The step of discovering the neural network structure may include selectively adjusting the number of channels based on a neural channel expansion technique.

The step of searching for the neural network structure may include expanding a channel for a layer constituting the neural network by using a search space in which the channel can be reduced or expanded.

In the step of searching the neural network structure, a search space is constructed for the number of channels using a search parameter, and in the constructed search space, the sensitivity of each layer to quantization and a channel selection based on hardware constraints are used. updating search parameters.

Exploring the neural network structure may include updating a weight parameter for reducing training loss through single-bit quantization or multi-bit quantization in the search space.

The step of searching for the neural network structure includes comparing the search parameter with a specific threshold value in order to check whether the channel extension is required in each layer with the search parameter updated in relation to the maximum number of channels expandable in the neural network. may include steps.

In the step of searching the neural network structure, when the updated search parameter related to the maximum number of channels through the comparison exceeds a specific threshold value, the channel expansion of each layer is activated, and the channel expansion is applied to the activated layer. adding the updated weight parameter and copying the updated search parameter.

A quantization system for quantizing neural network data includes: a structure search unit for searching a neural network structure with channel information adjusted in order to derive a robust neural network model against quantization errors; and a model derivation unit for deriving a quantized neural network model through training on the searched neural network structure.

The structure search unit may selectively adjust the number of channels based on a neural channel expansion technique.

The structure search unit may expand a channel for a layer constituting a neural network by using a search space in which a channel can be reduced or expanded.

Using a new neural network structure discovery technique called neural channel extension, uniform precision quantization through channel extension can be performed to uniformly control the number of channels in the entire layer.

Correction of quantization error by selective channel expansion can be facilitated by using a new neural network structure search technique called neural channel expansion.

The quantization accuracy can be significantly improved by modifying the structure of the target neural network.

1 is an example of an algorithm for describing a neural network data quantization operation according to an embodiment.
2 is a graph for explaining the effect of channel extension on a dynamic activation range according to an embodiment.
3 is a graph for explaining the effect of channel extension on channel search according to an embodiment.
4 illustrates a neural channel expansion operation according to an embodiment.
5 is a block diagram illustrating a configuration of a quantization system according to an embodiment.
6 is a flowchart illustrating a method for quantizing neural network data in a quantization system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

In the embodiment, a new neural network structure search-based quantization operation called Neural Channel Expansion will be described. Since neural channel expansion is equipped with a simple but innovative channel expansion mechanism, it is possible to balance the number of channels in the entire layer through uniform precision quantization. An in-depth analysis of neural channel expansion can be performed to understand the effect of channel expansion on quantization error compensation. The quantization accuracy can be significantly improved by modifying the structure of the target neural network.

1 is an example of an algorithm for describing a neural network data quantization operation according to an embodiment.

를 사용하여 채널 C={1:cout}의 개수에 대해 검색 공간을 구성할 수 있다. 출력 활성화는 채널 별 보간(channel-wise interpolation)을 통해 정렬된 다른 채널을 사용하여 샘플링된 활성화의 가중합으로 계산될 수 있다. The quantization system may perform neural network data quantization based on a new neural network structure search technique called Neural Channel Expansion. First, a neural channel expansion operation will be described. The quantization system is a search parameter

can be used to construct a search space for the number of channels C={1:cout}. The output activation can be calculated as a weighted sum of activations sampled using different channels aligned through channel-wise interpolation.

Equation 1:

는 양자화기 Q에 의해 양자화된 입력 활성화 X 및 가중치 매개변수 W로 계산되며, I는 C의 샘플링된 서브셋(subset)이다. Here, enable the output

is computed with the input activation X and the weight parameter W quantized by the quantizer Q, where I is a sampled subset of C.

During the search, the search parameters may be updated through channel selection based on a trade-off between cross-entropy loss and hardware constraint loss. In the prior art, the number of channels is fixed, and the search range is limited to pruning. In an embodiment, when a search parameter related to the maximum number of channels exceeds a certain threshold, channel extension of an individual layer may be activated. If one layer is vulnerable to quantization errors, the search parameters can be updated with default settings for more channels to reduce cross-entropy loss. Using such a simple extension condition, it is possible to extend the channel to the layer most susceptible to quantization error and prune the channel of another layer that is strong in quantization. Accordingly, the overall hardware constraint may be satisfied.

)의 반복적인 업데이트가 수행될 수 있다. 최대 채널 수와 관련된 업데이트된 검색 매개변수가 각 계층에서 채널 확장이 필요한 지 여부를 확인하기 위해 임계 값(하이퍼 파라미터로 미리 결정됨)과 비교될 수 있다. 채널 확장이 발생되면, 가중치 매개변수가 해당 계층에 추가되고 검색 매개변수도 복사되어 채널 수가 증가될 수 있다. 검색이 완료됨에 따라 생성된 후보 모델은 승자독식 전략에 의해 도출될 수 있다. 다시 말해서, 각 계층에 대해 가장 큰 크기의 검색 매개변수를 가진 채널 수가 선택될 수 있다. 1 is an algorithm summarizing the overall operation for quantizing neural network data. Neural channel expansion can consist of three steps, including warm-up, discovery, and training. In the preparation phase, a warm-up of the entire supernet may be performed so that all supernet weight parameters can be reasonably initialized. In the retrieval stage, the weight (w) and retrieval parameters (

) may be repeatedly updated. An updated search parameter related to the maximum number of channels may be compared with a threshold (predetermined as a hyperparameter) to determine whether channel extension is required at each layer. When channel expansion occurs, the weight parameter is added to the corresponding layer and the search parameter is also copied, so that the number of channels can be increased. As the search is completed, the generated candidate model may be derived by a winner-take-all strategy. In other words, for each layer, the number of channels with the largest size of the search parameter may be selected.

The quantization system will describe in detail the operation of exploring a more robust network structure for uniform precision quantization errors while maintaining hardware constraints based on neural channel extension. First, the structure of the extended channel can reduce the dynamic range of input activation and suppress quantization errors. In other words, the structure of the network applied by neural channel expansion can be trained independently from scratch, and can exhibit robustness to quantization. Correction of quantization error may be facilitated by the channel selected through neural channel expansion.

2 is a graph for explaining the effect of channel extension on a dynamic activation range according to an embodiment.

2(a) and 2(b) show the standard deviation for each layer in ResNet20 with 1X channel or 2X channel, and FIG. 2(c) shows the standard deviation for each layer in VGG16 with 1X channel or 2X channel. 2(d) shows the ResNet20 accuracy test with 1X channel or 2X channel (reduced accuracy from W32A32 to W2A2).

Channel segmentation may reduce the dynamic range of the weight parameter. However, it is not clear how much the structure itself affects quantization during neural network training. In order to understand the effect of the extended channel structure on the quantization of a neural network (e.g. DNN), we first show that quantization applied to a given network increases the dynamic range of activation, preventing the derivation of successful QDNNs. Figure 2(a) shows the standard deviation (STDEV) of input activation for ResNet20 trained from scratch on the CIFAR10 data set, with or without a quantizer during training. W{X} and A{Y} indicate that the weight and activation are quantized into X bits and Y bits, respectively. The large increase in standard deviation for W32A2 and W2A2 means that large quantization errors can occur when the input activations are quantized. The increase in dynamic range may partly explain why quantization using fixed model structures often suffers from significant accuracy degradation when ultra-low-bit precision is applied uniformly. This phenomenon can be observed, for example, in the VGG16 model of FIG. 2(c).

In addition, FIGS. 2(b) and 2(c) show the effect of the extended channel structure on the dynamic range. The "2X" model means that the number of channels in the corresponding layer is doubled. At this time, all hyperparameter settings are the same. Both the 2X ResNet 20 and VGG16 models using 2-bit quantization (W2A2) can reduce the standard deviation to a full precision 1X model.

Dynamic range reduction may be considered due to weight initialization of the 2X channel. The initial weight parameter can be described as having little effect on the dynamic range of input activation. Specifically, the dynamic range of the initial weight may be determined by the number of channels. However, as shown in Fig. 2(a), the standard deviation of 2X ResNet20 to which W2A2 quantization is applied can still be reduced even if the weights are initialized in the same way as the 1X ResNet model is initialized.

Finally, it can be seen that the test accuracy is improved by reducing the dynamic range of the input activation. As can be seen in Fig. 2(c), the 1X ResNet20 model suffers from a large accuracy degradation (-0.84%) due to quantization (-2.22%), whereas the 2X ResNet20 model suffers from a relatively small accuracy degradation (-0.84%). . The increased robustness to such quantization errors is due to the mechanism by which the extended channel structure reduces the dynamic range of activation.

According to an embodiment, the structure of the extended channel may play a decisive role in compensating for a quantization error. Therefore, when channel extension is used in the neural network structure search (NAS) framework, a new network structure that is more powerful for uniform precision quantization can be searched for when training from the beginning by adjusting the channels of the layer.

3 is a graph for explaining the effect of channel extension on channel search according to an embodiment.

)에 대한 교차 엔트로피 손실의 기울기를 보여주는CIFAR10-ResNet20에 대한 실험 및 실험 결과를 나타낸 것이다. 도3(a)는 완전 정밀도, 도 3(b)는 2비트 양자화 사용, 도 3(c)는 모든 계층의 Kendall 순위 상관 점수를 나타낸 것이다. In Figure 3, the search parameters of the hierarchy during the search (

) shows the experimental and experimental results for CIFAR10-ResNet20 showing the slope of the cross-entropy loss. 3(a) shows full precision, FIG. 3(b) uses 2-bit quantization, and FIG. 3(c) shows Kendall rank correlation scores of all layers.

The quantization system may selectively allow channel extension only when it is desirable to select a larger number of channels to compensate for the quantization error. Otherwise, channels can be cleaned up to meet the overall hardware constraint. 3 is an example for explaining the compensation mechanism of neural channel expansion.

,

과 같이 많은 수의 채널과 관련된 검색 매개변수는 큰 음의 기울기를 수신한다. 예를 들면, 도 3(a)는 완전한 정밀도로 CIFAR10-ResNet20의 종래의 기술(TAS 검색) 중 검색 매개변수의 기울기를 나타낸다. 최대 채널 수(

)와 관련된 검색 매개변수는 처음에는 음의 기울기를 수신할 수 있다. 반대로, 채널 수가 가장 적은 검색 매개변수(

)는 양의 기울기를 수신할 수 있다. 이를 통해 계층이 처음에는 많은 수의 채널을 선호하지만, 검색 에포크(epoch)보다 선호도가 감소한다고 추측할 수 있다. First, we indicate that the channel selection preference can be observed as a slope for the search parameters. An important compromise explored during the search is between cross-entropy loss and hardware constraint loss. In particular, a layer sensitive to cross entropy may select a large number of channels. In other words,

,

Search parameters associated with a large number of channels, such as , receive large negative slopes. For example, Fig. 3(a) shows the slope of the search parameters among the prior art (TAS search) of CIFAR10-ResNet20 with full precision. Maximum number of channels (

) and associated search parameters may initially receive a negative slope. Conversely, the search parameter with the smallest number of channels (

) can receive a positive slope. From this, it can be inferred that the layer initially prefers a large number of channels, but decreases in preference over the search epoch.

Next, we show that the quantization during the search outperforms the preference for multiple channels. In the same experimental setup, if quantization is applied during the search, the preference for multiple channels becomes more pronounced as shown in Fig. 3(b). This phenomenon is because the quantization error has more influence on the cross-entropy loss. To quantize these channel preferences, a Kendall rank correlation score for the slope of the search parameters and the average of the search epochs can be calculated. The more consistent the preference for many channels, the higher the Kendall score. Figure 3(c) shows the Kendall score for each layer with or without quantization. The Kendall score increases if quantization is applied during neural network structure search. This increased Kendall score means that quantization leads to a strong preference for a larger number of channels. Accordingly, it is possible to increase the search space to enable channel expansion. Thanks to this new search space, channel expansion can be made selectively for layers that prefer many channels. In this case, the new search space may obtain a higher Kendall score as shown in FIG. 3(c). This shows that the search space can favor a much larger number of channels. In other words, a new search space that selectively extends channels can compensate for quantization errors by relaxing the limit of selecting a larger number of channels.

4 illustrates a neural channel expansion operation according to an embodiment.

In FIGS. 2 and 3 , the operation of facilitating channel expansion to compensate for quantization error will be described. In FIG. 4 , the advantages of neural channel expansion will be described.

The advantages of quantization-aware structure search will now be described. The quantization system can reflect the quantization during neural network search in order to search for a better neural network structure for quantization in which neural channel expansion is performed. Quantization affects the slope for the search parameters, resulting in differences in the network structure after the search.

Fig. 4(a) shows the accuracy test after the search regardless of whether quantization is performed, Fig. 4(b) shows the model structure after the search regardless of whether quantization is performed, and Fig. 4(c) shows the accuracy of the channel extension strategy. It is a graph showing the vs. hardware constraint.

Fig. 4(a) shows that neural channel expansion for CIFAR10-ResNet20 was performed regardless of whether W2A2 was quantized during the search. You can then train the neural network model from scratch with or without W2A2 quantization by importing the neural network models after each search. After full precision training, it can be seen that both networks (quantized or non-quantized searched networks) achieve the same level of accuracy. However, in the case of W2A2, the network searched for quantization can obtain more gains in average accuracy compared to the network searched without quantization. 4(b) shows the channel section difference between the neural network model W2A2 for which quantization is searched and the neural network model for which quantization is not searched (W32A32). It can be confirmed that W2A2 prefers more channels in a later layer. We show that neural channel expansion can perform quantization-recognized neural network structure searches.

The advantages of selective channel extension will now be described. There are two options for searching for channel extensions. One option is to start the extended channel as a search space and then clean it up, the other option is to expand the channel selectively like neural channel expansion. To understand the effect of selective expansion, when retrieving the model, an experiment in which a 1X channel with 8 search parameters for each layer but neural channel expansion and a 2X channel with 16 search parameter layers was searched was constructed. can be If 1X-neural channel expansion searches for an appropriate network structure, 2X-neural channel expansion also needs to search for an appropriate network structure. However, the search results of 2X-neural channel dilatation were found to be inferior to neural channel dilatation. As can be seen in Fig. 4(c), for the same target hardware constraint, the network structure retrieved by 2X may have lower test accuracy than neural channel extension.

According to an embodiment, heterogeneous sensitivities for each layer may be resolved through uniform precision quantization using a neural network structure search technique.

An activation range during quantization recognition training may be reduced by providing a neural network structure with a layer in which the channel according to the embodiment is extended.

5 is a block diagram illustrating a configuration of a quantization system according to an embodiment, and FIG. 6 is a flowchart illustrating a method of quantizing neural network data in the quantization system according to an embodiment.

The processor of the quantization system 100 may include a structure search unit 510 and a model derivation unit 520 . These components of the processor may be representations of different functions performed by the processor according to control instructions provided by program code stored in the quantization system. The processor and components of the processor may control the quantization system to perform steps 610 to 620 included in the method for quantizing neural network data of FIG. 5 . In this case, the processor and components of the processor may be implemented to execute instructions according to the code of the operating system and the code of at least one program included in the memory.

The processor may load the program code stored in the file of the program for the neural network data quantization method into the memory. For example, when a program is executed in the quantization system, the processor may control the quantization system to load a program code from a file of the program into the memory according to the control of the operating system. At this time, each of the structure search unit 510 and the model derivation unit 520 executes the instruction of the corresponding part of the program code loaded into the memory to execute the subsequent steps 610 to 620 with different functional representations of the processor. can take

In step 610 , the structure search unit 510 may search for a neural network structure in which channel information is adjusted in order to derive a robust neural network model against quantization errors. The structure search unit 510 may selectively adjust the number of channels based on a neural channel expansion technique. The structure search unit 510 may expand a channel for a layer constituting a neural network by using a search space in which a channel can be reduced or expanded. The structure search unit 510 constructs a search space for the number of channels by using the search parameters, and selects the search parameters through channel selection based on the sensitivity of each layer to quantization and hardware constraints in the configured search space. can be updated. The structure search unit 510 may update a weight parameter for reducing training loss through single-bit quantization or multi-bit quantization in the search space. The structure search unit 510 may compare the search parameter with a specific threshold value in order to check whether the channel extension is required in each layer for the search parameter updated in relation to the maximum number of channels expandable in the neural network. The structure search unit 510 activates the channel extension of each layer when the updated search parameter related to the maximum number of channels exceeds a specific threshold through comparison, and the updated weight parameter for the channel extension is activated. and copy the updated search parameters.

In step 620 and the model derivation unit 620 may derive a quantized neural network model through training on the searched neural network structure. Accordingly, the neural network structure trained by neural channel expansion can be trained independently from the beginning and exhibit robustness against quantization.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

In the neural network data quantization method performed by a quantization system including a structure search unit and a model derivation unit,
In the structure search unit, based on a target neural network configured to derive a robust neural network model against quantization errors, searching for a neural network structure in which channel information is adjusted through a search parameter related to the maximum number of channels expandable to the target neural network ; and
deriving, in the model derivation unit, a quantized neural network model through training on the searched neural network structure;
including,
Exploring the neural network structure comprises:
In a search space where a weight parameter is initialized and configured for the number of channels using the search parameter, through channel selection based on the sensitivity and hardware constraints of each layer to quantization, the search parameter and updating a weight parameter for reducing training loss, and comparing the size of the updated search parameter with a specific threshold predefined by a hyperparameter of the target neural network to determine whether to activate channel expansion for each layer step
including,
The search parameter is updated through channel selection based on a trade-off between cross entropy loss and hardware constraint loss.
A quantization method of neural network data comprising a. According to claim 1,
Exploring the neural network structure comprises:
Selectively adjusting the number of channels based on the neural channel expansion technique
A quantization method of neural network data comprising a. 3. The method of claim 2,
Exploring the neural network structure comprises:
Expanding the channel for the layers constituting the neural network using a search space that can shrink or expand the channel.
A quantization method of neural network data comprising a. According to claim 1,
Exploring the neural network structure comprises:
Constructing a search space for the number of channels using search parameters
A quantization method of neural network data comprising a. 4. The method of claim 3,
Exploring the neural network structure comprises:
In the search space, updating a weight parameter for reducing training loss through single-bit quantization or multi-bit quantization;
A quantization method of neural network data comprising a. 5. The method of claim 4,
Exploring the neural network structure comprises:
In order to determine whether channel expansion is required in each layer, the updated search parameter in relation to the maximum number of channels expandable in the target neural network is set with a specific threshold predetermined by the hyperparameter of the target neural network. step to compare
A quantization method of neural network data comprising a. 7. The method of claim 6,
Exploring the neural network structure comprises:
When the size of the search parameter updated in relation to the maximum number of channels through the comparison exceeds a specific threshold, channel extension of each layer is activated, and the updated weight parameter is applied to the layer in which the channel extension is activated. and copying the updated search parameters.
A quantization method of neural network data comprising a. A quantization system for quantizing neural network data, comprising:
a structure search unit for searching a neural network structure in which channel information is adjusted through a search parameter related to the maximum number of channels expandable to the target neural network based on the target neural network configured to derive a robust neural network model against quantization errors; and
A model derivation unit for deriving a quantized neural network model through training on the searched neural network structure
including,
The structure search unit,
In a search space where a weight parameter is initialized and configured for the number of channels using the search parameter, through channel selection based on the sensitivity and hardware constraints of each layer to quantization, the search parameter and updating a weight parameter for reducing training loss, and comparing the size of the updated search parameter with a specific threshold predefined by a hyperparameter of the target neural network to determine whether to activate channel expansion for each layer including that,
The search parameter is updated through channel selection based on a trade-off between cross entropy loss and hardware constraint loss.
A quantization system comprising a. 9. The method of claim 8,
The structure search unit,
Selectively adjusting the number of channels based on the neural channel expansion technique
A quantization system, characterized in that. 9. The method of claim 8,
The structure search unit,
It expands the channel for the layers constituting the neural network using a search space that can shrink or expand the channel.
A quantization system, characterized in that.

Images