KR20180013674A - Method for lightening neural network and recognition method and apparatus using the same - Google Patents

Abstract

Disclosed are a method for lightening a neural network, a recognition method using the same, and a device thereof. According to an embodiment of the present invention, the neural network comprises a plurality of layers which individually include neurons and synapses which connect the neurons included in the neighboring layers. At least part among synapse weights having a value greater than 0 and less than a (a > 0) is set to 0.

Description

TECHNICAL FIELD [0001] The present invention relates to a method of weight reduction of a neural network, a method of recognizing the neural network,

The following embodiments relate to a neural network lightening method, a recognition method using the neural network, and a device thereof.

Object recognition is a method of recognizing a specific object included in input data. The object may include various data to be recognized according to a certain pattern such as image and voice. For example, an image-based object classifier can automatically find a specific object included in an input image. The object classifier may include a plurality of synapses connecting neurons and neurons and may be trained based on training data. Depending on the number of neurons and synapses included in the object classifier, the memory usage for learning and recognition may increase dramatically, and biased or excessive learning may lead to overfitting.

The neural network includes a plurality of layers each including neurons and synapses connecting the neurons contained in neighboring layers.

According to one aspect, at least some of the synaptic weights having a value greater than 0 and less than a (a > 0) in the neural network are set to zero. The synapse weights set to 0 may correspond to synapses connecting neighboring layers of some of the plurality of layers. The a may be set differently for the plurality of layers. Alternatively, the value a may be set differently for each output map channel in a specific layer.

At least some of the synaptic weights having a value greater than b (b > a) may be set to b. The synapse weights set in b may correspond to synapses connecting neighboring layers of some of the plurality of layers. The b may be set differently for each of the plurality of layers. Alternatively, b may be set differently for each output map channel in a specific layer.

Wherein each of the synaptic weights having a value equal to or greater than a is a maximum number of bits corresponding to log ₂ (max-a), wherein max is a maximum synapse weight greater than a, and a and max are integers . Each of the synaptic weights having a value equal to or greater than the a and having a value less than or equal to b may be expressed by the number of bits corresponding to log ₂ (ba), where a and b are integers.

According to one aspect, at least some of the synaptic weights having a value greater than b (b> 0) in the neural network are set to b. The synapse weights set in b may correspond to synapses connecting neighboring layers of some of the plurality of layers. The b may be set differently for each of the plurality of layers. B may be set differently for each output map channel in a specific layer. Each of the synapse weights having a value less than or equal to b may be expressed by the number of bits corresponding to log2 (b), where b is an integer.

A method according to one aspect of the present invention includes obtaining parameters that are regularized based on a distribution of parameters corresponding to layers included in a neural network; De-normalizing the normalized parameters based on a normalization variable corresponding to the layer; Applying the non-normalized parameters to the layer; And recognizing input data using the neural network.

The normalization variable corresponding to the layer may be set differently for a plurality of layers included in the neural network or may be set differently for a plurality of output map channels included in the layer.

The normalization variable may include an offset to shift the normalized parameters with respect to zero. Wherein when the denormalized parameters are m bit integers and the neural network receives an n-bit real number greater than the m bits, applying the denormalized parameters to the layer further comprises: de-quantizing by an error of n bits; And applying the non-quantized parameters to the layer.

Wherein the applying the non-normalized parameters to the layer comprises: obtaining a bit sequence corresponding to the layer, the bit sequence indicating whether the value of the parameter is zero; Decompressing said non-normalized parameters constituting a non-zero sequence based on said bit sequence; And applying the decompressed parameters to the layer. Decompressing the denormalized parameters comprises determining a decompressed parameter of the first index by multiplying a bit value of the first index in the bit sequence by a parameter of a second index in the non-zero sequence, ; Increasing the second index by a bit value of the first index in the bit sequence; And increasing the first index by one.

A recognition device according to one side comprises: a processor; And a memory containing instructions readable by the computer, wherein when the instruction is executed in the processor, the processor obtains parameters that are regularized based on a distribution of parameters corresponding to layers included in the neural network De-normalizes the normalized parameters based on a normalization variable corresponding to the layer, applies the non-normalized parameters to the layer, and recognizes input data using the neural network do.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a view of a lightening device for a neural network according to one embodiment.
FIG. 2 illustrates a quantization process according to an embodiment; FIG.
3 illustrates a cut-off operation in accordance with one embodiment;
4 illustrates a cutting operation according to one embodiment;
5A and 5B are diagrams illustrating a normalization operation and a distribution of parameters according to an embodiment.
6 is an operational flow diagram illustrating an iterative normalization process according to an embodiment;
Figures 7A and 7B are diagrams illustrating a method for determining a candidate range according to embodiments.
8 illustrates a variation of a sequence by normalization according to one embodiment;
FIG. 9 illustrates a variation of a sequence by compression in accordance with one embodiment; FIG.
10 illustrates a variation of the distribution of parameters by weighting according to one embodiment.
11 illustrates a post-processing process of learned parameters in accordance with an embodiment;
12 illustrates a parameter tuning process according to an embodiment;
13 illustrates a learning process according to an embodiment;
FIG. 14 illustrates a recognition process according to an embodiment; FIG.
15 illustrates a variation of the distribution of parameters by restoration according to an embodiment;
Figure 16 illustrates a variation of a sequence by decompression according to one embodiment;
FIG. 17 is a diagram illustrating a sequence change by division according to an embodiment; FIG.
18 shows a variation of a sequence by division and compression according to an embodiment;
19 is an operational flow diagram illustrating a method of weight reduction according to one embodiment.
20 is an operational flowchart illustrating a recognition method according to an embodiment;
21 illustrates an electronic system according to one embodiment.

The specific structure or functions of the embodiments disclosed herein are merely illustrative of the technical concept, and may be embodied in various other forms.

The terms first or second may be used to describe various components, but such terms should be understood only for the purpose of distinguishing one component from another. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the words "comprising ", etc. are intended to designate the presence of stated features, integers, steps, operations, elements, parts or combinations thereof, It is to be understood that they do not preclude the presence or addition of components, components or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

FIG. 1 is a view illustrating an apparatus for reducing weight of a neural network according to an exemplary embodiment of the present invention. Referring to FIG. 1, the light-weighting apparatus 100 acquires parameters corresponding to layers included in the neural network. The neural network may include a plurality of layers, and each of the layers may comprise a plurality of neurons. Neurons in neighboring layers can be connected to synapses. Depending on the learning, weights may be assigned to the synapses, and the parameters may include these weights.

The lightening device 100 lightens the obtained parameters. The lightening device 100 may lighten the parameters through at least one of quantization, regularization, and compression. The quantization is to transform the representation to reduce the size of the data and the normalization is to reduce the range of values that the parameters have through at least one of the truncation operation and the cutoff operation, And the size of the data is reduced by classifying the parameters. Concrete details about quantization, normalization and compression will be described later.

In the graphs shown in FIG. 1, the horizontal axis represents the values of the parameters, and the vertical axis represents the frequency of the parameters. The parameters may have various values depending on the learning process, and the light weighting device 100 may limit the values of the parameters through weight reduction. The parameters before lightening are distributed in the first range. The values of the parameters are limited according to the weight reduction, and the parameters are distributed in the second range narrower than the first range. As the distribution range of the parameters is limited, the memory usage of the neural network can be reduced. Hereinafter, the first range is referred to as an original range and the second range is referred to as a light-weight range.

For sophisticated learning, the number of layers included in the neural network can be increased. As the number of layers increases, the resources required for learning through the neural network and recognition through the neural network increase sharply. The weight reduction of the neural network can suppress the increase of such resources. Hereinafter, weight reduction of the neural network may mean weight reduction of the parameters for the neural network.

As neural networks become lightweight, learning of neural networks can be possible not only in servers that can utilize highly sophisticated resources but also in user devices with limited available resources. Accordingly, the user can learn an optimized model online through a user equipment such as a mobile terminal. On the other hand, excessive or excessive learning can cause overfitting in the neural network, and the performance of the neural network may be degraded depending on the overlay. The light weight of the neural network can eliminate or compensate for unnecessary parameters that cause over sum. Therefore, the performance of the neural network can be enhanced by reducing the weight of the neural network.

The lightweighting process of lighter weight neural networks can be applied to various operations for learning and recognition. For example, the lighterization process may be applied to post-processing of learned parameters, to tuning of learned parameters, or to learning of parameters directly. Through this weight reduction process, the memory occupied by the learned parameters is reduced, and the performance of the neural network can be improved by solving the overarching agreement.

The lightened parameters can be stored in the storage device and used in the recognition process. The restoration device may restore the parameters through at least one of de-quantization, de-regularization, and de-compression in accordance with the lightening technique applied to the parameters.

2 is a diagram illustrating a quantization process according to an embodiment. Quantization means transforming the representation to reduce the size of the data. The parameters may have a predetermined expression depending on the system. For example, parameters can be represented as a decimal number of floating-point type. The light weighting device can convert the expression of the parameters so that the size of the data decreases in order to reduce the weight. For example, a lightweight device may convert the representation of parameters into an integer of a fixed-point type. According to one aspect, the lightweighting device may use a quantization function ( ^2Q ) for quantization. For example, parameters expressed as 32-bit floating-point numbers may be represented as 16-bit fixed-point numbers through a quantization function ( ^2Q ).

The decimal range, integer range, floating point, and fixed point are only examples, and other well-known representations can be applied to quantization. In addition, since at least one of quantization, normalization and compression can be applied to light weight of the neural network according to the embodiment, weight reduction of the neural network can be performed by normalization and compression alone. However, for convenience of explanation, the following description will be made on the assumption that parameters are quantized into 16-bit fixed-point integers. In this case, the parameters may be expressed in an integer range of -2 ¹⁵ to 2 ¹⁵ -1.

3 is a diagram illustrating a cutoff operation according to an exemplary embodiment. Referring to FIG. 3, the parameters may have various values depending on the learning process. The lightening device may normalize the parameters through at least one of a cutting operation and a cut-off operation. The cut-off operation is to set the parameters to a predetermined maximum or minimum value. The light weighting device can set the parameter that is the maximum value or more to the maximum value and the parameter that is the minimum value can be set to the minimum value. The maximum value and the minimum value can be determined by the variable (b).

Having the various values of the parameters may help the performance of the recognizer, but the performance of the recognizer may be degraded if the parameters are excessively large or too small. Therefore, the performance of the recognizer can be improved by restricting the range of values of the parameters through the cutoff operation. In addition, since the size of the data for expressing the parameter can be reduced by limiting the value of the parameter, the weight can be reduced through the cut-off operation. In order to improve or at least maintain the performance of the recognizer while reducing the size of the data, it is necessary to cut the parameters to the appropriate maximum or minimum value. The process of determining the variable (b) for the cutoff will be described in detail later.

4 is a view illustrating a cutting operation according to an embodiment. The lightening device may normalize the parameters through a cutting operation. Referring to Fig. 4, the lightening device sets the parameter within the cut-off range to 0 among the input parameters. The cutting range can be determined by the variable (a). A parameter close to 0 may have a small effect on the performance of the recognizer, but a parameter having a value of 0 in a compression process to be described later can greatly reduce the size, so that by setting a parameter close to 0 to 0, can do. However, in order to improve or at least maintain the performance of the recognizer while reducing the size of the data, it is necessary to cut the parameters to an appropriate cutting range. The process of determining the variable (a) for cutting will be described in detail later.

5A and 5B are diagrams illustrating a normalization operation and a distribution of parameters according to an embodiment. Referring to FIG. 5A, a case where both the cutoff operation and the cutting operation are performed will be described as an example. The lightweighting device can extract the appropriate lightweight range from the original range to improve or at least maintain the performance of the recognizer while reducing the size of the data.

Referring to FIG. 5B, a distribution diagram 510 of pre-normalization parameter values and a distribution diagram 520 of parameter values after normalization are shown. In Figure 5b, the x-axis represents the sequence of parameters and the y-axis represents the value of the parameter in that sequence. The original range corresponds to the entire range in which the input parameter values are distributed, and can be represented by a minimum y value to a maximum y value in the distribution diagram 510. The lightweight range corresponds to the range extracted by the normalization in the original range, and may be represented by a range (for example, -b to -a and a to b) in which the y value is distributed in the distribution diagram 520. [ As such, the lightweight range can be determined by a normalization variable including the variable (a) and the variable (b).

Referring again to FIG. 5A, the lightening device may perform at least one of increasing the variable a and decreasing the variable b to adjust the normalization variable. According to one embodiment, the lightening device may adjust the normalization variable iteratively. For example, the lightweight device may set a normalization variable for each iteration and obtain a performance index according to the normalization variable based on a predefined performance function. The light weighting device can repeat the preceding operations while iteratively adjusting the normalization variable until the performance index satisfies a predetermined criterion. If the performance index meets a predetermined criterion, the lightening device may terminate the iteration and determine the lightweight range based on the final normalization variable. The performance indicator may include a recognition rate and / or an error rate.

The predetermined criterion may be set as a first criterion that permits repeated iterations to be repeated as much as possible without degrading the performance after normalization to that before the normalization or may be set as a second criterion that maximizes performance after normalization . As will be described in detail below, weight reduction can be maximized when a first criterion is used, and recognition performance can be maximized when a second criterion is used.

6 is a flowchart illustrating an iterative normalization process according to an exemplary embodiment of the present invention. Referring to FIG. 6, a target recognition rate is determined through steps 625 and 635. At step 625, features are extracted from the verification data in the verification database using the original trained parameters. Each verification data may be composed of a pair of data. For example, the verification data may be composed of a pair of data corresponding to the same person or a pair of data corresponding to another person. At step 625, the characteristics of each data can be extracted.

In step 635, the target recognition rate is determined through feature matching. More specifically, by matching features extracted from a pair of data belonging to the same verification data, the recognition rate can be calculated. Here, the recognition rate may be a verification rate (VR) indicating a successful recognition rate of the same face. If the feature matching result of a pair of data corresponding to the same person indicates the same person, the recognition rate increases, and if the person indicates another person, the recognition rate may be reduced. Since the features are extracted using the original learning parameters in step 625, the target recognition rate may be the recognition rate based on the original learning parameters.

The iterative regularization process consecutively searches the lightweight range until the difference between the normalization variable a (for example, cutoff point) and b (for example, cutoff point) becomes minimum for each layer of the neural network Process.

According to one embodiment, iterative normalization may be performed through iterations of steps 610, 620, 630, and 640. More specifically, in step 610 of the first iteration, normalization of the original learning parameters is performed. In step 620, the characteristics of the verification data are extracted using the normalized parameters. In step 630, the characteristics of the verification data are matched, so that a regularized recognition rate is calculated. In step 640, the normalized recognition rate and target recognition rate are compared. If the normalized recognition rate is not less than the target recognition rate, the process returns to step 610 to perform the second iteration. According to another embodiment, an error rate can be used instead of the recognition rate. In this case, step 640 can be modified to repeat the iteration if the normalized error rate is less than the target error rate.

In the second and later iteration step 610, the normalized parameters in the previous iteration are updated by further normalization. For example, the lightweighting device may reduce the candidate range through at least one of increasing the variable (a) and decreasing the variable (b). The candidate range may be a range in which the parameter values according to the intermediate normalization variable before distribution are distributed.

According to one aspect, the normalization variable may include an offset that shifts the normalized parameter by zero based on the normalization variable. The lightening device may increase or decrease the normalization variable through a shift operation. For example, when the variable a is 0 and the variable b is 15, the candidate range may be determined to be 2 ⁰ to 2 ¹⁵ -1. When the lightening device further decreases the candidate range, the variable a can be increased to 1 and the variable b can be decreased to 14. At this time, the candidate range can be easily determined from 2 ¹ to 2 ¹⁴ -1 through a shift operation. The light-weighting device may include a shift register for a shift operation. According to the other party, the lightening device can reduce the candidate range in mini batches. For example, a lightweight device can reduce the candidate range by 2 ⁹ units. If the candidate range is reduced to a relatively large unit, the performance change due to the iteration can be changed into a predictable form, so that it is possible to determine an appropriate normalization variable by reducing the candidate range in units of mini batches.

Steps 620, 630, and 640 are performed based on further normalized parameters. If iteration is repeated and the normalized recognition rate becomes smaller than the target recognition rate, the iteration may be terminated and the optimal normalized parameter may be output in step 640. If an error rate is used instead of the recognition rate according to another embodiment, the optimal normalized parameter may be output if the normalized error rate is not less than the target error rate.

According to one embodiment, the repetitive normalization process described above can be performed by the lightening device 1120 of FIG. 11 to be described later. For example, the lightening device 1120 receives the learned parameters corresponding to the original learning parameters from the learning device 1110, and outputs the lightened parameters corresponding to the optimal normalized parameters through the above-described iterative normalization process have. As will be described in greater detail below, the lightening device 1120 may perform quantization and compression as well as normalization.

7A and 7B are diagrams illustrating a method for determining a candidate range according to embodiments. Referring to FIG. 7A, as the iteration is repeated, the authentication success rate of the user increases rather than until a predetermined iteration time, and then decreases again.

When parameter normalization is performed, at least some of the parameters lose the previously learned value. Nevertheless, the increase in the recognition rate up to a certain iteration time is due to the normalization of the distortions of the noise and errors that existed in the existing learned database. For example, when the cutoff value is increased, the finer linkage between the learned neurons is excluded, and the proportion of 0 in the distribution of the entire parameter values increases. Further, the degree of freedom with respect to the maximum value of the parameter is limited by the cutoff value. This can mitigate the noise and errors that existed in existing learned databases.

As described above, the light weighting device can repeat the parameter normalization until the authentication threshold is met while the success rate of authentication is increased. As a result, the lightweighting device can determine the optimal normalization parameters a and b using a first criterion that minimizes the lightweight range without degrading performance. Alternatively, the normalization parameters may be determined using a second criterion in which the normalization is repeated until the recognition rate becomes maximum. In this case, the degree of lightening can be reduced, but the performance can be maximized.

Referring to FIG. 7B, normalization may be performed using an error rate instead of the recognition rate. The error rate can be defined in various ways. For example, as an example of the error rate, a face classification learning loss indicating a loss in face recognition may be used.

In this case, as the iteration is repeated, the error rate can be reduced and increased. For example, the initial error rate may be a first reference value. Depending on the iteration i ₁ times repeated error rate is decreased to the second reference value. Thereafter, the error rate gradually increases and increases to the first reference value as the iteration is repeated i ₂ times.

As described above, in order to maximize the weight reduction, the lightening device can determine the candidate range according to the repetition of i ₂ times as the lightweight range by using the first criterion. In some cases, light weight device can be repeated using a second reference only i ₁ times to maximize performance with a certain degree of weight to determine the weight range as the candidate range. In this case, the degree of lighter weight can be reduced compared to repeating i ₂ times, but the performance can be maximized.

According to one embodiment, normalization may be performed together while the parameters are being learned. For example, as will be described later with reference to FIG. 12 or FIG. 13, normalization is performed on intermediate parameters in which learning is in progress by the learning apparatus, so that normalization may affect learning. In this case, the number of iterations of learning and normalization can be determined based on the above-described criteria through Figs. 7A and 7B.

FIG. 8 is a diagram illustrating a variation of a sequence by normalization according to an embodiment. Referring to FIG. 8, a sequence 810 and a normalized sequence 820 are shown before normalization. In the sequences 810 and 820, V _i represents a parameter, and i represents an index of a parameter other than 0. Although the sequences 810 and 820 are represented as including sixteen parameters, the sequences 810 and 820 may include parameters corresponding to the number of synapses. As described with reference to FIG. 2, the parameters may be quantized into a 16-bit fixed-point type integer. Thus, the parameters of the sequence 810 may have a 16-bit size.

The lightening device may reduce the size of the data representing the parameters based on the lightweight range. As the distribution range of the parameters decreases from the original range to the lightweight range, the number of bits for representing the parameters decreases, so that the light weighting device can express the parameters according to normalization with a smaller number of bits than before normalization. For example, a lightweight device may determine the minimum number of bits that can represent elements in a finite set corresponding to a lightweight range, and may express parameters based on the determined minimum number of bits. Specifically, the light-weighting device can determine min (x) satisfying the condition (2 ^x ? A) as m and max (x) satisfying the condition (2 ^x ? B) as n. In this case, the parameters of the sequence 820 will have a size of (nm) bits. Since the lightweight range is narrower than the original range, (nm) is less than 16. Thus, the size is reduced by 16- (nm) bits per parameter according to the normalization.

9 is a diagram illustrating a change of a sequence by compression according to an embodiment. A parameter with a value of 0 before compression occupies the same memory space as a parameter with a non-zero value. By minimizing the memory space occupied by the parameter having a value of 0 through the compression described below, weight reduction due to normalization can be maximized.

Referring to FIG. 9, a sequence 910, a bit sequence 920, and a non-zero sequence 930 are shown before compression. The bit sequence 920 and the non-zero sequence 930 represent the compression state of the sequence 910. V _i represents a parameter, and i represents an index of a parameter other than 0. The parameters of the sequence 910 have a size of (nm) bits according to the normalization. The light weighting device can compress the sequence 910 losslessly based on the parameter whose value is zero. As will be described later, the sequence 910 before compression can be restored as it is in the compressed sequence 920, 930, so that compression according to the embodiment corresponds to lossless compression.

The lightening device includes a non-zero sequence 920 including only parameters whose values are not 0 among the parameters included in the sequence 910 and a bit sequence 930 indicating whether the value of the parameter is 0 in the sequence 910 Can be generated. In the bit sequence 930, each data may be represented by one bit. The bit sequence 930 may comprise the same number of bits as the number of parameters included in the sequence 910. [ Each bit included in the bit sequence 930 may correspond to each parameter in the sequence 910 on a one-to-one basis. It indicates that the value of the corresponding parameter in sequence 910 is not zero if the particular bit contained in bit sequence 930 is 1 and indicates that the value of the corresponding parameter in sequence 910 is not zero if the particular bit contained in bit sequence 930 is zero. It may indicate that the value of the corresponding parameter is zero. (n-m) * sequence 910 having a bit size as many as the total number of parameters can be represented by (n-m) * non-zero parameter number + parameter number of bits through lossless compression described above. This compression effect can be maximized as the number of parameters having a value of 0 increases.

FIG. 10 is a diagram illustrating a distribution change of parameters by weight reduction according to an embodiment. Referring to FIG. 10, a distribution 1010 before lightening, a distribution 1020 after quantization, a distribution 1030 after normalization, and a distribution 1040 after compression are shown. It can be seen that the number of parameters having a value of 0 is the largest among the respective distributions, and the frequency decreases as the value of the parameter increases. Since the quantization only changes the representation format of the parameters, there is no change in the distribution according to the quantization.

The parameters within the cut-off range (-a to a) are truncated to zero, and the parameters having a value of the maximum value (b) or more and the parameters having the value of the minimum value (-b) (-b). Thus, referring to distribution 1030, the frequency of the parameters having a value of zero increases due to the cutting operation, and the frequency of the parameters having the maximum value b and the minimum value -b due to the cutoff operation increases.

The parameters with a value of zero in distribution 1030 are removed upon compression. The distribution 1040 includes only the non-zero parameters described above in accordance with compression. Parameters with a value of zero can be identified through the bit sequence described above. In the distribution 1030, the parameters are distributed over -b to -a, 0, and a through b, and the lightweighting device can reduce the absolute value of the parameters by a to minimize the number of bits that can represent the parameters in the normalization process . In this case, referring to distribution 1040, the values of the positive parameters may be decreased by a and the values of the negative parameters may be increased by a. As the absolute value of the parameters decreases, the maximum value decreases from b to ba, so that the number of bits capable of expressing the parameters can be reduced. Thus, the lightweight range can be determined from - (ba) to ba. In this case, each of the parameters may be represented by a number of bits corresponding to log ₂ (ba). For example, if the parameters have integer values in the lightweight range, the number of cases for each of the parameters may be 2 * (ba) +1. Each of the parameters may be represented by bits of least natural number equal to or greater than log ₂ (2 * (ba) +1).

According to one embodiment, the weight reduction process described above can be applied to each layer in the neural network. In other words, lightweighting can be applied differently for each layer. For example, in the case of a neural network including a total of 10 convolution layers, weight reduction is not applied to the first layer, only weight reduction by the cutoff operation is applied to the second to eighth layers, And lightening by the cutting operation is applied, and only the lightening by the cutting operation can be applied to the tenth layer. The recognition performance of the optimal lightweight neural network is the same as before the weight reduction, rather than before, and the model size for the neural network can be greatly reduced.

In addition, if the output map channels in each layer can be classified, such as a convolution layer, weighting may be applied differently for each channel. For example, in the case of a convolution layer, an output map channel may exist as many as the number of channels of the kernel corresponding to the corresponding layer.

The neural network according to an embodiment may be one in which normalization by a cutting operation is applied. For example, the neural network may include a plurality of layers each including neurons and synapses connecting neurons contained in neighboring layers, wherein the synapse weights greater than 0 and less than a (a > 0) Is set to zero. At this time, the synapse weights set to 0 may correspond to synapses connecting neighboring layers of some of the plurality of layers. The normalization variable a may be set differently for a plurality of layers or differently for each output map channel in a specific layer.

In addition, the neural network may be a normalized by the cutting operation and the cutoff operation. For example, at least some of the synaptic weights having a value greater than b (b > a) in the neural network may be set to b. At this time, the synapse weights set to b may correspond to synapses connecting neighboring layers of some of the plurality of layers. The normalization variable b may be set differently for a plurality of layers or differently for each output map channel in a specific layer.

The neural network according to another embodiment may be one in which the normalization by the cutoff operation is applied. For example, the neural network may comprise a plurality of layers each including neurons and synapses connecting neurons contained in neighboring layers, wherein at least some of the synaptic weights having a value greater than b (b > 0) Is set to b. At this time, the synapse weights set to b may correspond to synapses connecting neighboring layers of some of the plurality of layers. The normalization variable b may be set differently for a plurality of layers or differently for each output map channel in a specific layer.

11 is a diagram illustrating a post-processing process of learned parameters according to an embodiment. 11, the learning apparatus 1110 includes a learning database 1111, a parameter adjusting unit 1112, and a plurality of layers 1113. [ The learning apparatus 1110 can learn a plurality of layers 1113 with the learning database 1111. [ The parameter adjuster 1112 can adjust the parameters of the plurality of layers 1113 based on the loss through the first layer to the nth layer. The learning device 1110 may send the learned parameters to the lightening device 1120. [ The lightening device 1120 can lighten the parameters learned through the post-processing process. The lightening device 1120 can lighten the learned parameters based on at least one of quantization, normalization, and compression described above. The lightening device 1120 may store the lightened parameters in the storage device 1130. [ The lightened parameters stored in the storage device 1130 can be used in the recognition process. 11, the learning apparatus 1110, the light-weighting apparatus 1120 and the storage apparatus 1130 are shown as being separated from each other, but the learning apparatus 1110, the light-weighting apparatus 1120 and the storage apparatus 1130 All three can be configured as a single device. The lightening device 1120 may comprise a hardware module or software module for quantization, normalization and / or compression.

12 is a diagram illustrating a parameter tuning process according to an embodiment. Referring to FIG. 12, the parameter tuning device 1220 includes a learning database 1221 and a plurality of layers 1222. The parameter tuning device 1220 may further learn a plurality of layers 1222 to fine tune the learned parameters received from the learning device 1210. [

The parameter tuning device 1220 applies the learned parameters received from the learning device 1210 to the plurality of layers 1222 and adds each of the first layer to the nth layer included in the plurality of layers 1222 . At this time, the parameters of the remaining layers other than the layer to be subjected to the additional learning can be fixed. For example, when the parameter tuning apparatus 1220 further learns the n-th layer, the parameters of the first layer to the (n-1) -th layer can be fixed.

The lightening device 1230 may be used for additional learning. The lightening device 1230 can normalize the parameter of the layer to be subjected to the additional learning using a function for evaluating the loss of the feature vector. Specifically, the lightening device 1230 may normalize the candidate range that minimizes the loss of feature vectors to a lightweight range. Although not shown in FIG. 12, the lightening device 1230 may quantize the parameters.

The lightening device 1230 may compress the normalized parameters to produce lightweight parameters. The lightening device 1230 may store the lightened parameters in the storage device 1240. [ The lightened parameters stored in the storage device 1240 may be used in the recognition process.

In FIG. 12, a plurality of layers 1222 are shown as overlapping layers in each step. However, this is for convenience of description. The plurality of layers 1222 include only one layer of each layer . In this case, each of the first layer to the n-th layer included in the plurality of layers 1222 may be additionally learned sequentially.

12, the learning apparatus 1210, the parameter tuning apparatus 1220, the light-weighting apparatus 1230 and the storage apparatus 1240 are shown as being separated from each other, but the learning apparatus 1210, the parameter tuning apparatus 1220, Two or more of the lightening device 1230 and the storage device 1240 may be included in the same device. The lightening device 1230 may include hardware modules or software modules for quantization, normalization and / or compression.

13 is a diagram illustrating a learning process according to an embodiment. 13, the learning apparatus 1310 includes a learning database 1311, a plurality of layers 1312, and a lightening device 1320. [ The lightening device 1320 may normalize the parameters in the learning process of the plurality of layers 1312. [

The lightening device 1320 may learn a plurality of layers 1312 based on parameters represented by data of reduced size in accordance with the normalization. For example, the learning apparatus 1310 may learn a plurality of layers 1312 based on the learning database 1311. [ The lightening device 1320 may quantize the parameters of the n-th layer and normalize the parameters of the n-th layer so that the loss of the n-th layer is minimized. The parameters of the normalized nth layer may be stored in the storage device 1330 after being compressed. The parameters of the normalized nth layer can be dequantized and applied to the nth layer. Next, the lightening device 1320 can perform quantization, normalization, compression, and dequantization on the (n-1) th layer in the same manner as the nth layer. The lightened parameters stored in the storage device 1330 can be used in the recognition process.

13, the learning device 1310 is shown as including the lightening device 1320, but the learning device 1310 and the lightening device 1320 can be separated from each other. 13, the learning device 1310 and the storage device 1330 are shown as being separated from each other, but the learning device 1310 and the storage device 1330 can constitute one device. The lightening device 1230 may include hardware modules or software modules for quantization, dequantization, normalization and / or compression.

FIG. 14 is a diagram illustrating a recognition process according to an embodiment. 14, the recognition apparatus 1410 includes a plurality of layers 1411, a recognizer 1412, and a restoration apparatus 1420. [ The restoration device 1420 can restore the lightened parameters and apply the restored parameters to the plurality of layers 1411. [ The plurality of layers 1411 extracts a feature vector from the input data based on the applied parameters, and the recognizer 1412 can recognize the object from the input data based on the feature vector.

The restoration device 1420 may obtain the lightened parameters from the above-mentioned storage device or the like. The restoration device 1420 may restore the parameters by performing at least one of denormalization, dequantization, and decompression according to techniques applied to the lightened parameters. For example, if normalization is applied to lightened parameters, the restoration device 1420 can perform non-normalization on the lightened parameters. The restoration device 1420 may increase the absolute value of the parameters by a through denormalization. Thus, the value of positive parameters increases by a, and the value of negative parameters decreases by a.

When quantization is applied to the lightened parameters, the reconstruction device 1420 can perform dequantization on the lightened parameters. The decompression apparatus 1420 may convert the representation of quantized parameters through dequantization into a system-specific representation. For example, if the parameters are quantized to 16-bit fixed-point type integers, they can be unquantized as a real number of 32-bit floating point type. According to an embodiment, when a plurality of layers 1411 use a fixed-point data type, dequantization may not be performed.

If the lightened parameters are compressed, the decompression device 1420 can decompress the lightened parameters. The decompression device 1420 may decompress the lightweighted parameters based on the bit sequence and the non-zero sequence indicating whether the value of the parameter is zero. In order to prevent the unnecessary resource consumption due to the data processing on the parameters having the value of 0, the decompression apparatus 1420 performs decompression, dequantization, and decompression The decompression can be applied the slowest.

FIG. 15 is a diagram illustrating a distribution change of parameters by restoration according to an embodiment. Referring to FIG. 15, a distribution 1510 of lightened parameters, a distribution of unqualified parameters 1520, a distribution of unquantized parameters 1530, and a distribution of decompressed parameters 1540 are shown. In the light-weighted state, the parameters are distributed in the light-weight range of - (b-a) to (b-a). In the distribution 1520, the absolute values of the parameters are increased by a with non-normalization, and the parameters are distributed over -b to -a and a to b. Since the dequantization only affects the representation form of the data, there is no change in the distribution according to the dequantization in the distribution (1530). In distribution 1540, a number of parameters with a value of zero are generated in response to decompression.

16 is a diagram illustrating a change in sequence by decompression according to an embodiment. Referring to Figure 16, a non-zero sequence (V _k ), a bit sequence (LO _i ) and a decompressed sequence (W _i ) are shown. Here, i and k represent indices. The decompression device can be easily decompressed through the following Table 1.

for (i = 0, k = 0; i <Len; i ++)
{
W _i = V _k * LO _i ;
k + = LO _i ;
}

Here, Len represents the length of the bit sequence (LO _i ). Referring to Table 1 and FIG. 16, the restoration apparatus multiplies the bit value of the index (i = 0) in the bit sequence (LO _i ) by the parameter of the index (k = 0) in the non-zero sequence (V _k ) (V ₁ ) decompressed in the decompressed sequence W _i (i = 0). Next, the restoration device increases the index k by the bit value (1) of the index (i = 0) in the bit sequence (LO _i ). Therefore, the index k is increased only when the bit value in the bit sequence LO _i is 1. Next, the restoration device increments the index (i) by one. By repeating these steps according to Table 1, the decompressed sequence W _i can be obtained. Since the decompression technique described with reference to FIG. 16 is only one embodiment, the compression of the lightened parameters can be released with other decompression techniques.

FIG. 17 is a diagram illustrating a sequence change by division according to an embodiment. Referring to FIG. 17, a sequence 1710 before division and a sequence 1720 after division are shown. As mentioned above, the compression effect can be improved as the number of parameters having a value of 0 in the compression process increases. Therefore, it is possible to increase parameters having a value of 0 through division. Divide V of sequence 1710 by a predetermined divisor, then sequence 1710 can be represented as sequence 1720. [ In the sequence 1720, the sub-parameter Vq represents a quotient and the sub-parameter Vr represents the remainder. Referring to the distributions of FIGS. 10 and 15, the parameters have values that are generally not large. Therefore, the quotient can be made 0 by appropriately selecting the divisor, and the parameters having a value of 0 in the sequence can be increased.

FIG. 18 is a diagram showing a change of a sequence by division and compression according to an embodiment. Referring to FIG. 18, a sequence 1810 before division, a sequence 1820 after division, a non-zero sequence 1830, and a bit sequence 1840 are shown. The parameters of the sequence 1810 may have a size of (nm) -bit and the parameters of the sequence 1820 may have the size of (nm) / 2-bit. As an example, the sequence 1820 can be obtained by dividing the parameters of the sequence 1810 by a divisor of 2 ^{(nm) / 2} . Referring to sequence 1820, the quotient of dividing the parameters v ₂ , v ₄ and v ₅ by 2 ^{(nm) / 2} is 0 and v ₂ , v ₄ and v ₅ are less than 2 ^{(nm) / 2} Able to know. Thus, sub-parameters with a value of zero are additionally generated through division.

The non-zero sequence 1830 includes only subparameters whose value is not zero in the sequence of subparameters, and the bit sequence 1840 indicates whether the subparameter is zero. In order to express the sequence 1810, bits corresponding to the number of (nm) * parameters are required. When compressing the sequence 1810 without applying the division, (nm) The number of bits is required for the number of parameters. When compressing the sequence 1810 with division applied, bits as many as (n-m) / 2 * non-zero sub-parameters + total non-zero sub-parameters are required. As the number of subparameters with a value of 0 increases by division, the compression effect can be improved.

19 is an operational flowchart illustrating a lightening method according to an embodiment. Referring to Fig. 19, in step 1910, the lightening device obtains parameters corresponding to the layers included in the neural network. In step 1920, the lightening device extracts a lightweight range that corresponds to a portion of the original range in which the parameters are distributed, based on a predefined performance function. In step 1930, the lightening device reduces the size of the data representing the parameters based on the lightweight range. In addition, since the above-described operations can be applied to the lightening method, detailed description will be omitted.

20 is an operation flowchart showing a recognition method according to an embodiment. Referring to Fig. 20, at step 2010, the recognition device obtains normalized parameters based on the distribution of parameters corresponding to the layers included in the neural network. In step 2020, the recognizing device denormalizes the normalized parameters based on the normalization variable corresponding to the layer. In step 2030, the recognizing device applies un-normalized parameters to the layer. In step 2040, the recognizing device recognizes input data using a neural network. In addition, since the above-described operations can be applied to the recognition method, a detailed description will be omitted.

21 is a diagram illustrating an electronic system according to one embodiment. 21, the electronic system includes a sensor 2110, a processor 2120, and a memory 2130. [ Sensor 2110, processor 2120 and memory 2130 may communicate with each other via bus 2140. The lightening device, restoration device, learning device, recognition device, and the like described above can be implemented as at least a part of the electronic system.

The sensor 2110 may include an image sensor and a microphone for sensing image data and voice data for object recognition. The sensor 2110 may sense an image or the like in a well-known manner (e.g., a manner of converting an optical image into an electrical signal, etc.). The output of the sensor 2110 is output to the processor 2120 or the memory 2130.

The processor 2110 may include at least one of the devices described above with reference to Figures 1 to 20, or may perform at least one of the methods described above with respect to Figures 1 to 20. [ For example, processor 2110 may be configured to obtain parameters corresponding to the layers included in the neural network to perform weight reduction, and based on a predefined performance function, , And reduce the size of data representing the parameters based on the lightweight range. The processor 2110 may also be configured to obtain normalized parameters based on a distribution of parameters corresponding to the layers included in the neural network to perform object recognition and to perform the normalization on the basis of the normalization variable corresponding to the layer Denormalized parameters, applying the non-normalized parameters to the layer, and recognizing the input data using the neural network.

The memory 2130 may store the aforementioned lightening variables, performance functions, performance indicators, and lightweight parameters. Also, the memory 2130 may include instructions readable by a computer. The processor 2110 may perform the aforementioned operations as the instructions stored in the memory 2130 are executed in the processor 2110. The memory 2130 may be a volatile memory or a non-volatile memory.

The processor 2110 may execute the program and control the electronic system. The electronic system is connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown) and can exchange data. The electronic system can be used in various electronic systems such as mobile devices, smart phones, PDAs, tablet computers, mobile devices such as laptop computers, computing devices such as personal computers, tablet computers, netbooks, or electronic devices such as televisions, smart televisions, Lt; / RTI &gt; The user can learn the model from a user device such as a mobile device through a lightweight neural network.

The embodiments described above may be implemented in hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented within a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate Such as an array, a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the drawings, various technical modifications and variations may be applied to those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Claims (28)

A neural network comprising a plurality of layers each including neurons and synapses connecting the neurons contained in neighboring layers,
Wherein at least some of the synapse weights having a value greater than 0 and less than a (a > 0) are set to zero. The method according to claim 1,
Wherein the synapse weights set to zero correspond to synapses connecting neighbors of some of the plurality of layers. The method according to claim 1,
wherein at least some of the synaptic weights having a value greater than b (b > a) are set to b. The method of claim 3,
Wherein the synapse weights set to b correspond to synapses connecting neighboring layers of some of the plurality of layers. The method according to claim 1,
Wherein the a is set differently for each of the plurality of layers. The method according to claim 1,
Wherein a is set differently for each intra-layer output map channel. The method of claim 3,
And b is set differently for the plurality of layers. The method of claim 3,
And b is set differently for each intra-layer output map channel. The method according to claim 1,
Wherein each of the synaptic weights having a value equal to or greater than a is a maximum number of bits corresponding to log ₂ (max-a), wherein max is a maximum synaptic weight greater than a, and a and max are integers Neural network. The method of claim 3,
Wherein each of the synaptic weights having a value equal to or greater than a and equal to or less than b is expressed by a number of bits corresponding to log ₂ (ba), where a and b are integers. A neural network comprising a plurality of layers each including neurons and synapses connecting the neurons contained in neighboring layers,
wherein at least some of the synapse weights having a value greater than b (b > 0) are set to b. 12. The method of claim 11,
Wherein the synapse weights set to b correspond to synapses connecting neighboring layers of some of the plurality of layers. 12. The method of claim 11,
And b is set differently for the plurality of layers. 12. The method of claim 11,
And b is set differently for each intra-layer output map channel. 12. The method of claim 11,
Wherein each of the synapse weights having a value less than or equal to b is expressed by a number of bits corresponding to log ₂ (b), where b is an integer. Obtaining regularized parameters corresponding to the layers included in the neural network;
De-normalizing the normalized parameters based on a normalization variable corresponding to the layer;
Applying the non-normalized parameters to the layer; And
Recognizing input data using the neural network
/ RTI &gt; 17. The method of claim 16,
The normalization variable corresponding to the layer is
Wherein the plurality of output map channels are set differently for each of a plurality of layers included in the neural network or different for each of a plurality of output map channels included in the layer. 17. The method of claim 16,
The normalization variable
And an offset to shift the normalized parameters based on zero. 17. The method of claim 16,
When the denormalized parameters are m-bit integers and the neural network receives an n-bit real number greater than the m-bit,
Wherein applying the non-normalized parameters to the layer comprises:
De-quantizing the denormalized parameters by a real number of n bits; And
Applying the non-quantized parameters to the layer
&Lt; / RTI &gt; 17. The method of claim 16,
Wherein applying the non-normalized parameters to the layer comprises:
Obtaining a bit sequence corresponding to the layer, the bit sequence indicating whether the value of the parameter is 0;
Decompressing said non-normalized parameters constituting a non-zero sequence based on said bit sequence; And
Applying the decompressed parameters to the layer
/ RTI &gt; 21. The method of claim 20,
Wherein decompressing the denormalized parameters comprises:
Determining a decompressed parameter of the first index by multiplying a bit value of the first index in the bit sequence by a parameter of a second index in the non-zero sequence;
Increasing the second index by a bit value of the first index in the bit sequence; And
Increasing the first index by one
/ RTI &gt; 21. A computer program stored in a medium for executing the method of any one of claims 16 to 21 in combination with hardware. A processor; And
A memory containing instructions that can be read by the computer
Lt; / RTI &gt;
If the instruction is executed in the processor, the processor obtains regularized parameters corresponding to the layers contained in the neural network, and normalizes the normalized parameters based on the normalized variable corresponding to the layer de-regularization, applying the non-normalized parameters to the layer, and recognizing input data using the neural network. 24. The method of claim 23,
The normalization variable corresponding to the layer is
Wherein the plurality of output map channels are set differently for each of a plurality of layers included in the neural network or different for each of a plurality of output map channels included in the layer. 24. The method of claim 23,
The normalization variable
And an offset to shift the normalized parameters based on zero. 24. The method of claim 23,
When the denormalized parameters are m-bit integers and the neural network receives an n-bit real number greater than the m-bit,
Wherein the processor de-quantizes the non-normalized parameters by the real number of the n bits and applies the non-quantized parameters to the layer. 24. The method of claim 23,
The processor
Corresponding to the layer, obtaining a bit sequence indicating whether the value of the parameter is 0, decompressing the non-normalized parameters constituting the non-zero sequence based on the bit sequence, And applies the decompressed parameters. 28. The method of claim 27,
The processor
Determining a decompressed parameter of the first index by multiplying a bit value of a first index in the bit sequence by a parameter of a second index in the non-zero sequence to decompress the non-normalized parameters, Increases the second index by a bit value of the first index in the bit sequence and increases the first index by one.

Images