KR102037483B1 - Method for normalizing neural network data and apparatus thereof - Google Patents

Abstract

Provided are a normalization method and an apparatus thereof capable of preserving style information contained in neural network data even if the size of arrangement is small. According to some embodiments of the present invention, a neural network data normalization method is capable of obtaining first feature data for a first training image outputted from a specific layer of the neural network, obtaining second feature data for a second training image outputted from the specific layer, generating a first style adjustment value for the first feature data based on the first feature data, generating a second style adjustment value for the second feature data based on the second feature data, normalizing the first feature data using the generated first style adjustment value, and normalizing the second feature data using the generated second style adjustment value. Therefore, even if normalization is performed, style information contained in feature data may be preserved.

Description

Neural network data normalization method and apparatus therefor {METHOD FOR NORMALIZING NEURAL NETWORK DATA AND APPARATUS THEREOF}

The present disclosure relates to a neural network data normalization method and apparatus thereof. More specifically, the present invention relates to a method for normalizing the neural network data in consideration of style information inherent in the neural network data, and an apparatus for supporting the method.

A neural network is a machine learning model that simulates the structure of human neurons. The neural network consists of one or more layers, and the output data of each layer is used as the input of the next layer. In recent years, research on the use of a deep neural network composed of a plurality of layers has been intensively conducted. The deep neural network plays an important role in improving recognition performance in various fields such as speech recognition, natural language processing, and lesion diagnosis.

Deep neural networks contain a number of hidden layers, allowing them to learn a variety of nonlinear relationships. However, learning a large number of hidden layers may result in overfitting, vanishing gradient problems, and the like. In order to solve this problem, the normalization technique (normalization) is mainly used in the field of machine learning. In addition, the normalization technique is used for various purposes such as stabilization of learning and speed of learning.

Korean Patent Publication No. 10-2017-0108081 (published Sep. 26, 2017)

The technical problem to be solved by some embodiments of the present disclosure is to provide a normalization method that can preserve the style information of neural network data associated with the training image, even if the batch size is not large enough, and an apparatus supporting the method. will be.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, a neural network data normalization method according to some embodiments of the present disclosure includes a method for normalizing data of a neural network in a computing device, the method comprising: generating a first training image output from a specific layer of the neural network; Obtaining first feature data, obtaining second feature data for a second training image output from the specific layer, and generating a first style adjustment value for the first feature data based on the first feature data Generating a second style adjustment value for the second feature data based on the second feature data; normalizing the first feature data using the generated first style adjustment value; and generating Normalizing the second feature data using the second style adjustment value.

In some embodiments, generating the first style adjustment value comprises extracting style information of the first feature data and generating the first style adjustment value using the extracted style information. can do.

In some embodiments, the style information may include at least one of an average and a standard deviation of the feature data.

In some embodiments, the generating of the first style adjustment value may include generating the first style adjustment value by reflecting a value of a style adjustment parameter updated with the weight of the neural network in the extracted style information. It may include.

In some embodiments, the feature data includes first feature map data of a first channel and second feature map data of a second channel, and the style information includes a first feature for the first feature map data. And second style information about the second feature map data, wherein the generating of the first style adjustment value comprises: the first style information and the first style information through a fully-connected layer; And aggregating second style information to generate the first style adjustment value.

In some embodiments, the feature data includes first feature map data of a first channel and second feature map data of a second channel, and the style information includes a first feature for the first feature map data. And second style information about the second feature map data, wherein the generating of the first style adjustment value comprises: using the first style information, a style adjustment value corresponding to the first channel; And generating a style adjustment value corresponding to the second channel by using the second style information.

In some embodiments, normalizing the first feature data may include converting the first feature data into first normalized data using the statistical information of a batch to which the first training image belongs and generating the first feature data. And performing affine transformation on the first normalized data by using the first style adjustment value.

In some embodiments, the generating of the first style adjustment value comprises: a first style adjustment value through a convolution operation with a convolution filter updated with the first feature data and the weight of the neural network; It may include the step of generating.

In order to solve the above technical problem, the neural network data normalization apparatus according to some embodiments of the present disclosure is output in a specific layer of a neural network by executing a memory for storing one or more instructions and the stored one or more instructions. Obtaining first feature data for a first training image, obtaining second feature data for a second training image output from the particular layer, and a first style for the first feature data based on the first feature data Generate an adjustment value, generate a second style adjustment value for the second feature data based on the second feature data, normalize the first feature data using the generated first style adjustment value, A processor to normalize the second feature data using the generated second style adjustment value It can be included.

In order to solve the above technical problem, a computer program according to some embodiments of the present disclosure may be combined with a computing device to obtain first feature data for a first training image output from a specific layer of a neural network, wherein the specific layer Obtaining second feature data for the second training image output from the step of generating a first style adjustment value for the first feature data based on the first feature data, based on the second feature data Generating a second style adjustment value for the second feature data, normalizing the first feature data using the generated first style adjustment value, and using the generated second style adjustment value In order to carry out the step of normalizing the second characteristic data, it may be stored in a computer-readable recording medium.

In order to solve the above technical problem, a neural network data normalization method according to another embodiment of the present disclosure, in a method of normalizing data of a neural network in a computing device, is characterized in that a training image is output from a first layer of the neural network. Obtaining data, extracting style information of the feature data, normalizing the feature data based on the style information, and providing the normalized feature data as an input of a second layer of the neural network. can do.

In order to solve the above technical problem, the neural network data normalization apparatus according to another embodiment of the present disclosure, in the first layer of the neural network by executing a memory and one or more stored instructions to store one or more instructions; Obtaining feature data for the output training image, extracting style information of the feature data, normalizing the feature data based on the style information, and converting the normalized feature data into an input of a second layer of the neural network. It may include a processor to provide as input of the second layer of provision.

In order to solve the above technical problem, a computer program according to another exemplary embodiment of the present disclosure may be combined with a computing device to obtain feature data for a training image output from a first layer of a neural network, and the style of the feature data. Extracting information, normalizing the feature data based on the style information, and providing the normalized feature data as input of a second layer of the neural network to a computer-readable recording medium. Can be.

1 is a diagram illustrating a neural network learning apparatus and its learning environment according to some embodiments of the present disclosure.
2 is a diagram for explaining a batch normalization technique.
3 is a conceptual diagram schematically illustrating a neural network data normalization method according to some embodiments of the present disclosure.
4 is an exemplary flowchart illustrating a neural network data normalization method according to some embodiments of the present disclosure.
5 is an exemplary diagram for describing a structure and a normalization layer of a neural network that may be referenced in various embodiments of the present disclosure.
6 is an exemplary diagram for describing feature data that may be referred to in some embodiments of the present disclosure.
7 is an exemplary flowchart for describing a style adjustment process according to some embodiments of the present disclosure.
8 to 10 are exemplary diagrams for further describing a style adjusting process according to some embodiments of the present disclosure.
11 is an exemplary diagram for describing a style adjusting process according to some other embodiments of the present disclosure.
12 is an exemplary diagram for describing a neural network data normalization method according to another exemplary embodiment of the present disclosure.
13 illustrates an example computing device that may implement an apparatus in accordance with various embodiments of the present disclosure.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Advantages and features of the present disclosure, and methods of accomplishing the same will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the technical spirit of the present disclosure is not limited to the following embodiments, and may be implemented in various forms, and only the following embodiments make the technical spirit of the present disclosure complete and in the technical field to which the present disclosure belongs. It is provided to fully inform those skilled in the art the scope of the present disclosure, and the technical spirit of the present disclosure is only defined by the scope of the claims.

In adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible even though they are shown in different drawings. In addition, in describing the present disclosure, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present disclosure, the detailed description thereof will be omitted.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art to which the present disclosure belongs. In addition, the terms defined in the commonly used dictionaries are not ideally or excessively interpreted unless they are specifically defined clearly. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

In addition, in describing the components of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only for distinguishing the components from other components, and the nature, order or order of the components are not limited by the terms. If a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected to or connected to that other component, but there may be another configuration between each component. It is to be understood that the elements may be "connected", "coupled" or "connected".

As used herein, “comprises” and / or “comprising” refers to a component, step, operation and / or element that is mentioned in the presence of one or more other components, steps, operations and / or elements. Or does not exclude additions.

Prior to the description herein, some terms used herein will be clarified.

In the present specification, a task refers to a task to be solved through machine learning or a task to be performed through machine learning. For example, when face recognition, facial expression recognition, gender classification, pose classification, and the like are performed from face data, each of face recognition, facial expression recognition, gender classification, and pose classification may correspond to an individual task. As another example, when performing recognition, classification, prediction, or the like of abnormality from medical image data, each of the abnormality recognition, the abnormality classification, and the abnormality prediction may correspond to an individual task. And the task may be referred to as the destination task.

In this specification, a neural network is a term encompassing all kinds of machine learning models designed to mimic neural structures. For example, the neural network may include all kinds of neural network based models such as an artificial neural network (ANN), a convolutional neural network (CNN), and the like.

In this specification, neural network data may include all kinds of data generated or input / output in each layer constituting the neural network. For example, the neural network data may include feature data (e.g. feature map) of the training image generated as the training image is forwarded through the neural network.

Instructions herein refer to a series of computer readable instructions, bound by function, that refer to components of a computer program and to be executed by a processor.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 illustrates a neural network learning apparatus 10 and a learning environment in accordance with some embodiments of the present disclosure.

As shown in FIG. 1, the neural network learning apparatus 10 is a computing device that performs machine learning on a neural network to perform a target task. In this case, the object task may be a task (e.g. object recognition) associated with the image, but the technical scope of the present disclosure is not limited to the type of task or neural network. Hereinafter, for convenience of explanation, the neural network learning apparatus 10 will be abbreviated as the learning apparatus 10.

The computing device may be a laptop, a desktop, a laptop, a server, or the like, but is not limited thereto and may include any type of device having a computing function. An example of the computing device is described with reference to FIG. 13.

Although FIG. 1 illustrates that the learning device 10 is implemented as one computing device, the first function of the learning device 10 is implemented in the first computing device in an actual physical environment, and the learning device 10 is illustrated in FIG. The second function of may be implemented in the second computing device. In addition, the learning device 10 may be composed of a plurality of computing devices, and the plurality of computing devices may be implemented by dividing the first function and the second function.

As shown in FIG. 1, the learning apparatus 10 may train a neural network (e.g. a convolutional neural network) using a dataset 11 composed of a plurality of training images. At least some of the plurality of layers constituting the neural network may include a normalization layer. In addition, the learning apparatus 10 may perform normalization on feature data (e.g. feature map) extracted from a training image through the normalization layer.

In some embodiments, the normalization may be performed based on a batch normalization technique. A description of the batch normalization technique is shown in FIG. As shown in FIG. 2, the batch normalization technique is a technique for normalizing neural network data through a normalization process 15 and an affine transformation process 17. At this time, the normalization process 15 is performed based on batch statistical information (ie, mean and standard deviation) of neural network data.

On the other hand, when the batch normalization technique is analyzed in terms of the style information of the image, it can be seen that the preservation of the style information is closely related to the batch size. Specifically, when the batch size is set large enough, the batch normalization technique may well preserve the style information embedded in the neural network data, but in the opposite case, the style information may be distorted or lost through batch normalization. This is because the smaller the batch size, the larger the bias included in the batch statistics information, and thus, the style information contained in the neural network data may be lost. This can be seen as the same principle that the style information is completely removed by the instance normalization technique.

However, when training neural networks, there are cases where the batch size cannot be set sufficiently large for various reasons such as memory constraints. In addition, style information may play an important role in a specific image recognition problem. For example, when trying to recognize the weather or time zone from an image, style information such as brightness and contrast of the image may be an important clue for recognizing the weather or time zone. In this case, if the neural network is trained with a small batch size, the neural network performance is inevitably degraded due to loss of style information.

Accordingly, there is a need for a normalization method that can preserve style information embedded in neural network data even when the batch size is not large enough. Hereinafter, various embodiments of the present disclosure will be described in detail.

In some embodiments, the learning apparatus 10 may perform normalization so that style information embedded in the feature data is preserved. Specifically, the learning apparatus 10 generates a style adjustment value of the feature data based on the style information, adjusts a parameter value of an affine transformation with the style adjustment value, and performs an affine transformation with the adjusted value. Can be done. By doing so, distorted or lost style information in the normalization process (ie, the normalization process performed before the affine transformation) can be corrected or recovered. According to this embodiment, even when the batch size is not large enough, the style information embedded in the feature data can be preserved, whereby the performance of the neural network can be improved.

In embodiments related to normalization as described above, the learning device 10 may be referred to as the neural network data normalization device 10. The normalization operation of the neural network data normalization apparatus 10 will be described later in more detail with reference to the drawings below.

After the training of the neural network is completed, the learning apparatus 10 may perform the object task using the neural network. For example, the learning apparatus 10 may provide a prediction result based on a prediction value (e.g., a confidence score for each class) obtained by inputting the real image 13 to the neural network. The confidence score for each class may be a final output probability distribution for each class of the model that has passed the softmax activation function.

So far, the learning apparatus 10 and its learning environment according to some embodiments of the present disclosure have been described with reference to FIGS. 1 and 2. Hereinafter, a neural network data normalization method according to various embodiments of the present disclosure will be described.

3 is a conceptual diagram schematically illustrating a neural network data normalization method according to various embodiments of the present disclosure.

As shown in FIG. 3, various neural network data including the feature data 21 may be normalized through the normalization process 22 and the affine transformation process 23. In this case, the style information of the feature data e.g. 21 may be lost in the normalization process 22. To prevent loss of style information, the neural network data normalization method according to various embodiments of the present disclosure may determine the value 25 of the affine transformation parameter suitable for the feature data 21 through the style adjustment process 24. . The method may also recover lost style information by performing the affine transformation process 23 using the determined value 25. Accordingly, the normalized feature data 26 can continue to hold the style information of the original feature data 21.

The specific method of generating the affine transformation parameter value 25 may vary depending on the embodiment. Hereinafter, a method of normalizing neural network data according to various embodiments of the present disclosure will be described based on the method of generating the affine transformation parameter value 25.

Each step of the methods described below may be performed by a computing device. In other words, each step of the methods may be implemented with one or more instructions executed by a processor of the computing device. All steps included in the methods may be performed by one physical computing device, but the first steps of the methods are performed by the first computing device and the second steps of the methods are performed on the second computing device. It may also be performed by. In the following, it is assumed that each step of the above methods is performed by the apparatus 10 illustrated in FIG. Thus, when the subject of a specific operation is omitted in the following description, it will be understood that the operation can be performed by the apparatus 10 illustrated above. In addition, the method to be described below may be changed in the order of performing each operation within a range that can be changed logically if necessary.

4 is an exemplary flowchart illustrating a neural network data normalization method according to some embodiments of the present disclosure. However, this is only for describing the preferred embodiment for achieving the purpose of the present disclosure, some steps may be added or deleted as necessary.

As shown in FIG. 4, the normalization method starts at step S100 of obtaining feature data for a training image from a first layer of a neural network. Some examples of neural networks that may be referenced in various embodiments of the present disclosure are shown in FIG. 4.

As shown in FIG. 4, the neural network 30 may include a plurality of layers e.g. 31 to 33, and at least some of the plurality of layers e.g. 31 to 33 may be normalization layers 32. Since the neural network 30 is a model for performing a task on an image, it may be conventionally implemented based on a convolutional neural network, but the technical scope of the present disclosure is not limited thereto. However, in order to provide convenience of understanding, hereinafter, the neural network 30 is assumed to be based on the convolutional neural network to continue the description.

As illustrated in FIG. 4, the normalization layer 32 may serve to provide feature data of the training image 34 output from the first layer 31 as an input of the second layer 33. That is, the present step S100 and the following steps S200 to S400 may be understood as strictly operations performed in the normalization layer 32. The location or number of normalization layers 32 may of course vary depending on the embodiment.

In addition, the normalization layer 32 may perform normalization using the normalization parameter 38. The normalization parameter 38 may include an affine transformation parameter for affine transformation and a style adjustment parameter for generating a style adjustment value. It may include. The affine transformation parameter may include at least one of a scale and a shift parameter. Here, the affine transformation parameter and the style adjustment parameter are both learnable parameters and may be understood as parameters learned together with the neural network 30. For example, when the error 36 between the correct answer for the training image 34 and the predicted value 35 is backpropagated, the value of the normalization parameter 38 may be updated along with the weights of the neural network (eg 37, 39). . The normalization parameter 38 will be described in more detail later with reference to FIGS. 7 to 12.

In some embodiments, the first layer 31 may be a convolutional layer and the second layer 33 may be an activation layer. Here, the convolution layer is a layer for extracting a feature map through a convolution operation, and the activation layer refers to a layer for performing nonlinear transformation on data input through an activation function. The activation function may include a sigmoid function, a rectified linear unit function, and the like, but the technical scope of the present disclosure is not limited thereto. In some other embodiments, the types of the first layer 31 and the second layer 33 may vary.

The feature data may indicate one or more feature maps extracted from a training image (or feature map) through a convolution operation. For example, as shown in FIG. 6, one or more feature maps 45 may be extracted from the training image 41 through a convolution filter of the convolution layer 43, wherein the feature map 45 may be extracted from the training image ( 41) may mean feature data. In this case, when C convolution filters are used, the feature map of the C channel may be extracted.

This will be described with reference to FIG. 4 again.

In step S200, style information is extracted from the feature data. The style information may include at least one of various types of statistical information such as average and standard deviation. Therefore, the style information may be extracted from the feature data through statistical calculations such as mean or standard deviation. When the feature data is a feature map having a plurality of channels (or depths) (that is, when a plurality of feature maps are extracted through a plurality of filters), style information may be extracted for each channel (ie, for each feature map). have.

In step S300, the feature data is normalized based on the style information of the feature data. The normalization may be performed through a batch normalization technique, and the style information may be utilized in the affine transformation process of batch normalization.

According to various embodiments of the present disclosure, a style adjustment value may be generated for each feature data based on the style information of each feature data. In addition, a value of the affine transformation parameter may be determined for each feature data using the style adjustment value, and the affine transformation process may be performed with the determined value. That is, the affine transformation is performed on the first feature data based on the first style adjustment value for the first feature data, and is applied to the second feature data based on the second style adjustment value for the second feature data. An affine transformation may be performed. In this case, the first feature data may be feature data associated with a first training image, and the second feature data may be feature data associated with a second training image.

The specific way of determining the value of the affine transformation parameter may vary depending on the embodiment.

In some embodiments, the style adjustment value may be reflected in the value of the global affine transformation parameter to determine the value of the final affine transformation parameter. Here, the global affine transformation parameter may mean an affine transformation parameter that is commonly applied to a plurality of feature data (or training images). According to the present embodiment, the affine transformation may be performed using a global affine transformation parameter that learns a common feature among training images and a style adjustment value reflecting an individual feature of each training image. That is, normalization may be performed considering both common and individual features. Therefore, learning can be easily performed for most tasks, and the performance of the neural network can be improved. This embodiment will be described in detail later with reference to FIGS. 7 to 10.

In some other embodiments, the style adjustment value may be determined as the value of the affine transformation parameter. That is, unlike the above-described embodiment, the value of the global affine transformation parameter may not be used. According to the present embodiment, the affine transformation can be performed by focusing more on individual features of the feature data. Therefore, lost style information can be accurately recovered through the normalization process, and the performance of neural networks (or tasks) sensitive to the style information can be greatly improved. This embodiment will be described in detail later with reference to FIG. 11.

In step S400, normalized feature data is provided as an input of a second layer of the neural network. In addition, the normalized feature data may be forwarded through the second layer and used to calculate a prediction error (e.g. 36 in FIG. 4) for a training image. The prediction error is back propagated to the neural network and used to update the weights of the neural network (e. G. 37, 39 of FIG. 4). In this case, the above-described normalization parameter (e. G. 38 of FIG. 4) may also be updated.

So far, the neural network data normalization method according to some embodiments of the present disclosure has been described with reference to FIGS. 4 to 6. Hereinafter, a style adjustment process according to various embodiments of the present disclosure will be described with reference to FIGS. 7 to 11.

7 is an exemplary flowchart for describing a style adjustment process according to some embodiments of the present disclosure.

As illustrated in FIG. 7, the style adjustment process (e.g. S340, 360) may be performed in the above-described normalization step S300. In addition, in the normalization step S300, a normalization process (e.g. S320) and an affine transformation process (e.g. S380) may be further performed. Hereinafter, the normalization step S300 will be described centering on the style adjustment process.

The normalization step S300 starts at step S320 for normalizing the feature data using statistical information of the batch to which the training image belongs. According to an embodiment, steps S320 and S340 / S360 may be performed simultaneously or in reverse order. It may be.

Since batch normalization will already be apparent to those skilled in the art, a detailed description of step S320 will be omitted.

In step S340, a style adjustment value of feature data is generated based on the style information. The style adjustment value may be understood as a value used to adjust the value of the affine transformation parameter to correct (or recover) distorted (or lost) style information in the normalization process. Therefore, the style adjustment value is generated based on the style information of the feature data for each feature data (or for each training image). A detailed example of generating the style adjustment value will be described later with reference to FIGS. 8 to 10.

In step S360, the value of the affine transformation parameter is adjusted to reflect the style adjustment value. Herein, the affine transformation parameter may include at least some of parameters for adjusting the scale and the shift. As described above, the affine transformation parameter may be understood to refer to a global affine parameter that is equally applied to a plurality of training images. However, the style adjustment value may be generated as a different value for each training image (or each feature data). In general, the feature data associated with each training image may include different style information. In this step, the value of the affine transformation parameter may be adjusted to a specialized value to correct (or recover) the style information of each feature data.

In step S380, affine transformation is performed with a value of the adjusted affine transformation parameter. Accordingly, the style information embedded in the feature data may be corrected (or restored).

For convenience of understanding, the detailed process of the normalization step S300 will be described in detail with reference to FIGS. 8 to 10.

8 to 10 are exemplary diagrams for further explaining the normalization process.

8 converts feature data 52-1 through 52-4 of the four training images into normalized feature data 55-1 through 55-4 through normalization process 53 and affine transformation process 54. It is illustrated. Of course, the number of training images and / or feature data may vary.

As shown in Fig. 8, a style adjustment value is generated from the style information of each feature data 52-1 to 52-4, and the style adjustment value is utilized in the affine transformation process 54.

9 illustrates that the style information of the first feature data 52-1 is corrected (or restored) in the affine transformation process 54 using the style adjustment value of the first feature data 52-1. It is an illustration. Of course, the contents described with reference to FIG. 9 may be equally applied to other feature data (e. G. 52-2 to 52-4).

As shown in Fig. 9, the style information 61 of the feature data 52-1 is extracted from the feature data 52-1. As described above, the style information 61 may include at least one of average and standard deviation.

If the feature data 52-1 is a feature map having a plurality of channels, the average and / or standard deviation is calculated for each channel, and the average and / or standard deviation of each channel is aggregated (eg concatenation) to form a vector. Information 61 may be generated. That is, if the feature data 52-1 is a feature map having C channels, the style information 61 may be calculated as a vector having a C dimension.

Next, the style adjustment value 63 of the feature data 52-1 is generated from the style information 61. The style adjustment value 63 of the feature data 52-1 may be generated by reflecting the value of the style adjustment parameter in the style information of the feature data 52-1. As described above, the style adjustment parameter may refer to a learnable parameter used to generate a style adjustment value. The specific way of generating the style adjustment value may vary depending on the embodiment.

In some embodiments, as shown in FIG. 10, the style adjustment value 75 may be generated by reflecting the value of the filter 73 in the style information 71. That is, the style adjustment value 75 may be generated by performing a predetermined operation between the value of the style adjustment parameter constituting the filter 73 and the style information 71. Here, the filter 73 may exist for each dimension (ie, channels of the feature map) of the style information 71, and one filter may exist for a plurality of dimensions (ie, multiple channels of the feature map). This may vary depending on the embodiment. According to the present embodiment, a style adjustment value suitable for style information of a training image (or feature data) may be generated through the learnable parameter. That is, as the neural network is learned, a style adjustment value suitable for correcting or restoring style information of corresponding feature data may be automatically generated.

In some other embodiments, the feature data is feature map data having a plurality of channels, style information is extracted for each channel, and aggregates the channel-specific style information through a fully-connected layer. Style adjustment values can be generated in a manner. That is, in this embodiment, the weight parameters of the fully connected layer operate as the style adjustment parameter. According to the present embodiment, an appropriate style adjustment value may be generated in consideration of channel dependencies.

In some other embodiments, style adjustment values may be generated through channel-wise transformation. That is, a style adjustment value for the first channel is generated by converting style information in a first channel through a first conversion parameter, and converts style information in a second channel through a second conversion parameter to the second channel. A style adjustment value for can be generated. In the present embodiment, the first conversion parameter and the second conversion parameter operate as style adjustment parameters. According to the present embodiment, the style adjustment value may be generated using a smaller number of conversion parameters than the fully connected layer. That is, the number of parameters learned can be reduced. Thus, the computing cost consumed for learning is reduced, and the style adjustment value can be calculated quickly.

Meanwhile, in some embodiments, the style adjustment parameter may exist for each training image (or feature data). That is, the first style adjustment parameter applied to the first training image and the second style adjustment parameter applied to the second training image may exist separately from each other.

This will be described again with reference to FIG. 9.

As shown in FIG. 9, the value of the affine transformation parameter 65 is adjusted to reflect the style adjustment value 63. 9 illustrates that the parameter adjustment value is adjusted by multiplying the style adjustment value 63 by the affine transformation parameter 65. However, the method of reflecting the style adjustment value 63 may vary. .

So far, the style adjustment process according to some embodiments of the present disclosure has been described with reference to FIGS. 8 to 10. Hereinafter, a style adjustment process according to some other embodiments of the present disclosure will be described with reference to FIG. 11.

As shown in FIG. 11, even in this embodiment, the style information 83 may be extracted from the feature data 81, and the style adjustment value 85 may be generated based on the style information 83. Since the method of generating the style adjustment value 85 is similar to the above-described embodiments (e.g. fully connected layer, channel unit conversion), description thereof will be omitted.

However, in the present embodiment, the style adjustment value 85 may be used as an affine transformation parameter of the feature data 81 as it is. That is, the style adjustment value 85 may be set to the affine transformation parameter value of the feature data 81, and the affine transformation process 87 may be performed with the set value.

So far, the neural network data normalization method according to some embodiments of the present disclosure has been described with reference to FIGS. 5 to 11. According to the above method, even when the batch size is set small, normalization can be performed while preserving the style information. Therefore, when training neural networks for tasks in which style information plays an important role, the performance of neural networks can be greatly improved.

Hereinafter, a neural network data normalization method according to some other embodiments of the present disclosure will be described with reference to FIG. 12.

12 is an exemplary diagram for describing a neural network data normalization method according to another exemplary embodiment of the present disclosure. It demonstrates with reference to FIG. For the clarity of the present disclosure, descriptions of overlapping contents with the foregoing embodiments will be omitted.

As shown in FIG. 12, in the present embodiment, style information is not extracted from the feature data 91, and a style adjustment value 95 may be generated through a convolution operation. For example, the first feature data 93 is extracted from the feature data 91 through a first convolution operation, and the second feature data (not shown) from the first feature data 93 through a second convolution operation. ) Can be extracted. In addition, the final feature data 95 extracted through the iterative convolution operation may be used as the style adjustment value. The convolution operation may be performed between the feature data (e.g. 91, 93) and the convolution filter, and the number of times the convolution operation is performed may vary according to embodiments. In this embodiment, it can be understood that the weight of the convolution filter operates as a style adjustment parameter.

12 illustrates that the style adjustment value 95 is used intact in the affine transformation process 97. However, in some other embodiments, affine transformation may be performed using the style adjustment value 95 together with the value of the global affine transformation parameter (see FIG. 9).

According to the present embodiment, an appropriate style adjustment value may be generated in consideration of both the global context and the channel dependency included in the feature data.

So far, the neural network data normalization method according to some other embodiments of the present disclosure has been described with reference to FIG. 12.

Meanwhile, various embodiments of the present disclosure have been described on the assumption that neural network normalization method (hereinafter, “style based normalization”) is performed regardless of the batch size. However, in some other embodiments of the present disclosure, the style based normalization may be performed in response to determining that the batch size is below a reference value. That is, if the batch size is greater than or equal to the reference value (i.e., large enough), then the affine transformation is performed using the unadjusted affine transform parameter value, and vice versa. Affine transformation can be performed. If the batch size is large enough, style information can be well preserved.

In some embodiments, the reference value may be determined based on a relationship between an object task of the neural network and style information. For example, if the target task has a close relationship with the style information (ie, the style information plays an important role in the task), the reference value is higher than a relatively higher value (ie, other tasks that do not have a close relationship). High value). By doing so, style information can be better preserved through style-based normalization even with large batch sizes. In the opposite case, the reference value can be determined with a relatively lower value. For example, if the target task of the neural network is a task that does not require style information such as style transfer, the reference value may be determined to be a very low value.

Further, in some embodiments, the batch size may be determined based on the relationship between the destination task of the neural network and the style information. For example, if the objective task has a close relationship with the style information (ie, the style information plays an important role for that task), the batch size is larger than the other larger values (ie, less closely related tasks). High value). In this case, if the batch size is already set, the set batch size may be adjusted according to the relationship between the target task and the style information. In the opposite case, the batch size can be determined with a relatively smaller value.

Hereinafter, an exemplary computing device 100 that may implement an apparatus (e.g. learning device 10 of FIG. 1) according to various embodiments of the present disclosure will be described with reference to FIG. 13.

13 is an exemplary hardware diagram illustrating the computing device 100.

As shown in FIG. 13, the computing device 100 may include one or more processors 110, a bus 150, a communication interface 170, and a memory that loads a computer program executed by the processor 110. 130 and a storage 190 for storing the computer program 191. However, FIG. 13 illustrates only the components related to the embodiment of the present disclosure. Accordingly, it will be appreciated by those skilled in the art that the present disclosure may further include other general purpose components in addition to the components shown in FIG. 13. That is, the computing device 100 may further include various components in addition to the components illustrated in FIG. 13.

The processor 110 controls the overall operation of each component of the computing device 100. The processor 110 may include at least one of a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphics processing unit (GPU), or any type of processor well known in the art. It can be configured to include. In addition, the processor 110 may perform an operation on at least one application or program for executing a method / operation according to embodiments of the present disclosure. Computing device 100 may have one or more processors.

The memory 130 stores various data, commands, and / or information. Memory 130 may load one or more programs 191 from storage 190 to execute methods / operations in accordance with various embodiments of the present disclosure. The memory 130 may be implemented as a volatile memory such as a RAM, but the technical scope of the present disclosure is not limited thereto.

The bus 150 provides a communication function between components of the computing device 100. The bus 150 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The communication interface 170 supports wired and wireless Internet communication of the computing device 100. In addition, the communication interface 170 may support various communication methods other than Internet communication. To this end, the communication interface 170 may comprise a communication module well known in the art of the present disclosure. In some cases, the communication interface 170 may be omitted.

The storage 190 may non-temporarily store the one or more programs 191. Storage 190 is well known in the art, such as non-volatile memory, hard disks, removable disks, or the like to which the present disclosure belongs, such as Read Only Memory (ROM), Eraseable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, and the like. It may comprise any known type of computer readable recording medium.

Computer program 191 may include one or more instructions that, when loaded into memory 130, cause processor 110 to perform a method / operation in accordance with various embodiments of the present disclosure. That is, the processor 110 may perform methods according to various embodiments of the present disclosure by executing the one or more instructions.

For example, the computer program 191 obtains feature data for a training image output from a first layer of a neural network, extracts style information of the feature data, and normalizes the feature data based on the style information. And providing the normalized characteristic data as an input of the second layer of the neural network. In this case, the neural network learning device (e.g. 10) or the neural network data normalization device (e.g. 10) according to some embodiments of the present disclosure may be implemented through the computing device 100.

In another example, the computer program 191 obtains first feature data for a first training image output from a specific layer of a neural network, and obtains second feature data for a second training image output from the specific layer. Operation, generating a first style adjustment value for the first feature data based on the first feature data, generating a second style adjustment value for the second feature data based on the second feature data. Instructions for performing an operation, normalizing the first feature data using the generated first style adjustment value, and normalizing the second feature data using the generated second style adjustment value. can do. In this case, the neural network learning device (e.g. 10) or the neural network data normalization device (e.g. 10) according to some other embodiments of the present disclosure may be implemented through the computing device 100.

So far, various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 to 13. Effects according to the spirit of the present disclosure are not limited to the above-mentioned effects, other effects that are not mentioned will be clearly understood by those skilled in the art from the following description.

The technical idea of the present disclosure described with reference to FIGS. 1 to 13 may be implemented as computer readable codes on a computer readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer equipped hard disk). Can be. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device and installed in the other computing device through a network such as the Internet, thereby being used in the other computing device.

In the above description, it is described that all the elements constituting the embodiments of the present disclosure are combined or operated as one, but the technical spirit of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the present disclosure, all of the components may be selectively operated in one or more combinations.

Although the operations are shown in a specific order in the drawings, it should not be understood that the operations must be performed in the specific order or sequential order shown or that all the illustrated operations must be executed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various configurations in the embodiments described above should not be understood as such separation being necessary, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. Should be understood.

While the embodiments of the present disclosure have been described with reference to the accompanying drawings, a person of ordinary skill in the art may implement the present disclosure in other specific forms without changing the technical spirit or essential features. I can understand that there is. Therefore, it is to be understood that the embodiments described above are exemplary in all respects and not restrictive. The protection scope of the present disclosure should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto shall be interpreted as being included in the scope of the technical idea defined by the present disclosure.

Claims (20)

In a method for normalizing data of neural networks in a computing device,
Obtaining feature data for a specific training image output from a specific layer of the neural network;
Extracting style information of the feature data using statistical information of the feature data;
Generating a style adjustment value for the feature data using the style information;
Adjusting a value of an affine transformation parameter by reflecting the style adjustment value;
Generating a normalization value obtained by normalizing the feature data with statistical information of a batch;
Performing an affine transformation on the normalized value using a value of the adjusted affine transformation parameter: and
Outputting the affine transformed values as normalized feature data of the particular training image,
Neural network data normalization method. delete According to claim 1,
The style information includes at least one of an average and a standard deviation of the feature data;
Neural network data normalization method. According to claim 1,
The feature data is feature map data having a plurality of channels,
Extracting the style information,
Extracting the style information for each channel of the feature map data;
Neural network data normalization method. According to claim 1,
Generating the style adjustment value,
Generating the style adjustment value by reflecting a value of a style adjustment parameter updated with the weight of the neural network in the style information.
Neural network data normalization method. According to claim 1,
The feature data includes first feature map data of a first channel and second feature map data of a second channel,
The style information includes first style information about the first feature map data and second style information about the second feature map data.
Generating the style adjustment value,
Generating the style adjustment value by aggregating the first style information and the second style information through a fully-connected layer;
Neural network data normalization method. According to claim 1,
The feature data includes first feature map data of a first channel and second feature map data of a second channel,
The style information includes first style information about the first feature map data and second style information about the second feature map data.
Generating the style adjustment value,
Generating a style adjustment value corresponding to the first channel using the first style information; And
Generating a style adjustment value corresponding to the second channel by using the second style information;
Neural network data normalization method. delete delete According to claim 1,
Generating the style adjustment value,
Generating the style adjustment value through a convolution operation between a feature data output from a specific layer of the neural network and a convolution filter updated with weights of the neural network;
Neural network data normalization method. According to claim 1,
The specific training image is an image in which the size of the batch is less than a reference value,
The feature data of the training image whose size of the batch is greater than or equal to the reference value is normalized based on statistical information of the batch,
Neural network data normalization method. A memory for storing one or more instructions; And
A processor for executing the stored one or more instructions,
The processor is
Obtaining feature data for a specific training image output from a specific layer of the neural network;
Extracting style information of the feature data using statistical information of the feature data;
Generating a style adjustment value for the feature data using the style information;
Adjusting a value of an affine transformation parameter by reflecting the style adjustment value;
Generating a normalization value obtained by normalizing the feature data with statistical information of a batch;
Performing an affine transformation on the normalized value using a value of the adjusted affine transformation parameter: and
Outputting the affine transformed values as normalized feature data of the particular training image;
Neural Network Data Normalization Device. delete The method of claim 12,
The processor,
Generating the style adjustment value by reflecting a value of a style adjustment parameter updated with the weight of the neural network in the style information;
Neural Network Data Normalization Device. The method of claim 12,
The feature data includes first feature map data of a first channel and second feature map data of a second channel,
The style information includes first style information about the first feature map data and second style information about the second feature map data.
The processor,
Generating the style adjustment value by aggregating the first style information and the second style information through a fully-connected layer;
Neural Network Data Normalization Device. The method of claim 12,
The feature data includes first feature map data of a first channel and second feature map data of a second channel,
The style information includes first style information about the first feature map data and second style information about the second feature map data.
The processor,
Generating a style adjustment value corresponding to the first channel by using the first style information,
Generating a style adjustment value corresponding to the second channel by using the second style information;
Neural Network Data Normalization Device. delete delete The method of claim 12,
The processor,
Generating the style adjustment value through a convolution operation between a feature data output from a specific layer of the neural network and a convolution filter updated together with the weight of the neural network;
Neural Network Data Normalization Device. A computer program stored in a computer-readable recording medium,
Obtaining feature data for a specific training image output from a specific layer of the neural network;
Extracting style information of the feature data using statistical information of the feature data;
Generating a style adjustment value for the feature data using the style information;
Adjusting a value of an affine transformation parameter by reflecting the style adjustment value;
Generating a normalization value obtained by normalizing the feature data with statistical information of a batch;
Performing an affine transformation on the normalized value using a value of the adjusted affine transformation parameter: and
And instructions for executing the step of outputting the affine transformed value as normalized feature data of the particular training image.

Images