KR102243125B1 - Method and apparatus for classifying complex characteristics of images using neural network model - Google Patents

Abstract

Provided are a method and an apparatus for classifying complex features by using a neural network model that can accurately predict complex features. According to an embodiment of the present invention, the method for classifying the complex features of an image by using a neural network model comprises: (a) a convolution step of performing a convolution operation on an input image by using a convolution layer; (b) a pooling step of performing pooling by using a pooling layer on the output of the convolution step; and (c) a class-specific complete combining step of outputting a value obtained by multiplying the output of the pooling step by a class weight (w_(fc)(T_t)) by using a fully connected layers for each of a plurality of classes corresponding to each of the plurality of classes into which complex features are classified.

Description

Method and apparatus for classifying image complex characteristics using neural network model {METHOD AND APPARATUS FOR CLASSIFYING COMPLEX CHARACTERISTICS OF IMAGES USING NEURAL NETWORK MODEL}

The present invention relates to a method and apparatus for classifying complex characteristics of an image using a neural network model.

1A shows a neural network model according to the prior art.

Referring to (a) of FIG. 1, the neural network model includes an input layer, one or more convolutional layers, a pooling layer, and a fully coupled layer. Using such a neural network model, when there is any input, a result value of one of the four characteristics of A, B, C, and D, or a probability corresponding to each characteristic can be output. In this case, the total sum of the result values may be 100%.

However, some inputs can have complex properties that allow multiple outcomes. For example, when A is a male, B is a female characteristic, C is an Asian, and D is a non-Asian characteristic, A and C may be simultaneously possible as a result value.

However, in the case of the conventional neural network model, since the total sum of the result values is determined to be 100%, it is difficult to properly reflect such complex characteristics.

Patent Registration Publication No. 10-2095335 (2020.03.25)

An embodiment of the present invention is to provide a method and apparatus for classifying complex characteristics using a neural network model capable of properly predicting complex characteristics.

According to an embodiment of the present invention, a method for classifying complex characteristics of an image using a neural network model in which each step is performed by a processor includes: (a) a convolution operation for performing a convolution operation on an input image using a convolution layer step; (b) a pooling step of performing pooling using a pooling layer on the output of the convolution step; _{And (c) a weight for each class (w fc} (T _t )) for the output of the pooling step by using a fully connected layer for each class corresponding to each of a plurality of classes in which the complex characteristics are classified. It includes a complete combining step for each class that outputs the multiplied value of.

The method includes the steps of: (d) calculating observation maps for each class based on the output of the convolution step; And (e) generating a common observation map common to the plurality of fully coupled layers based on the observation maps for each of the plurality of classes.

The common observation map may be an average value of the observation maps for each class.

In the step (d), the observation map for each class may be calculated by the following equation.

(However, T _t is the class, w _fc (T _t ) is the weight of the fully coupled layer for each class, o _conv is the output of step (a), and C is the number of channels)

The method may further include (f) calculating a characteristic probability for each class according to the output of the fully coupled layer for each class by using a plurality of class-specific classifiers respectively corresponding to the plurality of fully coupled layers for each class. have.

The pooling layer may be a global average pooling layer.

An apparatus for classifying a complex characteristic of an image according to an embodiment of the present invention includes: a memory storing a neural network model; And a processor, wherein the neural network model includes a convolutional layer, a pooling layer for performing pooling on the output of the convolutional layer, and a plurality of classes each corresponding to a plurality of classes into which complex characteristics are classified. A combination layer is included, and the processor performs a convolution operation on an input image using the convolution layer, performs pooling on a result of the convolution operation performed using the pooling layer, and the The output of the pooling step is multiplied by the weight of each class (w _fc (T _t )).

The processor may calculate an observation map for each of a plurality of classes based on the output of the convolutional layer, and generate a common observation map common to the completely combined layer for each of the plurality of classes based on the observation maps for each of the plurality of classes. I can.

According to an embodiment of the present invention, it is possible to properly classify complex characteristics.

According to an embodiment of the present invention, the association between classes may be reflected.

According to an embodiment of the present invention, common localization between complex characteristics can be achieved.

Figure 1 (a) shows a one-to-one model according to the prior art, Figure 1 (b) shows the use of two one-to-one classification models according to a comparative example, Figure 1 (c) is an implementation of the present invention. It shows a one-to-many model according to an example.
2 is a diagram showing the structure of a neural network model 1 according to an embodiment of the present invention.
3 is a flowchart illustrating an inference method of a neural network model according to an embodiment of the present invention.
FIG. 4 is a diagram for describing a step of generating the common observation map of FIG. 3.
FIG. 5 shows parts corresponding to each step of FIG. 3 in the neural network model of FIG. 2.
6 is a diagram showing the configuration of a complex characteristic classification apparatus according to an embodiment of the present invention.

Based on the principle that the inventor can appropriately define the concept of terms in order to describe his own invention in the best way, terms or words used in the present specification and claims are consistent with the technical idea of the present invention. It should be interpreted as meaning and concept.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. In addition, when one component is said to be "connected", "transmitted", "transmitted", "received" or "transmitted" to another component, it is not only the case that is directly connected, transmitted, transmitted, received or transmitted, but also Indirect connection, transmission, transmission, and reception through components are also included. In addition, terms such as "...unit", "...group", "module", and "device" described in the specification mean units that process at least one function or operation, which is hardware or software, or a combination of hardware and software. It can be implemented as

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

FIG. 1(b) shows the use of two one-to-one classification models according to a comparative example, and FIG. 1(c) shows a one-to-many model according to an embodiment of the present invention.

In FIG. 1, A and B denote characteristics belonging to class I, and C and D denote characteristics belonging to class II. The classification criteria for Class I and Class II may be different. In this specification, characteristics belonging to a plurality of classes are referred to as complex characteristics. Inputs with complex characteristics may have a plurality of characteristics, that is, characteristics for each class.

Referring to FIG. 1B, when two one-to-one classification models are used, the sum of the prediction probabilities of the features A and B is 100%, and the sum of the prediction probabilities of the features C and D is 100%. In this case, the prediction result may be accurate, but since two models must be trained, the amount of computation may increase. Also, since the two models are trained independently, it is difficult to reflect the relationship between classes.

Referring to (c) of FIG. 1, in the one-to-many model, some layers, that is, a convolutional layer and a pooling layer, are shared between classes, and a fully coupled layer is provided for each class. At this time, since the fully coupled layer is provided for each class, the sum of the predicted probabilities of the features A and B is 100%, and the sum of the predicted probabilities of the features C and D is 100%. Therefore, since the prediction result in class I and the prediction result in class II are provided respectively, it is possible to obtain an accurate prediction result compared to FIG. 1A. In addition, since the convolution layer is shared between classes, the correlation between classes can be reflected while reducing the amount of computation compared to the model of FIG. 1B.

2 is a diagram showing the structure of a neural network model 1 according to an embodiment of the present invention. In FIG. 2, there are N _T classes (T ₁ , T ₂ , ... T _t , ... T _NT ), _{two characteristics belong to class T 1} , and two characteristics belong to class T ₂ It is assumed that this belongs, n kinds of characteristics belong to class T _t , and three characteristics belong to class T _NT.

Referring to FIG. 2, a neural network model 1 according to an embodiment of the present invention includes a convolution layer 10 for performing a convolution operation on an _{input image x i, and the convolution layer 10.} Each corresponding to a pooling layer 20 for performing pooling on the output of (o _conv ) and a plurality of classes into which complex characteristics are classified, and a weight for each class (w _fc ( It includes a plurality of fully coupled layers 30 for each class outputting a value multiplied by _{T t )).}

The neural network model (1) corresponds to each of the plurality of fully coupled layers 30 for each class and calculates a characteristic probability for each class according to the output of the fully coupled layer 30 for each class. It may further include.

The convolution layer 10 extracts a feature map by performing a convolution operation on an input image using a plurality of convolution filters. As shown in FIG. 2, the convolution operation may be performed a plurality of times. It is assumed that the output of the convolutional layer 10, that is, the feature map (o _conv ), has a height (H), a width (W), and the number of channels (C).

The pooling layer 20 is located between the convolutional layer 10 and the fully coupled layer 30, _{and reduces the size of the feature map (o conv} ) to reduce the operation required in the fully coupled layer 30 to be described later. , It serves to prevent overfitting. The pooling layer 20 may perform global average pooling in which an average value is output for each channel of the feature map (o _{conv ).}

The class-specific fully coupled layer 30 has weights for each class (w _fc (T ₁ ), w _fc (T ₂ ), ... w _fc (T _t ), ... w _{fc) on the output of the pooling layer 20} (T _NT )) multiplied by it. At this time, each of the weights for each class (w _fc (T ₁ ), w _fc (T ₂ ), ... w _fc (T _t ), ... w _fc (T _NT )) is a plurality of Can be a value.

The classifiers 40 for each class correspond to each of the fully coupled layers 30 for each class, and calculate a characteristic probability for each class according to the output of the fully coupled layer 30 for each class. 2, the classifier for the class T ₁ corresponds to the class T probability corresponding to the characteristic of belonging to the _{1 (P 1 (T 1)} , P 2 (T 1)) computing a, and class T ₂ classifier to classifier for the class T ₂ probability _{(P 1 (T 2),} P 2 (T 2)) computing a, and class T _t for the characteristics belonging to the each of the characteristics belongs to the class T _t probability that corresponds to the sorter for calculating the _{(P 1 (t t),} P 2 (t t), ... P n (t t)) and corresponds to the class t _NT is each of the attributes belonging to the class t _NT Calculate the corresponding probabilities (P ₁ (T _NT ), P ₂ (T _NT ), P ₃ (T _NT )). As the class-specific classifier 40, for example, a Softmax function, a Sigmoid function, or the like may be used.

Next, a method of classifying a complex feature using a neural network model according to an embodiment of the present invention will be described with reference to FIGS. 3 to 5.

3 is a flowchart illustrating a method for classifying a complex characteristic according to an embodiment of the present invention, FIG. 4 is a diagram for explaining the step of generating the common observation map of FIG. 3 (S150), and FIG. 5 is a neural network model of FIG. In Fig. 3, the corresponding part of each step is indicated. The neural network model of FIG. 2 is used as a learning method of a neural network model according to an embodiment of the present invention, and each step is performed by a processor.

Referring to FIGS. 3 and 5A, first _{, a convolution operation is performed on an input image x i} using the convolution layer 10 (S100).

Next, referring to FIGS. 3 and 5B, pooling is performed using the pooling layer 20 on _{the output (o conv) of the convolutional layer 10 (S110).}

Next, referring to FIGS. 3 and 5C, weights for each class for the output of the pooling layer 20 by using a plurality of class-specific fully coupled layers 30 corresponding to each of a plurality of classes into which the complex characteristics are classified. A value multiplied by (w _fc (T _t )) is output (S120).

Next, referring to FIG. 3 and FIG. 5D, using a plurality of class-specific classifiers 40 respectively corresponding to a plurality of class-specific fully-coupled layers, according to the output of each class-specific fully-coupled layer, the characteristic probability (P) _n (T _t )) is calculated (S130). Accordingly, a characteristic probability P _n (T _t ) for each class may be provided.

Meanwhile, referring to FIGS. 3, 4 and 5E, observation maps for each class are calculated based on _{the output (o conv) of the convolution step (S100) (S140).}

The observation map is also referred to as a Class Activation Map (CAM), and indicates how much each part of the input image has influenced the classification result. In this embodiment, the observation map is calculated for each class. 4 shows an example of an observation map for each class. 4 shows a case where the input image is an ultrasound image.

The observation map for each class may be calculated based on the output of the convolution step S100 (o _conv ), the weight of the fully coupled layer for each class (w _fc (T _t )), and a characteristic probability for each class. For example, the observation map for each class can be calculated by the following equation.

However, T _t is the class, w _fc (T _t ) is the weight of the fully coupled layer for each class, o _conv is the output of the convolution layer, and C is the number of channels. When characteristic A and characteristic B belong to the same class, when characteristic A is predicted, the weight of the fully coupled layer for each class belonging to characteristic A is multiplied, and when characteristic B is predicted, the fully coupled layer for each class belonging to characteristic B The weight of is multiplied.

)를 생성한다(S150). 이에 따라 복수의 클래스에 공통되는 국소화 영역이 제공된다.Next, referring to FIGS. 3, 4, and 5E, a common observation map common to a plurality of fully coupled layers for each class based on a plurality of observation maps for each class (

) Is generated (S150). Accordingly, a localized area common to a plurality of classes is provided.

The common observation map may be an average value of the observation map for each class, and may be calculated by the following equation.

는 전술한 클래스별 관찰 지도이고, N_T는 클래스의 수를 나타낸다.here,

Is the above-described observation map for each class, and N _T represents the number of classes.

However, this is only an example, and a common observation map may be calculated based on the observation map of some of the classes of each class or different ratios of the observation map of each class.

6 is a diagram illustrating a configuration of an apparatus 1000 for classifying a complex characteristic of an image according to an exemplary embodiment of the present invention.

The complex characteristic classification apparatus device 1000 includes a memory 1100 and a processor 1200 in which a neural network model is stored.

The neural network model stored in the memory 1100 has already been described with reference to FIG. 2.

The processor 1200 performs the complex characteristic classification method described with reference to FIGS. 3 to 5. That is, the processor 1200 performs a convolution operation on the input image using the convolution layer, performs pooling on the result of the convolution operation using the pooling layer, and classifies the output of the pooling step. Outputs the multiplied by the star weight (w _fc (T _{t )).}

According to an embodiment, the processor calculates an observation map for each of a plurality of classes based on the output of the convolutional layer, and generates a common observation map common to the fully coupled layers for each of a plurality of classes based on the observation maps for each of the plurality of classes. can do.

As described above, the present invention has been described in detail through preferred embodiments, but the present invention is not limited thereto, and various changes and applications can be made without departing from the technical spirit of the present invention. It is self-explanatory to the technician. Therefore, the true scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (8)

(a) a convolution step of performing a convolution operation on an input image having a complex characteristic including a plurality of single characteristics simultaneously using a convolution layer;
(b) a pooling step of performing pooling using a pooling layer on the output of the convolution step; And
(c) Using a fully connected layer for each class corresponding to each of a plurality of classes in which the complex characteristics are classified, a weight for each class (w _fc (T _t )) is calculated for the output of the pooling step. A complete combining step for each class outputting the multiplied value;
Including,
The plurality of classes respectively correspond to the plurality of single characteristics distinguished by different criteria,
Each of the plurality of classes is classified into characteristics of a plurality of classes,
A method for classifying composite features of an image using a neural network model in which each step is performed by a processor, capable of providing class-specific feature probabilities for class-specific features of each of the plurality of classes, according to the output of the complete combining step for each class .
The method of claim 1,
(d) calculating observation maps for each of a plurality of classes based on the output of the convolution step; And
(e) generating a common observation map common to the completely combined layers for each of the plurality of classes based on the observation maps for each of the plurality of classes.
A method for classifying complex characteristics of an image comprising a.
The method of claim 2,
The common observation map is an average value of the observation maps for each class.
The method of claim 2,
In step (d), the class-specific observation map is calculated by the following equation.

(However, T _t is the class, w _fc (T _t ) is the weight of the fully coupled layer for each class, o _conv is the output of step (a), and C is the number of channels)
The method of claim 1,
(f) calculating a characteristic probability for each class according to the output of the fully coupled layer for each class using a plurality of class-specific classifiers respectively corresponding to the plurality of fully coupled layers for each class.
The method for classifying complex characteristics of an image, characterized in that it further comprises.
The method of claim 1,
And the pooling layer is a global average pooling layer.
A memory in which the neural network model is stored; And
Processor
An apparatus for classifying a complex characteristic of an image comprising a,
The neural network model includes a convolutional layer, a pooling layer for performing pooling on the output of the convolutional layer, and a plurality of classes each corresponding to a plurality of classes into which complex characteristics including a plurality of single characteristics are classified. Including a star fully bonded layer,
The processor performs a convolution operation on the input image having the complex characteristic using the convolution layer, performs pooling on the result of the convolution operation performed using the pooling layer, and performs pooling. Outputs the value obtained by multiplying the result by class weight (w _fc (T _{t )),}
The plurality of classes respectively correspond to the plurality of single characteristics distinguished by different criteria,
Each of the plurality of classes is classified into characteristics of a plurality of classes,
An apparatus for classifying a composite feature of an image capable of providing class-specific feature probabilities for class-specific features of each of the plurality of classes according to the output of the class-specific complete coupling layer.
The method of claim 7,
The processor is configured to calculate a plurality of observation maps for each class based on the output of the convolution layer, and generate a common observation map common to the completely combined layers for each of the plurality of classes based on the observation maps for each class. An apparatus for classifying complex characteristics of an image, characterized in that.

Images