KR102223116B1 - Image analysis method and apparatus - Google Patents

Abstract

A computing device extracts a first feature map from an input image based on a neural network, generates a plurality of attention maps from the first feature map based on the neural network, and generates context information based on the plurality of attention maps. Also, the computing device generates a second feature map by combining the context information with the first feature map, and outputs a reading result of the input image from the second feature map based on the neural network. The present invention provides an image reading method and device capable of considering other areas.

Description

Image reading method and apparatus {IMAGE ANALYSIS METHOD AND APPARATUS}

The present invention relates to an image reading method and apparatus.

Recently, machine learning models using neural networks have been applied to the field of medical image reading, thereby increasing the accuracy and speed of image reading. In particular, a reading system has been developed that marks areas with abnormal findings such as lung nodules through X-ray image analysis and presents the possibility as an index.

However, when a doctor determines an abnormal region, it is possible to find an area that is meaningful when comparing not only the target area but also the target area, and considers the difference between the two areas to determine the abnormality of the target area. For example, when determining a pulmonary nodule, it is possible to determine whether it is an actual pulmonary nodule by comparing an area in one lung that is estimated to be a pulmonary nodule with a similar area in the other lung. However, the machine learning model can only determine whether a pulmonary nodule is a pulmonary nodule with only the characteristic information of a region estimated to be a pulmonary nodule, and does not consider other regions.

In a neural network, the receptive region can be widened to consider other regions outside the receptive field, but when the receptive region is increased by enlarging the machine learning model, a large amount of computing resources are required. Inference time can be increased. In addition, when the receiving area is increased through a pooling operation, high resolution texture information important for inference is lost, and inference performance may be degraded.

"CBAM: Convolutional Block Attention Module", Sanghyun Woo, Jongchan Park, Joon-Young Lee, In So Kweon, The European Conference on Computer Vision (ECCV), 2018

An object of the present invention is to provide an image reading method and apparatus that can consider other areas.

According to an embodiment of the present invention, a method of reading an image performed by a computing device is provided. The computing device extracts a first feature map from an input image based on a neural network, generates a plurality of attention maps from the first feature map based on the neural network, and based on the plurality of attention maps. Then, context information is generated, a second feature map is generated by combining the context information with the first feature map, and a reading result of the input image is output from the second feature map based on the neural network. .

The computing device may generate a plurality of vectors based on the plurality of attention maps and the first feature map, and may generate the context information by comparing the plurality of vectors.

The computing device generates the plurality of attention maps by applying a 1×1 convolution operation having different parameters to the first feature map, respectively, and weighted averages the first feature map based on each attention map. Each vector can be created.

The computing device may normalize each attention map before weighting the first feature map.

The 1×1 convolution operation may be a group convolution operation performed after grouping a plurality of channels of the first feature map into a predetermined number of channel groups.

The computing device may update the neural network by calculating a loss of the context information based on the plurality of vectors and backpropagating the loss to the neural network.

The computing device may calculate the loss based on an orthogonal loss of the plurality of vectors.

The computing device may normalize the first feature map before generating the plurality of attendance maps.

The computing device may normalize the first feature map by subtracting the C-dimensional average vector of the first feature map from the first feature map. Here, C indicates the number of channels of the first feature map.

The computing device may generate a channel feature vector for reducing the size of a specific channel among C channels based on the average vector, and combine the channel feature vector to a feature map obtained by combining the context information with the first feature map. have.

The computing device may generate the channel feature vector by applying a 1×1 convolution operation and an activation function to the average vector.

The 1×1 convolution operation may be a group convolution operation performed after grouping the C channels of the average vector into a predetermined number of channel groups.

When the first feature map has a C×H×W dimension, the context information may be a C×1×1 dimension. Here, C, H, and W indicate the number of channels, height, and width of the first feature map, respectively.

The computing device generates a plurality of second attention maps from the second feature map based on the neural network, generates second context information based on the plurality of second attention maps, and stores the second context information. A third feature map may be generated by combining with the second feature map, and a read result of the input image may be output from the third feature map based on the neural network.

The computing device calculates a loss of the context information, calculates a loss of the second context information, calculates a context loss based on the loss of the context information and the loss of the second context information, and the context loss The neural network can be updated by backpropagating to the neural network.

According to another embodiment of the present invention, an image reading apparatus including a processor and a memory for storing instructions is provided. The processor extracts a first feature map from an input image based on a neural network by executing the command, and a first vector representing a region of interest from the first feature map based on the neural network and a first vector corresponding to the region of interest Generate a second vector representing a region, generate context information based on the first vector and the second vector, combine the context information with the first feature map to generate a second feature map, and the neural network On the basis of, the reading result of the input image is output from the second feature map.

The processor may generate a plurality of attention maps from the first feature map based on the neural network, and may generate the first vector and the second vector based on the plurality of attention maps.

According to another embodiment of the present invention, a computer program executed by a computing device and stored in a recording medium is provided. The computer program includes, by the computing device, extracting a first feature map from an input image based on a neural network, a first vector representing a region of interest from the first feature map based on the neural network and a corresponding region of interest Generating a second vector representing an area, generating context information based on the first vector and the second vector, and generating a second feature map by combining the context information with the first feature map And, based on the neural network, a step of outputting the reading result of the input image from the second feature map is performed.

1 is a diagram illustrating a learning device and a learning environment according to an embodiment of the present invention.
2 is a diagram illustrating a neural network of a learning device according to an embodiment of the present invention.
3 is a flowchart illustrating an image reading method according to an embodiment of the present invention.
4, 5, and 6 are diagrams illustrating neural networks according to various embodiments of the present invention.
7 is a flowchart illustrating a method of generating context information according to an embodiment of the present invention.
8, 9, and 10 are diagrams illustrating a context module according to various embodiments of the present invention.
11 is a flowchart illustrating a learning method according to an embodiment of the present invention
12 is a diagram illustrating a learning method according to an embodiment of the present invention.
13 is a diagram illustrating a context module according to another embodiment of the present invention.
14 is a diagram illustrating a neural network according to another embodiment of the present invention.
15 is a diagram illustrating a neural network according to another embodiment of the present invention.
16 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Expressions described in the singular in the following description may be interpreted as the singular or plural unless an explicit expression such as "one" or "single" is used.

In the following description, terms including ordinal numbers such as first and second may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

In the flowchart described with reference to the drawings, the order of operations may be changed, several operations may be merged, certain operations may be divided, and specific operations may not be performed.

In the following description, a target model is a model that performs a task and may mean a model to be built through machine learning. Since the objective model can be implemented based on any machine learning model including a neural network, the present invention is not limited by the implementation method of the objective model.

Also, a neural network is a term that encompasses all kinds of machine learning models designed by mimicking neural structures. For example, a neural network may include all kinds of neural network-based models, such as an artificial neural network (ANN) or a convolutional neural network (CNN).

Next, an image reading method and apparatus according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a learning device and a learning environment according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating a neural network of a learning device according to an embodiment of the present invention.

Referring to FIG. 1, the learning apparatus 100 for reading an image is a computing device that performs machine learning on a neural network in order to perform a target task. In some embodiments, the objective task may include a task for image diagnosis, and may include, for example, a task associated with visual recognition. In one embodiment, the target task may include a task of detecting a lesion from an image.

In some embodiments, the computing device may be a notebook, a desktop, a laptop, a server, etc., but is not limited thereto and may be any type of device equipped with a computing function. An example of such a computing device will be described with reference to FIG. 16.

1 illustrates an example in which the learning device 100 is implemented as one computing device, the function of the learning device 100 in an actual physical environment may be implemented by a plurality of computing devices.

The learning apparatus 100 may learn a neural network (eg, a convolutional neural network) using the data set 110 including a plurality of training samples. Each training sample may include an image 111 in which the correct answer 111a is labeled. In some embodiments, the training device 100 inputs the training sample 111 into a neural network and backpropagates the loss between the predicted value and the correct answer 111a labeled on the training sample to the neural network to perform the target task. You can learn. The learning device 100 may predict a result (eg, a lesion) 130 by inputting the image 120 to the learned neural network and performing a target task. That is, the learning apparatus 100 may read an image by performing a target task based on the learned neural network.

In some embodiments, the neural network may include a plurality of layers 210 as shown in FIG. 2. In some embodiments, the plurality of layers 210 may include a convolutional layer.

In some embodiments, the first layer 210 of the plurality of layers 210 may receive an input image and extract a feature from the input image. These features may be expressed in the form of a feature map. The other layer 210 may extract features from the features transmitted by the previous layer 210 and transfer them to the next layer 210. In some embodiments, the layer 210 may extract features by applying an operation of a corresponding layer to an input image or input feature. In some embodiments, the convolutional layer 210 may extract features by applying a convolution filter to an input image or input feature. The convolution filter may extract a feature by performing a convolution operation on an input image or a receptive of the input feature.

In some embodiments, the plurality of layers 210 may further include other layers such as a pooling layer and an activation layer. The pooling layer may extract features by performing a pooling operation on the input features. The activation layer is a layer that performs nonlinear transformation on data input through an activation function, and may include a sigmoid function, a ReLU (Rectified Linear Unit) function, and the like.

In some embodiments, the plurality of layers 210 may further include a fully connected layer that performs classification from features output through the various layers 210. In some embodiments, a plurality of fully connected layers may be provided, and the fully connected layers may provide a probability for each label.

When each of the plurality of layers 210 is expressed as a function, the neural network may be expressed as a composite function of _{a plurality of functions (f 1} , f ₂ , …, f _{N) corresponding to the plurality of layers as shown in Equation 1.} .

In Equation 1, X is an input image, Y is a final output of the plurality of layers 210, that is, a task output according to a target task, and is a feature extracted from a layer corresponding to _{each function (f n ()) (} n is an integer from 1 to (N-1)).

3 is a flowchart illustrating an image reading method according to an embodiment of the present invention.

Referring to FIG. 3, the computing device extracts a feature map from the input image by inputting an input image into a neural network (S310). In some embodiments, the computing device may extract a feature map from the input image through at least one layer.

The computing device extracts context information for accurate reading of the input image from the feature map based on the neural network (S320), and combines the context information with the feature map (S330). In this case, since the context information is a result of comparing features representing an important region in the feature map, by combining the context information with the feature map, the comparison result of the important region in the feature map can be better identified. Accordingly, it is possible to compare the characteristics of the region of interest and the corresponding region without increasing the receiving region, thereby preventing an increase in computing resources, an increase in memory capacity, and an increase in inference time. It can also prevent loss of high resolution texture information, which is important for inference.

The computing device reads the image based on the feature map combined in the neural network (S340). In some embodiments, the computing device may extract the feature map again from the feature map combined through at least one layer, and read the image based on the extracted feature map. In some embodiments, the computing device may extract context information again from the feature map extracted from the feature map combined through at least one layer, and combine the extracted context information with the feature map.

4, 5, and 6 are diagrams illustrating neural networks according to various embodiments of the present invention.

Referring to FIG. 4, the neural network 400 includes a first layer 410, a context module 420, a combination module 430, and a second layer 440.

The first layer 410 receives an image or a feature map extracted from a previous layer, and extracts a feature map h _n-1 from the input image or the input feature map. In some embodiments, the first layer 410 may include any one of the plurality of layers 210 described with reference to FIG. 2.

_{The context module 420 generates a plurality of attention maps from the feature map h n-1} extracted from the first layer 410, and generates a plurality of region feature vectors each representing regions corresponding to the plurality of attention maps. _{And generates context information (c n} (h _n-1 )) based on the comparison result of a plurality of vectors. In some embodiments, the context module 420 may include a layer for generating an attention map.

The combining module 430 combines the context information c _n (h _{n-1 ) generated by the context module 420 to the feature map h n-1} extracted from the first layer 410. Since the vector generated in the context module 420 is a feature representing an important area in the input feature map, the context information (c _n (h _n-1 )) is combined with the feature map (h _n-1 ) in the combining module 430 By doing so, the comparison result of the important area in the feature map can be better identified.

The second layer 440 processes the feature map combined in the combining module 430. In some embodiments, the second layer 440 may extract the _{feature map h n} from the feature map combined in the combining module 430. In some embodiments, the second layer 440 may include any one of the plurality of layers 210 described with reference to FIG. 2.

Referring to FIG. 5, a neural network 500 according to another embodiment includes a first layer 510, a context module 520, a combination module 530, and a second layer 540. The first layer 510, the context module 520, and the second layer 540 perform operations similar to those of the first layer 410, the context module 420, and the second layer 440 described with reference to FIG. 4. As it is performed, its description is omitted.

As shown in FIG. 5, the neural network 500 includes an summing module as the combining module 530. The summing module 530 may add the context information c _n (h _{n-1 ) generated by the context module 520 to the feature map h n-1} extracted from the first layer 510.

Referring to FIG. 6, a neural network 600 according to another embodiment includes a first layer 610, a context module 620, a combination module 630, and a second layer 640. The first layer 610, the context module 620, and the second layer 640 perform similar operations to the first layer 410, the context module 420, and the second layer 440 described with reference to FIG. 4. As it is performed, its description is omitted.

As shown in FIG. 6, the neural network 600 includes a multiplication module as the combining module 630. The multiplication module 630 may multiply the context information (c _n (h _n-1 )) generated by the context module 620 by the feature map (h _n-1 ) extracted from the first layer 610.

4 to 6 illustrate one context module inserted between two layers, when a neural network includes a plurality of layers, a plurality of context modules may be used. In this case, each context module may be inserted between two layers of a plurality of layers.

Next, a context module and a method of generating context information according to various embodiments of the present invention will be described with reference to FIGS. 7 to 10.

7 is a flowchart illustrating a method of generating context information according to an embodiment of the present invention.

Referring to FIG. 7, the computing device receives a feature map from a previous layer (S710) and generates a plurality of attention maps from the input feature map (S720).

The computing device generates an area feature vector representing an area corresponding to each attention map by applying each attention map to the input feature map (S730). The computing device generates context information by comparing a plurality of region feature vectors (S740). In this case, since the region feature vector is a feature representing an important region in the input feature map, the context information can provide a comparison result of the important region in the feature map.

8, 9, and 10 are diagrams illustrating a context module according to various embodiments of the present invention. In FIGS. 8, 9, and 10, for convenience of description, a feature map h _n-1 input from a previous layer to a context module is represented by X.

Referring to FIG. 8, the context module 800 according to an embodiment includes an attention module 810, a vector generation module 820, and a comparison module 830.

The attention module 810 receives a feature map X from a previous layer and generates a plurality of attention maps from the input feature map X. When the dimension of the input feature map (X) is C×H×W (X ∈ R ^C×H×W ), the attention module 810 generates a plurality of attention maps having a size of H×W. I can. 8 illustrates a case in which two attention maps are generated for convenience of explanation. Here, C denotes the number of channels of the input feature map, H denotes the height of the input feature map, and W denotes the width of the input feature map. In some embodiments, each attention map may be a single channel attention map in the R ^{1×H×W dimension.}

In some embodiments, a plurality of attention maps generated by the attention module 810 may be different from each other. In some embodiments, the attention module 810 may generate an attention map by applying a 1×1 convolution operation to the input feature map X. In this case, the attention module 810 may generate an attention map by applying a learnable parameter to the input feature map X. The attention module 810 may generate different attention maps by applying different _{learnable parameters W K} and W _{Q to the input feature map X.} In some embodiments, the learnable parameters (W _K , W _Q ) may be the weights of the 1×1 convolution operation (W _K ∈ R ^C×1×1 _, W _Q ∈ R ^C×1×1 ). Accordingly, the attention module 810 may generate an attention map by applying a 1×1 convolution operation to the input feature map X as shown in Equation 2.

In Equation 2, X _i,j denotes a vector at the spatial position (i,j) of the input feature map, and A ^K _i,j denotes the spatial position (i,j) of the attention map to which the parameter (W _{K) is applied.} Represents a vector at, and A ^Q _i,j represents a vector at a spatial position (i,j) of the attention map to which the parameter (W _{Q) is applied.}

Next, the vector generation module 820 generates region feature vectors (K, Q) representing regions corresponding to the corresponding attention maps by applying each attention map to the input feature map. The generated region feature vectors (K, Q) may be a vector having a dimension of C×1×1 (K ∈ R ^C×1×1 , Q ∈ R ^C×1×1 ). In some embodiments, the vector generation module 820 is based on the sum of the result of an element-wise operation (e.g., element-wise multiplication) between the attention map and the input feature map. Q) can be generated. The two region feature vectors K and Q generated as described above may represent features that focus on two spatial regions of the input feature map X. That is, the region feature vectors (K, Q) can be used as features representing an important region in the input feature map (X).

Of the two region feature vectors (K, Q), one region feature vector (K) is a feature representing the region of interest, and the other region feature vector (Q) may be a feature representing another region (i.e., context) corresponding to the region of interest. have.

The comparison module 830 outputs a comparison result of a plurality of region feature vectors (K, Q) generated by a plurality of attention maps. In some embodiments, the comparison module 830 may output a result of subtracting the two region feature vectors K and Q (K-Q). In this case, the two region feature vectors may be subtracted element-wise.

Referring to FIG. 9, a context module 900 according to another embodiment includes an attention module 910, a normalization module 940, a vector generation module 920, and a comparison module 930. 9 illustrates a case in which two attention maps are generated for convenience of description. The attention module 910, the vector generation module 920, and the comparison module 930 may operate similarly to the attention module 810, vector generation module 820, and comparison module 830 described with reference to FIG. 8. And a detailed description thereof will be omitted.

The attention module 910 receives a feature map X from a previous layer and generates a plurality of attention maps from the input feature map X. In some embodiments, the attention module 910 may generate different attention maps by applying different learnable _{parameters W K} and W _{Q to the input feature map X.} In some embodiments, the learnable parameters (W _K , W _Q ) may be the weights of the 1×1 convolution operation (W _K ∈ R ^C×1×1 _, W _Q ∈ R ^C×1×1 ).

The normalization model 940 normalizes the attention map generated by the attention module 910. In some embodiments, the normalization model 940 may normalize the attention map based on a softmax function. For example, the normalization model 940 may normalize the attention map as shown in Equation 3. Equation 3 indicates a normalized vector _{of a vector (X i,j} ∈ R ^C×1×1 ) at the spatial location (i,j) of the input feature map.

In Equation 3, X _i,j denotes a vector at the spatial position (i,j) of the input feature map, and A ^K _i,j denotes the spatial position (i,j) of the attention map to which the parameter (W _{K) is applied.} Denotes the normalized vector at, and A ^Q _i,j denotes the normalized vector at the spatial position (i,j) of the attention map to which the parameter (W _{Q) is applied.}

The vector generation module 920 generates region feature vectors (K, Q) representing regions corresponding to the corresponding attention map by applying each attention map to the input feature map. The generated region feature vectors (K, Q) may be a vector having a dimension of C×1×1 (K ∈ R ^C×1×1 , Q ∈ R ^C×1×1 ). In some embodiments, the vector generation module 920 may generate region feature vectors (K, Q) by weighted average of the input feature map (X) based on the normalized map. For example, the vector generation model 920 may generate a region feature vector as shown in Equation 4.

The comparison module 930 outputs a comparison result of a plurality of region feature vectors (K, Q) generated by a plurality of attention maps. In some embodiments, the comparison module 830 may output a result of subtracting the two region feature vectors K and Q (K-Q).

Although the example in which the context module generates two attention maps has been described above, the number of attention maps generated by the context module is not limited to two, and three or more attention maps may be generated. Hereinafter, an example in which three attention maps are generated will be described with reference to FIG. 10.

Referring to FIG. 10, a context module 1000 according to another embodiment includes an attention module 1010, a vector generation module 1020, and a comparison module 1030. The attention module 1010 and the vector generation module 1020 may operate similarly to the attention module 810 and the vector generation module 820 described with reference to FIG. 8, and detailed descriptions thereof will be omitted.

The attention module 1010 receives a feature map X from a previous layer and generates a plurality of attention maps from the input feature map X. In some embodiments, the attention module 1010 may generate different attention maps by applying different learnable _{parameters W K} , W _Q , and W _{J to the input feature map X.} In some embodiments, the learnable parameters (W _K , W _Q , W _J ) may be the weights of the 1×1 convolution operation (W _K ∈ R ^C×1×1 _, W _Q ∈ R ^C×1×1 _, W _J ∈ R ^C×1×1 ).

The vector generation module 1020 generates region feature vectors (K, Q, J) representing regions corresponding to the corresponding attention map by applying each attention map to the input feature map. The generated region feature vectors (K, Q, J) may be C×1×1 dimensional vectors (K ∈ R ^C×1×1 , Q ∈ R ^C×1×1 , J ∈ R ^{C×1× 1} ). The three region feature vectors K, Q, and J generated as described above may represent a feature that focuses on three spatial regions of the input feature map X.

The comparison module 1030 is a comparison result of two region feature vectors (K, Q) of one combination among a plurality of region feature vectors (K, Q, J) generated by a plurality of attention maps and two region features of another combination. The final comparison result is output based on the comparison result of the vectors (Q, J). In some embodiments, the comparison module 1030 is a subtraction module 1031 that subtracts two region feature vectors (K, Q) of one combination and a subtraction module that subtracts two region feature vectors (Q, J) of another combination. (1032) may be included. In addition, the comparison module 1030 may further include a combining module 1033 that outputs a final comparison result based on the subtraction result KQ of the subtraction module 1031 and the subtraction result QJ of the subtraction module 1032. . In some embodiments, the combining module 1033 may combine two subtraction results (K-Q and Q-J) to generate a final comparison result. The combining module 1033 may combine the subtraction results using various operations.

In some embodiments, as described with reference to FIG. 10, the normalization model may be applied to a context module that generates three or more attention maps.

Next, a learning method of an image reading apparatus according to an embodiment of the present invention will be described with reference to FIGS. 11 and 12.

11 is a flowchart illustrating a learning method according to an embodiment of the present invention

Referring to FIG. 11, the computing device uses a data set including a plurality of images given a label as a training sample.

The computing device inputs the image of the data set into the neural network (S1110). A neural network is a neural network that includes a context module as described above. The computing device generates context information in each context module included in the neural network (S1120), and calculates a context loss (S1130). In some embodiments, the computing device may input the input image into a neural network to extract a feature map from the input image, extract context information from the feature map, and combine the context information with the feature map.

The computing device predicts an object included in the image while performing a target task on the image through a neural network (S1140). The computing device calculates a task loss between the predicted class of the object and the correct answer class of the label (1150).

The computing device updates the parameters of the neural network by backpropagating the context loss and the class loss to the neural network (S1160). The parameters of the neural network may include weights used in a plurality of layers included in the neural network and weights used in a context module. Accordingly, not only the layer used in the neural network but also the context module can be learned.

12 is a diagram illustrating a learning method according to an embodiment of the present invention.

Referring to FIG. 12, the neural network 1200 predicts an object of an image by performing a target task on an image 1230 of an image set. The neural network 1200 may predict an object of an image through various layers 1210. In this case, the task loss 1220 may be calculated based on the target task 1240 output by performing the target task in the neural network 1200 and a label given as a correct answer to the input image 1230. In some embodiments, task loss 1220 may be calculated based on a loss function, such as an L1 loss function or an L2 loss function. The L1 loss function stands for least absolute errors, and the L2 loss function stands for least square errors.

The context loss 1260 may be calculated based on the region feature vector representing the region corresponding to the attention map calculated by the context module 1220 of the neural network. In some embodiments, context loss 1260 may be calculated based on orthogonal loss between region feature vectors. The orthogonal loss can be calculated based on the region feature vector and the number of channels. _{For example, a context loss (l ctxt} (K, Q)) between two region feature vectors (K, Q) can be calculated as in Equation (5).

In Equation 5, C means the number of channels.

The final loss 1270 in the neural network 1200 is calculated based on the task loss 1250 and the context loss 1260. In some embodiments, the final loss 1270 may be calculated based on the sum of the context loss 1260 included in the neural network 1200 and the task loss 1250. For example, the final loss 1270 may be calculated as in Equation 6.

In Equation 6, l _fin represents the final loss, l _task represents the task loss, λ is a constant for controlling the effect of the context loss, and K _m and Q _m are regions created in the m-th context module of the neural network Represents a feature vector, and M represents the number of context modules included in the neural network (M is an integer greater than or equal to 1).

The final loss 1270 is backpropagated to the neural network 1200 so that a learnable parameter (eg, weight) of the neural network 1270 is updated. Accordingly, the neural network 1270 may be trained.

Meanwhile, the context module can normalize the input feature map, and this embodiment will be described with reference to FIG. 13.

13 is a diagram illustrating a context module according to another embodiment of the present invention.

Referring to FIG. 13, the context module 1300 includes a normalization module 1350, an attention module 1310, a vector generation module 1320, and a comparison module 1330. The attention module 1310, the vector generation module 1320, and the comparison module 1330 may operate similarly to the attention module 810, vector generation module 820, and comparison module 830 described with reference to FIG. 8. And the explanation is omitted.

The normalization module 1350 normalizes the feature map X input from the previous layer, and inputs the normalized feature map to the attention module 1310. The attention module 1310 generates a plurality of attention maps from the normalized input feature map.

In some embodiments, the normalization module 1350 may normalize the input feature map X by subtracting the C-dimensional mean vector of the input feature map X from the input feature map X. For example, the normalization module 1350 may normalize the input feature map X as shown in Equation 7.

In Equation 7, μ is the C-dimensional mean vector of X (μ ∈ R ^C×1×1 ).

13 illustrates a case in which two attention maps are generated for convenience of explanation, but three or more attention maps may be generated. In addition, the attention map may be normalized as described with reference to FIG. 9.

In some embodiments, a channel may be calibrated based on a vector used for normalization of an input feature map, and such an embodiment will be described with reference to FIG. 14.

14 is a diagram illustrating a neural network according to another embodiment of the present invention.

Referring to FIG. 14, the neural network includes a context module 1420, a combination module 1430 and 1460, and a channel recalibration module 1450.

The context module 1420 includes a normalization module 1425, an attention module 1411, a vector generation module 1422 and a comparison module 1423. The normalization module 1425, the attention module 1411, the vector generation module 1422, and the comparison module 1423 are the normalization module 1350, attention module 1310, and vector generation module 1320 described with reference to FIG. And the comparison module 1330 may operate similarly, and a description thereof will be omitted. The context module 1420 generates context information from the input feature map X. The combining module 1430 combines the context information generated by the context module 1420 with the input feature map X.

The channel recalibration module 1450 calculates a weight for each channel based on the C-dimensional average vector μ of the input feature map X. In some embodiments, the channel recalibration module 1450 may generate a channel feature vector P indicating which of the C channels to pay attention to. In some embodiments, the channel feature vector P may be a vector that scales down a specific channel among C channels. In some embodiments, the channel feature vector P may be a C×1×1 dimensional vector (P ∈ R ^C×1×1 ).

The combining module 1460 combines the channel feature vector P generated by the channel recalibration module 1450 to the feature map combined by the combining module 1430. In some embodiments, the combining module 1460 may multiply the feature map combined in the combining module 1430 by the channel feature vector generated by the channel recalibration module 1450.

In some embodiments, the channel recalibration module 1450 may generate a channel feature vector P by applying a 1×1 convolution operation and an activation function to the C-dimensional mean vector μ. In some embodiments, the channel recalibration module 1450 may use a plurality of combinations of 1×1 convolution operations and activation functions. In some embodiments, as shown in FIG. 14, the channel recalibration module 1450 may include a convolution module 1451, an activation module 1452, a convolution module 1453, and an activation module 1454. . In FIG. 14, it is shown that the activation module 1452 uses a ReLU (rectified linear unit) function as the activation function, and the activation module 1454 uses a sigmoid function as the activation function.

15 is a diagram illustrating a neural network according to another embodiment of the present invention.

A neural network according to another embodiment may use a grouped convolution operation as a convolution operation. In this case, the input and output are divided into G channel groups, and a group convolution operation may be separately performed for each group. When using the group convolution operation, it is possible to elaborately represent a number of important points in the input.

Referring to FIG. 15, the neural network includes a context module, combining modules 1530 and 1560, and a channel recalibration module.

The context module may include a normalization module 1525, an attention module 1521, a vector generation module 1522, a normalization module 1524, and a comparison module 1523, which may operate as described above, and the description thereof is provided. Omit it. In FIG. 15, a module that performs a group convolution operation by the attention module 1521 is shown, and a module that performs a softmax operation by the normalization module 1524 is shown. In this case, G attention maps may be output as a result of the group convolution operation 1521 and the softmax operation 1524. Region feature vectors (K, Q) each have G elements ([K ¹ ,...,K ^G ], [Q ¹ ,..., Q ^G ]), and can be expressed as Equation 8 . In FIG. 15, for convenience of explanation, a case where G is 3 is illustrated.

In Equation 8, g represents the g-th group.

The channel recalibration module may include a convolution module 1551, an activation module 1552, a convolution module 1535, and an activation module 1554, which may operate as described above, and a description thereof will be omitted. In FIG. 15, a module that performs a group convolution operation using convolution modules 1551 and 1553 is shown, and a module using a ReLU function and a module using a sigmoid function as the activation modules 1552 and 1554 are shown, respectively. I did.

Next, an exemplary computing device 1600 capable of implementing an image reading device according to an embodiment of the present invention will be described with reference to FIG. 16.

16 is a diagram illustrating a computing device according to an embodiment of the present invention.

Referring to FIG. 16, the computing device includes a processor 1610, a memory 1620, a storage device 1630, a communication interface 1640, and a bus 1650. The computing device 1600 may further include other general purpose components.

The processor 1610 controls the overall operation of each component of the computing device 1600. The processor 1610 may be implemented as at least one of various processing units such as a central processing unit (CPU), a microprocessor unit (MPU), a micro controller unit (MCU), and a graphic processing unit (GPU), and may be implemented as a parallel processing unit. May be. Also, the processor 1610 may perform an operation on a program for executing the image reading method described above.

The memory 1620 stores various types of data, commands, and/or information. The memory 1620 may load a computer program from the storage device 1630 to execute the image reading method described above. The storage device 1630 may non-temporarily store a program. The storage device 1630 may be implemented as a nonvolatile memory.

The communication interface 1640 supports wired/wireless Internet communication of the computing device 1600. In addition, the communication interface 1640 may support various communication methods other than Internet communication.

The bus 1650 provides communication functions between components of the computing device 1600. The bus 1650 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The computer program may include instructions that when loaded into the memory 1620 cause the processor 1610 to perform an image reading method. That is, the processor 1610 may perform an operation for an image reading method by executing a command.

In some embodiments, the computer program extracts a first feature map from the input image based on a neural network, generates a plurality of attention maps from the first feature map based on the neural network, and generates context information based on the plurality of attention maps. And a command for generating a second feature map by combining context information with the first feature map, and outputting a read result of the input image from the second feature map based on a neural network.

In some embodiments, the computer program extracts a first feature map from the input image based on the neural network, and a first vector representing the region of interest from the first feature map based on the neural network and a first vector representing the region corresponding to the region of interest. 2 Create a vector, generate context information based on the first vector and the second vector, combine the context information with the first feature map to generate a second feature map, and input from the second feature map based on a neural network It may include a command for outputting an image read result.

The image reading method or apparatus according to an embodiment of the present invention described above may be implemented as a computer program readable by a computer on a computer readable medium. In one embodiment, the computer-readable medium may be a removable recording medium or a fixed recording medium. In another embodiment, a computer program recorded in a computer-readable medium may be transmitted to another computing device through a network such as the Internet, and may be installed and executed in another computing device.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (19)

An image reading method performed by a computing device, comprising:
Extracting a first feature map from the input image based on the neural network,
Generating a plurality of attention maps from the first feature map based on the neural network,
Generating context information based on the plurality of attention maps,
Generating a second feature map by combining the context information with the first feature map, and
And outputting a read result of the input image from the second feature map based on the neural network,
Generating the context information,
Generating a plurality of vectors based on the plurality of attention maps and the first feature map, and
Comprising the step of comparing the plurality of vectors to generate the context information
How to read images. In claim 1,
The step of generating the context information by comparing the plurality of vectors includes comparing the plurality of vectors by subtracting two vectors from among the plurality of vectors. In claim 1,
Generating the plurality of vectors comprises:
Generating the plurality of attention maps by applying a 1×1 convolution operation having different parameters to the first feature map, and
Generating each vector by weighted average of the first feature map based on each attention map
Image reading method comprising a. In paragraph 3,
The generating of the plurality of vectors further comprises normalizing each attention map prior to weighted averaging the first feature map. In paragraph 3,
The 1×1 convolution operation is a group convolution operation performed after grouping a plurality of channels of the first feature map into a predetermined number of channel groups. In claim 1,
Calculating the loss of the context information based on the plurality of vectors, and
Updating the neural network by backpropagating the loss to the neural network
Image reading method further comprising a. In paragraph 6,
The step of calculating the loss comprises calculating the loss based on an orthogonal loss of the plurality of vectors. In claim 1,
The step of generating the plurality of attention maps includes normalizing the first feature map before generating the plurality of attention maps. In clause 8,
The step of normalizing the first feature map includes normalizing the first feature map by subtracting an average vector of the C dimension of the first feature map from the first feature map,
C indicates the number of channels of the first feature map
How to read images. In claim 9,
Generating a channel feature vector for reducing the size of a specific channel among the C channels based on the average vector,
The step of generating the second feature map includes combining the channel feature vector to a feature map in which the context information is combined with the first feature map.
How to read images. In claim 10,
The step of generating the channel feature vector comprises generating the channel feature vector by applying a 1×1 convolution operation and an activation function to the average vector. In clause 11,
The 1×1 convolution operation is a group convolution operation performed after grouping the C channels of the average vector into a predetermined number of channel groups. In claim 1,
When the first feature map is a C×H×W dimension, the context information is a C×1×1 dimension,
The C, H, and W respectively indicate the number, height, and width of the first feature map.
How to read images. In claim 1,
Outputting the read result of the input image
Generating a plurality of second attention maps from the second feature map based on the neural network,
Generating second context information based on the plurality of second attention maps,
Generating a third feature map by combining the second context information with the second feature map, and
Outputting a read result of the input image from the third feature map based on the neural network
Image reading method comprising a. In clause 14,
Calculating the loss of the context information,
Calculating the loss of the second context information,
Calculating a context loss based on the loss of the context information and the loss of the second context information, and
Updating the neural network by backpropagating the context loss to the neural network
Image reading method further comprising a. Memory to store instructions, and
By executing the command, a first feature map is extracted from an input image based on a neural network, and based on the neural network, a first vector representing a region of interest from the first feature map and a first vector representing a region corresponding to the region of interest 2 A vector is generated, context information is generated by comparing the first vector and the second vector, and a second feature map is generated by combining the context information with the first feature map, and the second feature map is generated based on the neural network. Processor for outputting a read result of the input image from a second feature map
Image reading device comprising a. In paragraph 16,
The processor generates a plurality of attention maps from the first feature map based on the neural network, and generates the first vector and the second vector based on the plurality of attention maps. A computer program executed by a computing device and stored in a recording medium,
The computer program is the computing device,
Extracting a first feature map from the input image based on the neural network,
Generating a first vector representing a region of interest and a second vector representing a region corresponding to the region of interest from the first feature map based on the neural network,
Generating context information by comparing the first vector and the second vector,
Generating a second feature map by combining the context information with the first feature map, and
Outputting a read result of the input image from the second feature map based on the neural network
A computer program that allows you to run. In paragraph 18,
Generating the first vector and the second vector comprises:
Generating a plurality of attention maps from the first feature map based on the neural network, and
Generating the first vector and the second vector based on the plurality of attention maps
Computer program comprising a.

Images