KR102289668B1 - Apparatus and method for semantic matching based on matching reliability - Google Patents

Abstract

Provided are a semantic matching apparatus based on matching reliability capable of significantly increasing semantic matching accuracy by being guided on the basis of a reliability volume, and a method thereof. According to the present invention, the semantic matching apparatus comprises: a correlation volume acquisition unit extracting features from each of the one pair of applied input images according to a pretrained pattern estimation method to generate one pair of feature maps formed of a plurality of feature vectors and analyzing a correlation according to the degree of similarity between the feature vectors between the generated pair of feature maps to acquire a correlation volume; a reliability volume acquisition unit calculating the reliability on the basis of mutual consistency between the correlations, which are elements of the correlation volume, performing reinforcement by estimating a pattern of an initial reliability volume according to the pretrained pattern estimation method, and filtering the reinforced reliability by a predetermined reference value to acquire a refined reliability volume; a guide map acquisition unit acquiring a displacement map by calculating displacements between the feature vectors corresponding to each other in the pair of feature maps according to the reliability of the refined reliability volume and generating a guide map by propagating the displacement of the displacement map to surrounding areas in a predetermined manner; and a matching unit compensating for the correlation so that an important region of the correlation volume is emphasized by using the guide map and estimating a pattern of a refined correlation volume compensated for the correlation according to the pretrained pattern estimation method to acquire a semantic matching map.

Description

Matching reliability-based semantic matching apparatus and method {APPARATUS AND METHOD FOR SEMANTIC MATCHING BASED ON MATCHING RELIABILITY}

The present invention relates to a semantic matching apparatus and method, and to a matching reliability-based semantic matching apparatus and method.

Conventionally, the corresponding point analysis of an image is an environment with geometrical constraints, such as stereo matching between two images acquired with a disparity at the same time point or optical flow between temporally continuous image frames. was used in

For example, depth information estimation through stereo matching searches only on the x-axis using an epipolar line, and object motion estimation through optical flow indicates that the range of motion between adjacent frames is small. Using the facts, the search was performed only within a window of a specified size.

However, recently, studies on semantic matching for searching for the same object or objects of various shapes belonging to the same semantically same category between images are being conducted. This semantic matching has to search for objects that have the same meaning in different images but appear in different shapes, so all pixels of the image must be searched without geometrical restrictions. Therefore, not only the search range is wide, but also the accuracy of the feature vector representing each pixel is not high due to a large change in the shape of the search target, so there is a problem in that a significant error occurs when measuring the similarity score that determines the matching level.

Korean Patent Publication No. 10-2018-0033037 (published on April 2, 2018)

An object of the present invention is to provide a semantic matching apparatus and method capable of generating a confidence volume for limiting the search space of semantic matching without geometrical constraints.

Another object of the present invention is to provide a semantic matching apparatus and method capable of significantly improving semantic matching accuracy by being guided based on a confidence volume.

A semantic matching apparatus according to an embodiment of the present invention for achieving the above object extracts features according to a pattern estimation method learned in advance from each of a pair of applied input images, and a feature map of a phase consisting of a plurality of feature vectors a correlation volume acquisition unit that generates a correlation volume and obtains a correlation volume by analyzing a correlation according to a degree of similarity between the feature vectors between the generated pair of feature maps; Reliability is calculated based on the mutual consistency between correlations, which are elements of the correlation volume, and the pattern of the initial confidence volume is estimated and reinforced according to a pre-learned pattern estimation method, and the reinforced reliability is filtered and refined by a predetermined reference value. a reliability volume obtaining unit for obtaining a reliability volume; A displacement map is obtained by calculating the displacement between the feature vectors corresponding to each other in the pair of feature maps according to the reliability of the refining reliability volume, and a guide map is generated by propagating the displacement of the displacement map to the surrounding area in a predetermined manner. a guide map acquisition unit; and compensating for correlation so that an important region of the correlation volume is emphasized using the guide map, and estimating the pattern of the refined correlation volume compensated for correlation according to a pre-learned pattern estimation method to obtain a semantic matching map includes wealth.

The correlation volume acquisition unit is implemented as two artificial neural networks in which a pattern estimation method is previously learned, respectively, and receives a corresponding input image from among the pair of input images, extracts features, and obtains the pair of feature maps. acquisition department; and a correlation analyzer configured to obtain the correlation volume by calculating a correlation between the feature vectors of the pair of feature maps using a cosine similarity.

The confidence volume obtaining unit may include: a mutual consistency analyzer generating an initial confidence volume indicating mutual consistency between correlations of the correlation volume; a reliability reinforcement unit for reinforcing the reliability of the initial reliability volume using an artificial neural network in which a pattern estimation method has been previously learned; and a reliability volume filtering unit configured to obtain the refined reliability volume by extracting a reliability value having a value exceeding a predetermined reference value from the reinforced reliability volume.

The reliability volume filtering unit may obtain the refined reliability volume by extracting a reliability value having a value exceeding a reference value from a result of multiplying the reinforced reliability volume by the reliability volume.

The guide map acquisition unit may include: a displacement map acquisition unit configured to obtain the displacement map by calculating a displacement indicating a degree of movement between corresponding feature vectors of a pair of feature maps based on the reliability of the refinement reliability volume; and a displacement map compensator configured to obtain the guide map by propagating the calculated displacement of the displacement map to the remaining entire region of the displacement map in a predetermined manner.

The displacement map obtaining unit may calculate a displacement, which is a position change between corresponding feature vectors of a pair of feature maps, using a soft argmax function according to the calculated reliability of the refinement reliability volume.

The displacement map compensator may propagate the displacements of the displacement map to the entire remaining area according to a moving least squares (MLS) technique.

The matching unit receives the guide map and transforms it to have a Gaussian distribution to adjust the displacement distribution of the guide map; a correlation volume guide unit that multiplies the transformed guide map and the correlation volume to compensate for the correlation of the correlation volume to obtain a refined correlation volume; and a matching map acquisition unit configured to estimate a pattern of the refined correlation volume using an artificial neural network in which a pattern estimation method has been previously learned, and obtain a semantic matching map that is a semantic matching result between the pair of input images.

The matching reliability-based semantic matching method according to another embodiment of the present invention for achieving the above other object is composed of a plurality of feature vectors by extracting features according to a pattern estimation method learned in advance from each of a pair of applied input images. generating a correlation volume by generating a feature map of one image and analyzing a correlation according to a degree of similarity between feature vectors between the generated pair of feature maps; Reliability is calculated based on the mutual consistency between correlations, which are elements of the correlation volume, and the pattern of the initial confidence volume is estimated and reinforced according to a pre-learned pattern estimation method, and the reinforced reliability is filtered and refined by a predetermined reference value. generating a confidence volume; A displacement map is obtained by calculating the displacement between the feature vectors corresponding to each other in the pair of feature maps according to the reliability of the refining reliability volume, and a guide map is generated by propagating the displacement of the displacement map to the surrounding area in a predetermined manner. to do; and compensating for correlation so that an important region of the correlation volume is emphasized using the guide map, and estimating a pattern of the refined correlation volume compensated for correlation according to a pre-learned pattern estimation method to obtain a semantic matching map. includes

Accordingly, the semantic matching apparatus and method according to an embodiment of the present invention generates a correlation volume according to the degree of similarity between features extracted from each image, calculates the mutual coherence for the degree of similarity of the correlation volume, and generates a confidence volume. By performing semantic matching by guiding the correlation volume with the obtained confidence volume, the matching accuracy can be greatly improved by limiting the search space of semantic matching without geometrical constraints.

1 shows a schematic structure of a semantic matching device according to an embodiment of the present invention.
FIG. 2 shows an example of an input image of the semantic matching apparatus of FIG. 1, a refinement reliability volume, a guide map, and a semantic matching map.
3 shows a semantic matching method according to an embodiment of the present invention.
4 and 5 are diagrams for comparing the performance of the semantic matching method according to the present embodiment.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless otherwise stated. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

1 shows a schematic structure of a semantic matching device according to an embodiment of the present invention, and FIG. 2 shows an example of an input image of the semantic matching device of FIG. 1, a refinement reliability volume, a guide map, and a semantic matching map.

Referring to FIG. 1 , the semantic matching apparatus according to the present embodiment may include a correlation volume obtaining unit 100 , a reliability volume obtaining unit 200 , a guide map obtaining unit 300 , and a matching unit 400 .

The correlation volume acquisition unit 100 receives a pair of input images (I ^s , I ^{t ), extracts features from the applied input images (I s} , I ^t ) according to a pre-learned pattern estimation method, and extracts a plurality of generating a feature map (F ^s, F ^t) consisting of the feature vector, and analyzing the correlation of the degree of similarity between the feature vector of the generated feature map (F ^s, F ^t) and obtains the correlation volume (C). Here, the input images I ^s and I ^{t may be a source image I s} and a target image I ^t for semantic matching, as shown in FIG. 2A .

The reliability volume acquisition unit 200 receives ^{a correlation volume C representing the correlation between the feature vectors of the feature maps F s} , F ^t , and calculates the reliability based on the mutual consistency of the correlation. The initial confidence volume (Q) is generated, the pattern of the initial confidence volume (Q) is estimated and reinforced according to the pre-learned pattern estimation method, and the reinforced reliability is filtered to refine the confidence volume ( Q') is obtained.

Meanwhile, the guide map acquisition unit 300 receives the refining reliability volume Q' and corresponds to the first and second feature maps F ^s and F ^{t according to the reliability remaining in the refining reliability volume Q'.} A guide map (G') for adjusting the correlation volume (C) by calculating the displacement between the feature vectors to obtain a displacement map (G) and propagating the displacement of the displacement map (G) to the surrounding area is shown in FIG. It is obtained as in (c).

The matching unit 400 receives the guide map (G') and transforms it according to the Gaussian model distribution, and guides and refines the correlation, which is the element value of the correlation volume (C), using the transformed guide map (G'). A correlation volume (C') is acquired, and a semantic matching map showing semantic matching between the ^{input images (I s} , I ^t ) from the pattern of the refined correlation volume (C') according to a pre-learned pattern estimation method is shown in FIG. It can be obtained as in (d).

Hereinafter, each configuration of the semantic matching device will be described in detail with reference to FIG. 1 .

First correlation volume acquisition unit 100 of FIG. 2 (a) and the respective characteristic input image (I ^s, I ^t) according to the same input image (I ^s, I ^t), the application receives a training pattern estimated in advance how extracted by analyzing the correlation between the generating a feature map (F ^s, F ^t) consisting of a plurality of feature vector characteristic map obtaining unit 110 and the feature map (F ^s, F ^t), a plurality of characteristic vector of the correlation It may include a correlation analyzer 120 to obtain the volume (C).

The feature map obtaining unit 110 may include first and second feature map obtaining units 111 and 112 in which a pattern estimation method is implemented as an artificial neural network previously learned, respectively, and the first feature map obtaining unit 111 . ^{obtains a first feature map (F s} ) including a plurality of feature vectors by extracting features of the source image (I ^s ), and the second feature map acquisition unit 112 obtains the features of the target image (I ^t ) It is possible to obtain ^{a second feature map (F t} ) including a plurality of feature vectors by extraction.

), 제1 및 제2 특징맵 획득부(111, 112)에 의해 획득되는 제1 및 제2 특징맵(F^s, F^t)은 각각 h × w × d의 크기(F^s, F^t

)로 획득될 수 있다.Here, if the input images (I ^s , I ^t ) have the same size (eg, I ^s , I ^{t )}

^{), the first and second feature maps F s} , F ^t acquired by the first and second feature map acquisition units 111 and 112 are h × w × d sizes (F ^s , F ^{t ), respectively.}

) can be obtained.

The first and second feature map acquisition units 111 and 112 may be implemented as, for example, a pre-trained convolutional neural network (CNN) or the like. CNN is a well-known artificial neural network that can effectively extract features of an image. Therefore, in the present embodiment, it is assumed that the first and second feature map acquisition units 111 and 112 use CNNs already learned according to the disclosed method. However, the first and second feature map acquisition units 111 and 112 may be implemented using other artificial neural networks.

When the first and second feature maps F ^s , F ^t are obtained by the feature map acquisition unit 110 , the correlation analysis unit 120 performs the first and second characteristics based on the cosine similarity as shown in Equation 1 A correlation volume (C) is obtained by calculating the correlation between each feature vector of the map (F ^s , F ^{t ).}

Where i, j are the first and second characteristics map (F ^s, F ^t) represents the position coordinates of the feature vectors in each of <> it is the cosine similarity function, ∥∥ ₂ is the L ₂ -norm function.

)로 획득될 수 있다.That is, the correlation volume (C) is ^{calculated as a correlation based on the mutual similarity between all the feature vectors of the first and second feature maps (F s} , F ^t ), and has a size of (h×w)×(h×w). (C

) can be obtained.

The confidence volume acquisition unit 200 uses an artificial neural network in which a pattern estimation method has been previously learned with a mutual consistency analysis unit 210 that generates an initial confidence volume Q indicating mutual consistency between correlations of the correlation volume C. The reliability reinforcement unit 220 for reinforcing the reliability of the initial reliability volume (Q) using the It may include a confidence volume filtering unit 230 for obtaining the refinement confidence volume (Q'). In FIG. 2 , for convenience of explanation, ^{positions of corresponding points in the input images (I s} , I ^t ) are connected to each other to indicate a refinement reliability volume (Q′).

The cross-coherence analyzer 210 calculates the maximum correlation among correlations between the feature vectors of ^{the second feature map F t} with respect to the feature vectors of ^{the first feature map F s} in the correlation volume C as shown in Equation 2 Among the correlations between the feature vectors of the first feature map (F ^s ) with respect to the feature vectors of the (max _i C _ij ) and the second feature map (F ^t ), the maximum correlation (max _j C _ji ) and between the two feature vectors A confidence value (Q _ij ) for each feature vector can be calculated based on the ratio of the correlation squared ((C _ji ) ^{2 ) of .}

According to Equation 2, the reliability value (Q _ij ) is calculated as a value between [0, 1], and is _{1 only when i and j correspond to each other (ie, C ij} = C _ji ), and is smaller than 1 otherwise. Calculated. Therefore, the closer to 1 the reliability (Q _ij ) is, the higher the reliability, and the closer to 0, the lower the reliability.

The mutual consistency analyzer 210 obtains the initial reliability Q by calculating the _{reliability values Q ij} for all correlations of the correlation volume C according to Equation 2 .

However, in the case of the initial reliability (Q), the first and second feature maps (F ^s , F ^t ) extracted from the first and second feature map acquisition units 111 and 112 of the characteristic map acquisition unit 110 are incomplete. Accuracy may be low due to gender. Although the first and second feature map acquisition units 111 and 112 are implemented with an artificial neural network such as CNN, it is possible to extract ^{the first and second feature maps F s} , F ^{t with higher accuracy than before,} Depending on the learning level of the first and second feature map acquisition units 111 and 112 or ^{the characteristics of the input image I s} , I ^t itself, the first and second feature maps F ^s , F ^t are extracted inaccurately. could be In addition, when the first and second feature maps F ^s and F ^t are incorrectly extracted, the accuracy of the initial reliability Q also decreases.

Accordingly, in the present embodiment, the reliability of the initial reliability volume Q is reinforced by providing the reliability reinforcement unit 220 . The reliability reinforcement unit 220 is implemented as an artificial neural network in which the pattern estimation method is previously learned to estimate the pattern of the initial confidence volume Q, and outputs the reinforced reliability volume by correcting the _{reliability Q ij according to the estimated pattern.} can do.

And, the reliability volume filtering unit 230 compares the reliability product obtained by multiplying each reliability of the reinforced reliability volume by each reliability of the reliability volume Q with a predetermined reference value ρ, as shown in Equation 3, and exceeds the reference value ρ. It is possible to obtain a refinement reliability volume (Q') by leaving only the reliability and removing the remainder.

Here, [F(Q;W _P )] represents the reliability volume reinforced by the reliability reinforcement unit 220 implemented as an artificial neural network, W _P indicates the weight of the reliability reinforcement unit 220, and T is the reference value ρ ) is a truncation function that removes values less than or equal to .

That is, the reliability volume filtering unit 230 obtains a refined reliability volume Q' by extracting only a high level of reliability exceeding the reference value ρ from the product of the reinforced reliability volume and the reliability volume Q.

However, in some cases, the reliability volume filtering unit 230 may be configured to take only the reliability exceeding the reference value ρ as it is from the reinforced reliability volume.

The guide map acquisition unit 300 receives the refining reliability volume Q', and based on the high-level reliability remaining in the refining reliability volume Q', corresponding to the first and second feature maps F ^s , F ^t . A displacement map acquisition unit 310 that obtains a displacement map G by calculating a displacement indicating a displacement between feature vectors, and a displacement map G with a displacement obtained from the displacement map G for high reliability The guide map G' may include a displacement map compensator 320 by propagating to the entire area of .

The displacement map obtaining unit 310 simply calculates the position (i, ) between the corresponding feature vectors of the first and second feature maps (F ^s , F ^t _{) from the reliability value (Q' ij} ) of the refinement reliability volume (Q'). j) The displacement map G can be obtained by calculating the difference, but in this case, the calculation is performed in a non-differentiable form so that the loss cannot be backpropagated during learning of the reliability reinforcement unit 220 implemented as an artificial neural network. do. That is, the reliability reinforcement unit 220 cannot be trained. Accordingly, in the present embodiment, the displacement map acquisition unit 310 obtains the displacement, which is a change in position between the corresponding feature vectors of ^{the first and second feature maps F s} and F ^{t , using the soft argmax function as shown in Equation (4).} A displacement map G is obtained by calculation.

where S denotes the confidence set of the refinement confidence volume (Q').

However, even if the displacement map G is obtained using the soft argmax function as in Equation 4, differentiation is still impossible due to invalidated values because they are not included in the confidence set S.

Accordingly, the displacement map compensator 320 propagates the displacements of the displacement map G to the entire remaining area of the displacement map G according to a moving least squares (MLS) technique as shown in Equation 5. A guide map G' is obtained as in (c).

이고, c_P는 미리 설정된 계수이다.where w is the weight function

, and c _P is a preset coefficient.

According to Equation (5), as the distance from the high-reliability position increases, the lower weight is reflected to fill the entire area of the displacement map (G). Also, since the weight function w is linear, the guide map G' has a differentiable form.

The matching unit 400 receives the guide map G' and converts it according to a Gaussian parametric model to adjust the displacement distribution of the guide map G'. Using the correlation volume guide unit 420 that multiplies the guide map and the correlation volume C to compensate for the correlation of the correlation volume C to obtain the refined correlation volume C' and the artificial neural network in which the pattern estimation method has been learned in advance. and a matching map acquisition unit 430 for estimating the pattern of the refinement correlation volume C' to obtain a semantic matching map that is a result of semantic matching between ^{the input images I s} and I ^{t .}

The guide distribution control unit 410 and the correlation volume guide unit 420 may obtain the refined correlation volume C′ according to Equation (6).

는 가우시안 파라메트릭 모델에 다라 변환된 가이드맵으로 k와 c_M은 각각 가우시안 모델의 분포를 조절하는 하이퍼파라미터이다. 즉 제1 및 제2 특징맵(F^s, F^t)의 각 특징 벡터간 상관 관계를 나타내는 상관 볼륨(C)의 상호 일관성에 따른 신뢰도 중 고수준의 신뢰도로부터 생성된 가이드맵(G')을 기반으로 상관 볼륨(C)의 상관 관계들을 강조하여, 정제 상관 볼륨(C')을 획득한다.here

is a guide map transformed according to the Gaussian parametric model, and k and c _M are hyperparameters that control the distribution of the Gaussian model, respectively. That is, based on the guide map (G') generated from the high level of reliability among the reliability according to the mutual consistency of the correlation volume (C) representing the correlation between the respective feature vectors of the first and second feature maps (F ^s , F ^{t )} By emphasizing the correlations of the correlation volume (C) with , a refined correlation volume (C') is obtained.

The matching map acquisition unit 430 receives the refined correlation volume C' in which the main region is emphasized based on the guide map G', and estimates the pattern of the refined correlation volume C', so that the input image I ^s , I ^t ), a semantic matching map (τ) indicating the semantically matched mutual positions is obtained as shown in (d) of FIG. 2 .

Here, the semantic matching map τ obtained by the matching map acquisition unit 430 may be expressed by Equation (7).

Here, [F(C';W _M )] _i represents each element of the matching map estimated by the matching map acquiring unit 430 implemented as an artificial neural network, and W _M is the weight of the matching map acquiring unit 430 . , τ _i denotes an element of the semantic matching map (τ).

Each element of the semantic matching map by adding the _{corresponding element (G' i} ) of the guide map (G') to each element ([F(C';W _M )] _{i ) of the matching map estimated in Equation 7} Acquiring (τ _i ) is to facilitate learning of the matching map acquisition unit 430 .

The matching map acquisition unit 430 may be learned to acquire the matching map ([F(C'; W _M )]) as the semantic matching map τ as it is without adding the guide map G'. . However, if the matching map ([F(C';W _M )]) is learned to be obtained as the semantic matching map (τ) as it is, the change in the matching map ([F(C';W _M )]) in the learning process As the width increases, it takes a long time for the learning to be completed stably.

On the other hand, as in Equation 7, the matching map obtaining unit 430 obtains the semantic matching map τ by adding the guide map G' _{to the matching map ([F(C';W M )]).} When configured to do so, the fluctuation range of the matching map ([F(C';W _M )]) is reduced during training of the artificial neural network, so that it can be learned more quickly. Accordingly, here, the semantic matching map τ is obtained by adding the guide map G' to the matching map ([F(C'; W _{M )]), but as described above, the matching map acquisition unit 430} ) may be configured such that the matching map ([F(C';W _M )]) is obtained as the semantic matching map τ as it is.

Meanwhile, although not shown, the semantic matching apparatus according to the present embodiment may further include a learning unit (not shown) for learning the reliability reinforcement unit 220 and the matching map acquisition unit 430 implemented as an artificial neural network.

In this embodiment, the learning unit calculates a loss by applying a training image with a pre-labeled semantic matching region as an input image (I ^s , I ^t ), and by backpropagating the calculated loss, the reliability reinforcement unit 220 and the matching map The acquisition unit 430 may be trained.

Learning unit calculates a consistent loss _(P L) in order to study the reliability reinforcing portion 220. The The loss of consistency (L _{P ) is calculated as the loss of geometry consistency (L geo} ), considering that the local geometries between the features of the source image (I ^s ) and the target image (I ^{t ) must be similar.} It is calculated and calculated by adding the _{silhouette consistency loss (L sil} ) considering that the silhouette of the semantic object must correspond in the refinement confidence volume (Q') and the initial confidence volume (Q).

Geometry consistent losses (L _geo) is the source image (I ^s) characteristic map (F ^s) and the warping results in accordance with the target image (I ^t) characteristic map (F ^t) to the guide map (G ') for for between It can be calculated as in Equation 8 as a difference of .

where N _i is a local area window centered at position i, o is a warping operator, and ||||| ² _F denotes a Frobenius norm function.

Meanwhile, the silhouette coherence loss (L _sil ) may be calculated according to Equation (9).

Here, since S ^* = {i,j|Q' _ij ≠ 0}, Q' _ij /Q _ij becomes the reliability volume ([F(Q;WP)] _ij ) reinforced by the reliability reinforcement unit 220 .

From Equations (8) and (9 _{), the loss of consistency (L P} ) can be calculated by Equation (10).

where λ is the loss weight.

And in order to learn the matching map acquisition unit 430, the learning unit uses the feature map (F ^t ) for the target image (I ^{t ) among the input images (I s} , I ^t ) in which the semantic matching area is pre-labeled as the semantic matching map. By warping according to (τ), the similarity ratio (P _i (τ)) with the feature map (F ^{s ) for the source image (I s} ) is calculated according to Equation 11, and the similarity ratio (P _i (τ)) Calculate the residual transformation loss (L _M ) according to Equation (12) using

_{In addition, the learning unit may calculate a regularization loss (L sm} ) by calculating the spatial smoothness of the semantic matching map (τ) with the L1 norm ( ) function.

Finally, as shown in Equation 13, the learning unit _{calculates the total loss by adding the coherence loss (L P} ), the residual transformation loss (L _M ), and the normalization loss (L _sm ), and backpropagates the calculated total loss to the reliability reinforcement unit 220 and the matching map acquisition unit 430 may be trained.

Here, λ _P , λ _{M ,} and λ _sm may be set to 1, 0, or 0.1 as weight parameters, respectively.

3 shows a semantic matching method according to an embodiment of the present invention.

Referring to FIG. 1, the semantic matching method of FIG. 3 is described. First, a pair of input images (I ^s , I ^t ) is acquired, features are extracted, and a correlation is analyzed based on the similarity between the features to obtain a correlation volume ( C) is generated (S10).

In the step of generating the correlation volume C (S10), a pair of input images I ^s and I ^t are first obtained (S11). Then, the feature maps F ^s , F ^{t are generated by extracting features from the input images I s} , I ^t obtained according to the pre-learned pattern estimation method ( S12 ). Then, the correlation volume C is obtained by analyzing the correlation according to the similarity between the feature vectors of the generated feature maps F ^s and F ^{t (S13).}

And when the correlation volume is generated, the reliability is calculated based on the mutual consistency of the correlation of the correlation volume C, and the refinement reliability volume Q' is generated by reinforcement and filtering (S20).

The step (S20) of generating the refinement reliability volume (Q') is to calculate the reliability value according to the mutual consistency between correlations, which are each element of the correlation volume (C), according to Equation 2 to obtain the initial confidence volume (Q). generated (S21). When the initial confidence volume Q is generated, a reinforced confidence volume is obtained by correcting the _{reliability Q ij} of the initial confidence volume Q using the artificial neural network in which the pattern estimation method has been previously learned ( S22 ). And by comparing each reliability of the reinforced reliability volume with a predetermined reference value (ρ), only the reliability exceeding the reference value (ρ) is removed and the remainder is removed to obtain a refined reliability volume (Q') (S23). In this case, by comparing the reliability product obtained by multiplying each reliability of the reinforced reliability volume by each reliability of the reliability volume Q with a predetermined reference value ρ, the refinement reliability volume Q' may be obtained.

"If the obtained, purified reliability volume (Q purified reliability volume (Q)") the first and second characteristics map (F ^s, F ^t) displacement map by calculating a displacement between the corresponding feature vector in accordance with the reliability of the ( G) is obtained, and a guide map G' is generated by propagating the displacement of the displacement map G to the surrounding area (S30).

The step S30 of generating the guide map G' is a displacement that is a change in position between the corresponding feature vectors of ^{the first and second feature maps F s} and F ^{t according to Equation 4 using the soft argmax function.} It is possible to obtain a displacement map (G) by calculating (S31). The guide map G' may be obtained by propagating the displacements of the obtained displacement map G to the periphery according to the moving least squares (MLS) technique of Equation 5 (S32).

On the other hand, when the guide map G' and the correlation volume C are generated, the generated guide map G' is converted to follow a Gaussian distribution, and the correlation of the correlation volume C is compensated with the converted guide map. After that, the semantic matching map τ is estimated by estimating the pattern of the refined correlation volume C′ compensated for the correlation according to the pre-learned pattern estimation method (S40).

In the step of estimating the semantic matching map τ ( S40 ), first, the guide map G′ is converted according to a Gaussian parametric model to follow a Gaussian distribution ( S41 ). Then, the converted guide map is multiplied by the correlation volume C to obtain a refined correlation volume C' for which the correlation is compensated (S42). When the refined correlation volume (C') is obtained, a semantic matching map (τ) between the ^{input images (I s} , I ^t ) is obtained from the pattern of the refined correlation volume (C') according to a pre-learned pattern estimation method ( S43).

4 and 5 are diagrams for comparing the performance of the semantic matching method according to the present embodiment.

In FIG. 4 , the left image and the middle image represent the source image I ^s and the target image I ^t , respectively, and the right image represents the source image I ^s warped by the semantic matching map τ. And Figure 5 shows the source image (I ^s) warped according to the semantic matching map obtained according to the conventional RTN and NCNet and FSNet techniques for the source image (I ^s) and the target image (I ^t) of FIG.

Comparing FIGS. 4 and 5 , it can be seen that the semantic matching method according to the present embodiment warps in a shape most similar to the object shape of the ^{target image It t compared to other existing techniques.} That is, semantic matching can be performed very accurately between the ^{source image I s} and the target image I ^{t .}

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is merely exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: correlation volume acquisition unit 110: feature map acquisition unit
120: correlation analysis unit 200: reliability volume acquisition unit
210: mutual consistency analysis unit 220: reliability reinforcement unit
230: reliability volume filtering unit 300: guide map acquisition unit
310: displacement map acquisition unit 320: displacement map compensation unit
400: matching unit 410: guide distribution control unit
420: correlation volume guide unit 430: matching map acquisition unit

Claims (20)

A feature map of a phase consisting of a plurality of feature vectors is generated by extracting features from each of the applied pair of input images according to a pattern estimation method learned in advance, and the degree of similarity between the feature vectors between the pair of generated feature maps is calculated. Correlation volume acquisition unit to obtain a correlation volume by analyzing the correlation according to the
Reliability is calculated based on the mutual consistency between correlations, which are elements of the correlation volume, and the pattern of the initial confidence volume is estimated and reinforced according to a pre-learned pattern estimation method, and the reinforced reliability is filtered and refined by a predetermined reference value. a reliability volume obtaining unit for obtaining a reliability volume;
A displacement map is obtained by calculating the displacement between the feature vectors corresponding to each other in the pair of feature maps according to the reliability of the refining reliability volume, and a guide map is generated by propagating the displacement of the displacement map to the surrounding area in a predetermined manner. a guide map acquisition unit; and
A matching unit that compensates for the correlation so that an important region of the correlation volume is emphasized using the guide map, and obtains a semantic matching map by estimating the pattern of the refined correlation volume compensated for the correlation according to a pre-learned pattern estimation method including,
The correlation volume acquisition unit
a feature map acquisition unit that is implemented with two artificial neural networks in which each pattern estimation method is learned in advance, receives a corresponding input image from among the pair of input images, extracts features, and obtains the pair of feature maps; and
and a correlation analyzer configured to obtain the correlation volume by calculating a correlation between the feature vectors of the pair of feature maps using a cosine similarity.
delete The method of claim 1, wherein the correlation analysis unit
The correlation between each feature vector of the pair of feature maps (F ^s , F ^{t ) is expressed by the equation}

(Here, i and j denote feature vector position coordinates in each of the first and second feature maps F ^s , F ^t , <> is a cosine similarity function, and │ ₂ is an L ₂ -norm function.)
A semantic matching device that calculates according to According to claim 1, wherein the reliability volume acquisition unit
a mutual consistency analyzer for generating an initial confidence volume indicating mutual consistency between correlations of the correlation volume;
a reliability reinforcement unit for reinforcing the reliability of the initial reliability volume using an artificial neural network in which a pattern estimation method has been previously learned; and
A semantic matching apparatus comprising: a confidence volume filtering unit configured to obtain the refined reliability volume by extracting a reliability value having a value exceeding a predetermined reference value from the reinforced reliability volume. 5. The method of claim 4, wherein the mutual consistency analyzer
Among the correlations between the feature vectors of the second feature map (F ^{t ) with respect to the feature vectors of the first feature map (F s} ) among the pair of feature maps in the correlation volume, the _{maximum correlation (max i} C _ij ) and the second feature Among the correlations between the feature vectors of the first feature map (F ^s ) with respect to the feature vectors of the map (F ^t _{), the maximum correlation (max j} C _ji ) and the squared correlation between the two feature vectors ((C _ji ) ² _{), the confidence value (Q ij} ) for each feature vector is calculated based on the ratio of

A semantic matching device that calculates according to The method of claim 4, wherein the reliability volume filtering unit
A semantic matching apparatus for obtaining the refined reliability volume by extracting a reliability value having a value exceeding a reference value from a result of multiplying the reinforced reliability volume by the reliability volume. 5. The method of claim 4, wherein the guide map acquisition unit
a displacement map acquisition unit configured to obtain the displacement map by calculating a displacement indicating a degree of movement between corresponding feature vectors of a pair of feature maps based on the reliability of the refinement reliability volume; and
and a displacement map compensator configured to obtain the guide map by propagating the calculated displacement of the displacement map to the remaining entire area of the displacement map in a predetermined manner. The method of claim 7, wherein the displacement map acquisition unit
A semantic matching device for calculating a displacement, which is a position change between corresponding feature vectors of a pair of feature maps, using a soft argmax function according to the calculated reliability of the refinement reliability volume. The method of claim 8, wherein the displacement map compensator
A semantic matching device for propagating the displacements of the displacement map to the entire remaining area according to a moving least squares (MLS) technique. The method of claim 9, wherein the matching unit
a guide distribution control unit that receives the guide map and converts it to have a Gaussian distribution to adjust the displacement distribution of the guide map;
a correlation volume guide unit that multiplies the transformed guide map and the correlation volume to compensate for the correlation of the correlation volume to obtain a refined correlation volume; and
A semantic matching apparatus comprising: a matching map acquisition unit configured to estimate a pattern of the refined correlation volume using an artificial neural network trained in a pattern estimation method in advance, and obtain a semantic matching map that is a semantic matching result between the pair of input images. A feature map of a phase consisting of a plurality of feature vectors is generated by extracting features from each of the applied pair of input images according to a pattern estimation method learned in advance, and the degree of similarity between the feature vectors between the pair of generated feature maps is calculated. generating a correlation volume by analyzing the corresponding correlation;
Reliability is calculated based on the mutual consistency between correlations, which are elements of the correlation volume, and the pattern of the initial confidence volume is estimated and reinforced according to a pre-learned pattern estimation method, and the reinforced reliability is filtered and refined by a predetermined reference value. generating a confidence volume;
A displacement map is obtained by calculating the displacement between the feature vectors corresponding to each other in the pair of feature maps according to the reliability of the refining reliability volume, and a guide map is generated by propagating the displacement of the displacement map to the surrounding area in a predetermined manner. to do; and
Compensating for the correlation so that the important region of the correlation volume is emphasized using the guide map, and estimating the pattern of the refined correlation volume compensated for the correlation according to a pre-learned pattern estimation method to obtain a semantic matching map; including,
The step of generating the correlation volume includes
obtaining the pair of feature maps by receiving a corresponding input image from among the pair of input images by using two artificial neural networks, each of which pattern estimation methods have been previously learned, and extracting features; and
and obtaining the correlation volume by calculating a correlation between each feature vector of the pair of feature maps using a cosine similarity.
delete 12. The method of claim 11, wherein obtaining the correlation volume comprises:
The correlation between each feature vector of the pair of feature maps (F ^s , F ^{t ) is expressed by the equation}

(Here, i and j denote feature vector position coordinates in each of the first and second feature maps F ^s , F ^t , <> is a cosine similarity function, and │ ₂ is an L ₂ -norm function.)
A semantic matching method that calculates according to 12. The method of claim 11, wherein generating the refinement confidence volume comprises:
generating an initial confidence volume representing mutual consistency between correlations in the correlation volume;
Reinforcing the reliability of the initial confidence volume using an artificial neural network in which the pattern estimation method has been previously learned; and
and extracting a confidence value having a value exceeding a predetermined reference value from the reinforced confidence volume to obtain the refined confidence volume. 15. The method of claim 14, wherein generating the initial confidence volume comprises:
Among the correlations between the feature vectors of the second feature map (F ^{t ) with respect to the feature vectors of the first feature map (F s} ) among the pair of feature maps in the correlation volume, the _{maximum correlation (max i} C _ij ) and the second feature Among the correlations between the feature vectors of the first feature map (F ^s ) with respect to the feature vectors of the map (F ^t _{), the maximum correlation (max j} C _ji ) and the squared correlation between the two feature vectors ((C _ji ) ² _{), the confidence value (Q ij} ) for each feature vector is calculated based on the ratio of

A semantic matching method that calculates according to 15. The method of claim 14, wherein obtaining the refinement confidence volume comprises:
A semantic matching method for obtaining the refined reliability volume by extracting a reliability value having a value exceeding a reference value from a result of multiplying the reinforced reliability volume by the reliability volume. 15. The method of claim 14, wherein generating the guide map comprises:
obtaining the displacement map by calculating a displacement indicating a degree of movement between corresponding feature vectors of a pair of feature maps based on the reliability of the refinement reliability volume; and
and acquiring the guide map by propagating the calculated displacement of the displacement map to the remaining entire area of the displacement map in a predetermined manner. The method of claim 17, wherein the obtaining of the displacement map comprises:
A semantic matching method for calculating a displacement, which is a position change between corresponding feature vectors of a pair of feature maps, using a soft argmax function according to the calculated reliability of the refinement reliability volume. 19. The method of claim 18, wherein obtaining the guide map comprises:
A semantic matching method for propagating the displacements of the displacement map to the entire remaining area according to a moving least squares (MLS) technique. The method of claim 19, wherein obtaining the semantic matching map comprises:
adjusting the displacement distribution of the guide map by receiving the guide map and transforming it to have a Gaussian distribution;
multiplying the transformed guide map and the correlation volume to compensate for the correlation of the correlation volume to obtain a refined correlation volume; and
and estimating the pattern of the refined correlation volume using an artificial neural network in which a pattern estimation method has been previously learned, and obtaining a semantic matching map that is a semantic matching result between the pair of input images.

Images