KR102395166B1 - Apparatus and method for calculating similarity based on feature coordinates - Google Patents

Abstract

Provided is a calculation device. The calculation device comprises: an acquisition unit acquiring an image in which an object exists; an extraction unit extracting a feature coordinate of the object; a matching unit that matches the features coordinate extracted from a plurality of images with each other; and a calculation unit that calculates a similarity between a plurality of images using the matching number of feature coordinates.

Description

Apparatus and method for calculating similarity based on feature coordinates}

The present invention relates to an apparatus and method for calculating a degree of similarity based on feature coordinates and the like.

Calculation of similarity may be generally performed with respect to the same dimension. However, in some cases, it is necessary to perform similarity calculation for different dimensions.

For example, it may be necessary to calculate the similarity between an object in a photograph and an object in a CAD drawing composed of only lines.

In this case, the similarity calculation methodology between an article defined as an object in a photograph and a drawing defined as an object in a CAD drawing is the cosine similarity or Euclidean distance is used.

However, the similarity calculation method using the cosine similarity or Euclidean distance has a disadvantage in that the explanation possibility of the model for calculating the similarity is low.

Korean Patent Publication No. 1282816 discloses a similarity measurement technique for images that can process normalized cross-correlation in real time.

Korean Patent Publication No. 1282816

An object of the present invention is to provide a calculation apparatus and method for calculating a degree of similarity between objects included in different types of images.

A calculation device of the present invention includes: an acquisition unit configured to acquire an image in which an object exists; an extraction unit for extracting feature coordinates of the object; a matching unit for matching feature coordinates extracted from a plurality of images with each other; and a calculation unit that calculates a degree of similarity between a plurality of images by using the matching number of the feature coordinates.

The calculation method of the present invention includes an acquisition step of acquiring an image in which an object exists; an extraction step of extracting a feature vector of the object; a matching step of matching feature vectors extracted from a plurality of images with each other; and calculating the similarity between a plurality of images by using the matching number of the feature vectors.

The present invention can provide a new method for calculating the similarity between objects included in images of different types and dimensions.

The present invention can present a similarity calculation methodology that combines high accuracy and visual description.

According to the present invention, the similarity between a design right having a drawing expressed only in lines and a photograph of a specific object can be numerically calculated. Also, according to the present invention, similarity matching results between images of different types such as drawings and photos can be visualized. The visualized matching result may provide an environment in which a user's manual interpretation is possible.

1 is a schematic diagram showing a calculation device of the present invention.
2 is a schematic diagram showing the operation of the calculation device.
3 is a schematic diagram illustrating an operation of a calculation device for another object.
4 is a schematic diagram illustrating an operation of a calculation device targeting another object.
5 is a flowchart illustrating a calculation method according to the present invention.
6 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present specification, duplicate descriptions of the same components will be omitted.

Also, in this specification, when it is said that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but other elements in the middle It should be understood that there may be On the other hand, in this specification, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In addition, the terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention.

Also, in this specification, the singular expression may include the plural expression unless the context clearly dictates otherwise.

Also, in this specification, terms such as 'include' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more It is to be understood that the existence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof, is not precluded in advance.

Also in this specification, the term 'and/or' includes a combination of a plurality of described items or any item of a plurality of described items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

1 is a schematic diagram showing a calculation device of the present invention.

The calculation device shown in FIG. 1 may include an acquisition unit 110 , an extraction unit 130 , a matching unit 150 , and a calculation unit 170 .

The acquisition unit 110 may acquire an image in which an object exists. As an example, the image may include a photo, a drawing, and the like. The photo may include a color photo or a black-and-white photo in RGB (Red, Green, Blue) scale. A drawing may be a representation of an object only with lines.

The acquisition unit 110 may be provided with a communication module that communicates with a generating means such as a camera that generates an image, or communicates with a server in which an image is stored. The acquisition unit 110 may acquire various images including an object from a generating means or a server through a communication module through a wired/wireless communication network.

The extractor 130 may extract feature coordinates or feature vectors of the object included in the image. The feature coordinates or feature vectors may be extracted using a feature coordinate extraction algorithm. As the feature coordinate extraction algorithm, an artificial intelligence-based model, Superpoint, etc. may be used. Alternatively, the feature coordinate extraction algorithm may use algorithms such as Scale Invariant Feature Transform (SIFT) and Speed Up Robust Features (SURF), which are traditional methodologies.

The matching unit 150 may match feature coordinates or feature vectors extracted from a plurality of images with each other. 'Matching' described in the present specification may mean determining whether the intrinsic features of objects included in each image match each other, or may mean determining whether feature coordinates corresponding to the coordinates of the intrinsic features match each other. Alternatively, 'matching' may mean tying the characteristic coordinates of each image matching each other as a pair in a state in which the intrinsic characteristics of the objects included in each image coincide with each other. 'Concordance' may mean not only that two values are completely the same, but that one value corresponds to a value within a setting range centered on another value.

The matching unit 150 may calculate whether the characteristic coordinates coincide with the unique features generated in different images by using a coordinate matching algorithm. The matching unit 150 may output the characteristic coordinates of the first image and the characteristic coordinates of the second image that match each other. In this case, the available coordinate matching algorithm may use an artificial intelligence-based model such as 'SuperGlue', or the 'Brute-Force' algorithm using the traditional method L1 or L2 loss.

The calculator 170 may calculate the similarity between the plurality of images by using the matching number of feature coordinates. According to the calculator 170 , the degree of similarity may be determined according to the number of matching feature coordinates. In addition, since the feature coordinates that match each other are identified, it is easy to visually indicate which feature coordinates match each other in each image. Through this, the user can visually determine the matching state between the objects in the two images.

2 is a schematic diagram showing the operation of the calculation device.

The acquisition unit 110 may acquire a plurality of images. In this case, the plurality of images may be irrespective of the type. For example, the acquisition unit 110 may acquire a first image and a second image, and both the first image and the second image may be photos. Alternatively, both the first image and the second image may be drawings. Alternatively, the first image may be a photograph and the second image may be a drawing.

2 shows a case in which the first image (the image on the right) is a photograph and the second image (the image on the left) is a drawing. As a result, in the embodiment of FIG. 2 , the acquisition unit 110 acquires a photo and a drawing.

The extractor 130 may extract a first feature vector of the first object ix included in the photo by using a feature coordinate extraction algorithm.

The extractor 130 may extract a second feature vector of the second object iy included in the drawing by using a feature coordinate extraction algorithm.

The matching unit 150 may match the first feature vector and the second feature vector with each other using a coordinate matching algorithm. In the drawing, the first feature coordinates fx included in the first feature vector and the second feature coordinates fy included in the second feature vector match each other.

The calculator 170 may calculate the similarity between the first object ix of the picture and the second object iy of the drawing according to the number of matches that are normally matched.

The function and action of each element will be described in more detail.

The acquisition unit 110 may acquire a first image in which the first object ix exists, and may acquire a second image in which the second object iy exists.

The acquisition unit 110 may convert a photograph of an object and a drawing of the object for estimating the similarity into a gray scale format of an 8-bit integer format.

The extractor 130 may extract the first characteristic coordinates of the first object ix and the first characteristic coordinates fx corresponding to the coordinates of the first intrinsic characteristic. The first intrinsic feature may include direction and length components of the first feature vector. The first feature coordinates may include coordinates of a start point or an end point of the first feature vector. Alternatively, the first feature coordinates may include coordinates of an arbitrary point on a line segment connecting the start point and the end point in the first feature vector.

The extractor 130 may extract a second characteristic coordinate of the second object iy and a second characteristic coordinate fy corresponding to the coordinates of the second intrinsic characteristic.

As an example, the extractor 130 may extract a feature vector and coordinates from the image transmitted from the obtainer 110 . The format of the extracted data is a 32-bit floating array, and the shape is (Number of key points, X, Y, Feature dimension). Here, X and Y are the X and Y coordinates of each feature, and the 'Feature dimension' is a feature vector expressing a unique feature in the corresponding coordinate, and may be the same as the 'Descriptor dimension', a term used in a general feature extractor.

Each output array obtained through feature extraction may be grouped into a pair and input to the matching unit 150 equipped with a feature matching algorithm. The matching unit 150 finds one-to-one matching coordinates for each feature through a corresponding algorithm and outputs a coordinate pair having a 'Confidence score' higher than 0.2. Here, the format of the output is a 32-bit floating array, and the format may be (Number of matching key points, X, Y).

Looking from another perspective, the matching unit 150 may calculate the number of first matches corresponding to the number of first feature coordinates fx that coincide with the second feature coordinate fy while matching the second unique feature in the first image.

The matching unit 150 may calculate the number of second matches corresponding to the number of second feature coordinates fy that coincide with the first feature coordinates fx while matching the first unique feature in the second image. In this case, the first matching number and the second matching number may be equal to each other. In this case, it is sufficient if the matching unit 150 calculates the matching number Nmatch corresponding to the number in which the first characteristic coordinates fx of the first object ix and the second characteristic coordinates fy of the second object iy coincide.

The calculator 170 may calculate the similarity between the first object ix and the second object iy by using at least one of the first matching number and the second matching number.

As an example, the matching unit 150 may calculate the first total number Nx corresponding to the total number of the first feature coordinates fx. The matching unit 150 may calculate a second total number Ny corresponding to the total number of second feature coordinates fy.

, 제2 전체 개수 Ny(y의 키 포인트(key point) 수)와 매칭 개수 Nmatch 간의 제2 비율

을 이용하여 제1 물체 ix와 제2 물체 iy 간의 유사도를 산정할 수 있다.The calculation unit 170 is a first ratio between the first total number Nx (the number of key points of x) and the number of matching Nmatch (the final number of key points matched one-to-one through a feature matching algorithm)

, a second ratio between the second total number Ny (the number of key points in y) and the number of matches Nmatch

can be used to calculate the similarity between the first object ix and the second object iy.

The calculator 170 may output a value obtained by adding the first ratio and the second ratio to the degree of similarity between the first object and the second object. In this case, the calculator 170 may multiply the smaller value of the first ratio and the second ratio by the first weight. The calculator 170 may multiply the larger value of the first ratio and the second ratio by the second weight. The first weight may be smaller than the second weight. The first weight may be selected within a range greater than 0 and less than 1. The second weight may be selected within a range greater than 1 and less than 2.

Summarizing the above, the calculation unit 170 may calculate the degree of similarity by Equation 1 below.

...수학식1

...Equation 1

은 제1 비율과 제2 비율 중 작은 값을 나타낼 수 있다.

은 제1 비율과 제2 비율 중 큰 값을 나타낼 수 있다.The similarity (x,y) represents the similarity between the first object ix and the second object iy. Each of x and y may be (Number of Key point, X, Y, Feature dimension) information extracted from the image. 0.5 may be the first weight. 1.5 may be the second weight.

may represent a smaller value of the first ratio and the second ratio.

may represent a larger value among the first ratio and the second ratio.

In the case of Figure 2, the object is a wrist watch. The second total number Ny is 431, the first total number Nx is 757, and the matching number Nmatch is 187. According to Equation 1, the similarity between the two images is 0.77.

3 is a schematic diagram illustrating an operation of a calculation device for another object.

3 , the object is a hand fan. The second total number Ny targeting the second object iy is 506, the first total number Nx targeting the first object iy is 674, and the matching number Nmatch is 176. Therefore, the similarity between the two images is 0.65.

4 is a schematic diagram illustrating an operation of a calculation device targeting another object.

4 , the objects are a hand fan and a cosmetic container. The second total number Ny for the second object iy corresponding to the hand fan is 506, the first total number Nx for the first object ix corresponding to the cosmetic container is 566, and the matching number Nmatch is 2. Therefore, the similarity between the two images is 0.001.

According to the matching unit 150 , the first characteristic coordinate fx and the second characteristic coordinate fy are specified, and a line segment connecting the two may be generated. Each feature coordinate displayed as a point along with the degree of similarity calculated by the calculator 170 , and a line segment connecting each feature coordinate may be provided to the user terminal by the matching part 150 or the calculating part 170 . The user may be provided with a screen (image) in which each feature coordinate is displayed as a point and each point is connected by a line segment through the terminal as shown in FIGS. 2, 3, and 4 . Through this, the user can visually confirm the matching result of each feature coordinate.

Also, according to the present invention, similarity calculation between different heterogeneous images can be accurately performed. According to the present invention, the similarity between a drawing, such as a design right, and a photograph of an actual article can be obtained numerically, and the user can visually interpret the numerically obtained similarity and matching result.

5 is a flowchart illustrating a calculation method according to the present invention.

The calculation method shown in FIG. 5 may be performed by the calculation device of FIG. 1 .

The calculation method may include an obtaining step (S510), an extraction step (S520), a matching step (S530), and a calculating step (S540).

In the acquiring step ( S510 ), an image in which an object exists may be acquired. The acquiring step ( S510 ) may be performed by the acquiring unit 110 .

The extraction step ( S520 ) may extract a feature vector of the object. The extraction step ( S520 ) may be performed by the extraction unit 130 .

In the matching step ( S530 ), feature vectors extracted from a plurality of images may be matched with each other. The matching step ( S530 ) may be performed by the matching unit 150 .

In the calculating step ( S540 ), the similarity between the plurality of images may be calculated using the number of matching feature vectors. The calculation step ( S540 ) may be performed by the calculation unit 170 .

6 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 6 may be a device (eg, a calculation device, etc.) described herein.

In the embodiment of FIG. 6 , the computing device TN100 may include at least one processor TN110 , a transceiver device TN120 , and a memory TN130 . In addition, the computing device TN100 may further include a storage device TN140 , an input interface device TN150 , an output interface device TN160 , and the like. Components included in the computing device TN100 may be connected by a bus TN170 to communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to an embodiment of the present invention are performed. The processor TN110 may be configured to implement procedures, functions, and methods described in connection with an embodiment of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may be configured as at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory TN130 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver TN120 may transmit or receive a wired signal or a wireless signal. The transceiver TN120 may be connected to a network to perform communication.

On the other hand, the embodiment of the present invention is not implemented only through the apparatus and/or method described so far, and a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded may be implemented. And, such an implementation can be easily implemented by those skilled in the art from the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also presented. It belongs to the scope of the invention.

110...Acquisition Department 130...Extraction Department
150...Matching Department 170...Calculating Department

Claims (8)

an acquisition unit that acquires an image in which an object exists;
an extraction unit for extracting feature coordinates of the object;
a matching unit for matching feature coordinates extracted from a plurality of images with each other;
Including; a calculation unit for calculating the similarity between a plurality of images by using the matching number of the feature coordinates;
Matching of the feature coordinates performed by the matching unit means determining whether the intrinsic features of the object included in each image match each other, or determining whether the feature coordinates corresponding to the coordinates of the intrinsic feature match each other Calculation device which means or means to bind feature coordinates of each image that match each other as a pair in a state where the intrinsic features of objects included in each image coincide with each other.
According to claim 1,
The acquisition unit acquires a first image in which a first object exists, and a second image in which a second object exists,
The extraction unit extracts a first characteristic coordinate of the first object and a first characteristic coordinate corresponding to the coordinate of the first characteristic characteristic,
The extraction unit extracts a second characteristic coordinate of the second object and a second characteristic coordinate corresponding to a coordinate of the second characteristic characteristic,
The matching unit calculates a first matching number corresponding to the number of first characteristic coordinates that coincide with the second characteristic coordinates while matching the second unique characteristic in the first image,
The matching unit calculates a second matching number corresponding to the number of second characteristic coordinates that coincide with the first characteristic coordinates while matching the first unique characteristic in the second image,
The calculator calculates the similarity between the first object and the second object by using at least one of the first matching number and the second matching number.
The method of claim 1,
The acquisition unit acquires a first image in which a first object exists, and a second image in which a second object exists,
The matching unit calculates a matching number corresponding to the number in which the first characteristic coordinates of the first object and the second characteristic coordinates of the second object match,
The matching unit calculates a first total number corresponding to the total number of the first feature coordinates,
The matching unit calculates a second total number corresponding to the total number of the second feature coordinates,
The matching unit calculates the similarity between the first object and the second object by using a first ratio between the first total number and the matching number and a second ratio between the second total number and the matching number.
4. The method of claim 3,
The calculator outputs a value obtained by adding the first ratio and the second ratio to the degree of similarity between the first object and the second object.
5. The method of claim 4,
The calculator multiplies a smaller value of the first ratio and the second ratio by a first weight;
The calculation unit multiplies a second weight by a larger value of the first ratio and the second ratio,
The first weight is a value smaller than the second weight.
5. The method of claim 4,
The calculator multiplies a smaller value of the first ratio and the second ratio by a first weight;
The calculation unit multiplies a second weight by a larger value of the first ratio and the second ratio,
The first weight is selected within a range greater than 0 and less than 1,
and the second weight is selected within a range greater than 1 and less than 2.
delete A calculation method performed by a calculation device, comprising:
an acquisition step of acquiring an image in which an object exists;
an extraction step of extracting feature coordinates of the object;
a matching step of matching feature coordinates extracted from a plurality of images with each other;
a calculation step of calculating the degree of similarity between a plurality of images by using the matching number of the feature coordinates;
Matching of the feature coordinates performed in the matching step means determining whether the unique features of objects included in each image match each other, or determining whether the feature coordinates corresponding to the coordinates of the unique features match each other Or, in a state in which the intrinsic features of objects included in each image coincide with each other, the calculation device means tying the coordinates of the features of each image that match each other as a pair.

Images