JP7417504B2 - Similar image difference extraction device, similar image difference extraction method, program and recording medium - Google Patents

Description

The present disclosure relates to a similar image difference extraction device.

In recent years, similar image search technology that searches for similar images similar to an image specified by a user has been attracting attention. For example, in fields such as the manufacturing industry, when designing a product, you can search for drawings similar to the design drawing and refer to drawings designed in the past, improving the efficiency of product design and estimates. It becomes possible to aim for.

Furthermore, in similar image search technology, in order to further improve efficiency, a technology that allows users to intuitively grasp the difference between a designated image and a searched similar image is desired.

On the other hand, in Patent Document 1, predetermined binary descriptors are calculated for each of two images, and based on these binary descriptors, the position, size, angle, distortion, etc. of objects in each image are calculated. An image registration method is disclosed that corrects differences and transforms each image to facilitate comparison.

Japanese Patent Application Publication No. 2018-28899

However, in Patent Document 1, the two images are simply transformed to make them easier to compare, so the user must visually compare them to identify the difference, and the efficiency of understanding the difference is not sufficient. I can't say that. Furthermore, Patent Document 1 also has a problem in that the user has to specify the type of binary descriptor depending on the image.

An object of the present disclosure is to provide a similar image difference extraction device, a similar image difference extraction method, a program, and a recording medium that can efficiently grasp differences between images.

A similar image difference extraction device according to one aspect of the present disclosure includes a feature extraction unit that generates a plurality of target feature maps that are feature maps indicating characteristics of each of a plurality of target images, and a similarity degree for each position of each target feature map. and an image processing section that generates a difference extraction image according to the similarity map.

According to the present invention, it is possible to efficiently understand differences between images.

FIG. 1 is a diagram illustrating a similar image difference extraction system according to an embodiment of the present disclosure. FIG. 3 is a diagram showing an example of an image registration model. FIG. 3 is a diagram showing an example of an image feature extraction model. FIG. 3 is a diagram showing an example of a conversion parameter regression model. FIG. 3 is a diagram for explaining an example of the operation of an image registration section. 3 is a flowchart for explaining an example of the operation of an image registration section. FIG. 3 is a diagram showing an example of an image difference extraction model. 7 is a flowchart for explaining an example of the operation of an image difference extraction section. It is a figure which shows the flow of an image difference extraction part. 2 is a flowchart for explaining an example of the operation of the entire similar image difference extraction system. It is a figure showing an example of a similar search result. FIG. 3 is a diagram showing an example of the result of image registration processing. FIG. 3 is a diagram illustrating an example of the results of image difference extraction processing. 7 is a flowchart for explaining an example of divided image difference extraction processing. FIG. 7 is a diagram illustrating an example of a result of divided image difference extraction processing. 3 is a flowchart for explaining an example of learning processing. 12 is a flowchart for explaining another example of learning processing.

Embodiments of the present disclosure will be described below with reference to the drawings.

<System configuration>
FIG. 1 is a diagram illustrating a similar image difference extraction system according to an embodiment of the present disclosure. As shown in FIG. 1, the similar image difference extraction system includes a user terminal 100 and an image processing server 104. The user terminal 100 and the image processing server 104 are connected to each other via a network 111 so that they can communicate with each other.

<User terminal>
The user terminal 100 includes an input/output section 101, a terminal management section 102, and a network section 103. The user terminal 100 can be configured with, for example, a general-purpose PC (Personal Computer).

<Input/output section>
The input/output unit 101 includes, for example, a user interface device such as a mouse, a keyboard, and a display. The input/output unit 101 has a function of receiving image processing commands from a user and a function of displaying response information according to the image processing commands. The response information is displayed using an application program such as a browser.

<Terminal management department>
The terminal management unit 102 has a function of transmitting image processing commands received by the input/output unit 101 to the image processing server 104 via the network unit 103, and a function of transmitting response information from the image processing server 104 via the network unit 103. and a receiving function. Here, the image processing command is transmitted and the response information is received using an application program such as a browser using a protocol such as HTTP, but other methods may be used. Note that the network unit 103 is a communication unit that has a communication function to communicably connect to the image processing server 104.

In this embodiment, the above image processing commands include an image search command, an image registration command, a difference extraction command, and a learning execution command.

<Image search command>
The image search command is a command that instructs execution of image search processing, which is a search for similar images similar to a query image specified by the user. In the image search process, a plurality of similar images may be searched.

<Image registration command>
The image registration command is a command that instructs execution of image registration processing on a plurality of specified images specified from a group of images including the query image and similar images. Image registration will be described later. In this embodiment, two specified images are illustrated as the plurality of specified images, but the present invention is also applicable to a case where there are three or more specified images.

<Difference extraction command>
The difference extraction command is a command that instructs execution of image difference extraction processing that extracts and visualizes differences between a plurality of target images. In this embodiment, two target images are illustrated as the plurality of target images, but the present invention is also applicable to a case where there are three or more target images. Furthermore, the two target images are two specified images on which image registration processing has been performed. The similar image difference extraction system has a continuous mode in which image registration commands and difference extraction commands are automatically executed in succession, and an individual mode in which these commands are executed independently according to user instructions. , the configuration may be such that the user can set any of these modes. In this embodiment, it is assumed that the individual mode is set.

<Learning execution command>
The learning execution command is a command that instructs execution of learning processing to construct a learning model used in image registration processing and image difference extraction processing. The learning model will be described later.

<Image processing server>
The image processing server 104 is a similar image difference extraction device that performs various processes according to image processing commands from the user terminal 100, and includes a server management section 105, an image search section 106, an image registration section 107, and an image difference extraction device. It has a section 108, a network section 109, and an image database 110. Note that the processing performed by each part of the image processing server 104 described below may be realized by a processor such as a CPU (Central Processing Unit) executing a program. Further, the program can also be recorded on a recording medium 112 that non-temporarily stores data, such as a semiconductor memory, a magnetic disk, an optical disk, a magnetic tape, and a magneto-optical disk. Note that although the recording medium 112 is provided separately from the image processing server 104 in FIG. 1, it may be provided within the image processing server 104.

<Server Management Department>
The server management unit 105 receives an image processing command from the user terminal 100 via the network unit 109, and performs various processing on the image search unit 106, image registration unit 107, and image difference extraction unit 108 according to the image processing command. It has the function to execute. The server management unit 105 also has a function of transmitting the processing results of the image search unit 106, image registration unit 107, and image difference extraction unit 108 to the user terminal 100 via the network unit 109 as response information. The server management unit 105 also has a function as a learning management unit that executes learning processing and constructs a learning model.

<Image search section>
The image search unit 106 has a function of executing an image search process of searching the image database 110 for a similar image similar to a query image specified by the user in accordance with an image search command. The image database 110 stores a plurality of images in advance. Note that the image database 110 may be installed in a storage device or the like separate from the image processing server 104. Furthermore, since there are various known techniques for specific methods of image search processing, detailed explanation thereof will be omitted.

<Image registration section>
The image registration unit 107 has a function of performing image registration processing on two specified images specified from a group of images including the query image and the similar images searched by the image search unit 106. Image registration processing estimates a geometric transformation model that exists between two images, and uses that transformation model to correct geometric differences such as size, position, angle, and distortion between each image. This is the process of By performing image registration processing, it becomes easy to extract the difference between two images.

In this embodiment, the image registration unit 107 does not estimate a conversion model using feature calculation and mapping using general image processing, but uses a learning model (specifically, a learning model using a neural network). The conversion model is estimated using the learning model. Hereinafter, the learning model used by the image registration unit 107 may be referred to as an image registration model.

<Image difference extraction section>
The image difference extraction unit 108 has a function of executing an image difference extraction process that extracts and visualizes the difference between the two specified images, which have been subjected to the image registration process by the image registration unit 107, as target images. has. In this embodiment, the image difference extraction unit 108, like the image registration unit 107, executes image difference extraction processing using a machine learning model using a neural network as a learning model. Hereinafter, the machine learning model used by the image difference extraction unit 108 may be referred to as an image difference extraction model.

The image registration unit 107 will be described in more detail below.

<Image registration model>
FIG. 2 is a diagram showing an example of an image registration model used by the image registration unit 107. The image registration model 200 shown in FIG. 2 includes image feature extraction models 201a and 201b, a conversion parameter regression model 202, and a feature relationship map calculation unit 203. The image registration unit 107 also includes an execution unit 204 that actually executes image registration processing based on the processing result of the image registration model.

The image registration model 200 receives two designated images as input and estimates transformation parameters that are parameters of a transformation model for performing image registration processing on the two designated images. The transformation model is, for example, an affine transformation model or a homography transformation model. In image registration processing, it is possible to correct geometric differences between two specified images by performing transformations such as scaling, rotation, and translation of images using a transformation model. Note that one of the two specified images is input to the image feature extraction model 201a, and the other is input to the image feature extraction model 201a.

<Image feature extraction model>
FIG. 3 is a diagram showing an example of an image feature extraction model. The image feature extraction models 201a and 201b constitute a designated feature extraction unit that generates designated feature maps, which are two feature maps indicating the respective features of the two input designated images. Furthermore, the image feature extraction models 201a and 201b are the same model, and are shown as an image feature extraction model 300 in FIG.

The image feature extraction model 300 shown in FIG. 3 is a learning model for calculating, from an input designated image, a designated feature map indicating the characteristics of the designated image. The image feature extraction model 300 can be realized by, for example, a neural network such as a CNN (Convolutional Neural Network). Here, it is assumed that the specified image is represented by a third-order tensor including a width component (horizontal pixel component), a height component (vertical pixel component), and a color component. In addition, the position indicated by the width component and the height component is the pixel position, the width (the number of width components) and the height width (the number of the height components) are the image size, the color component c is the channel, and the number of color components is the channel. Sometimes called numbers.

The image feature extraction model 300 includes a plurality of processing layers 301 including a convolution layer (denoted as Conv in the figure) 302 and a pooling layer (denoted as Pool in the figure) 303. The plurality of processing layers 301 are connected to each other in series. The convolution layer 302 of each processing layer 301 performs convolution processing on the input tensor, and the pooling layer 303 performs pooling processing to reduce the image size on the output tensor of the previous convolution layer 302, and It is output from the processing layer 301 as a feature map showing. The feature map output from the processing layer 301 is also a tensor. Here, it is assumed that the input tensor is outputted in each processing layer 301 with twice the number of channels as before input, and with an image size 1/2 times as large as before input, but the invention is not limited to this example. However, in the processing layer 301 at the forefront, the number of channels is not twice the number before input.

Note that each processing layer 301 may include a plurality of convolutional layers 302 and perform convolution processing multiple times, but here, it is assumed that there is only one convolutional layer 302. Although two processing layers 301 are shown in FIG. 3, in this embodiment, there are four processing layers 301. However, the number of processing layers 301 is not limited to four. Further, after the convolution process, post-processing such as batch normalization process or activation function (eg, ramp function) process may be performed, similar to a general neural network.

<Feature relevance map calculation unit>
The feature relevance map calculation unit 203 in FIG. 2 calculates the inner product of the designated feature maps output from each of the image feature extraction models 201a and 201b, and outputs it as a feature relevance map indicating the relevance between the designated feature maps. This is an arithmetic unit that performs

<Conversion parameter regression model>
FIG. 4 is a diagram showing an example of the conversion parameter regression model 202. The conversion parameter regression model 202 is a model that estimates conversion parameters, which are parameters of the conversion model, based on the feature relevance map output from the feature relevance map calculation unit 203.

数式（１）は、画像の（Ｘ，Ｙ）座標の点がアフィン変換により（Ｘ’、Ｙ’）の点に変換されることを示す。また、数（１）のａ，ｂ，ｔ_ｘ，ｃ，ｄ及びｔ_ｙが変換パラメータである。なお、推定する変換パラメータを変更することでホモグラフィ変換モデルなどの他の変換モデルに適用することができる。 Here, the transformation model is an affine transformation model that performs affine transformation. An affine transformation is a combination of translation and linear transformation. A linear transformation is a transformation that includes scaling, shearing, and rotating an image. When a homogeneous coordinate system is used, the transformation matrix of affine transformation is expressed by the following equation (1).

Equation (1) indicates that a point at coordinates (X, Y) of an image is transformed into a point at (X', Y') by affine transformation. Further, a, b, t _x , c, d, and _ty in number (1) are conversion parameters. Note that by changing the estimated transformation parameters, the method can be applied to other transformation models such as the homography transformation model.

The conversion parameter regression model 400 includes a processing layer 401 including a convolution layer (denoted as Conv in the diagram) 402 and a pooling layer (denoted as Pool in the diagram) 403, and an FC (Fully Connected) layer 404. There are a plurality of processing layers 401, and these processing layers 401 are directly connected to each other. The FC layer 404 is provided as the final layer after the last processing layer 401.

In each processing layer 401, a convolution layer 402 performs convolution processing on the input tensor, and a pooling layer 403 performs pooling processing on the output tensor of the convolution layer 402, and outputs the result as a feature map. Here, it is assumed that the input tensor is outputted in each processing layer 401 with twice the number of channels as before input and with an image size 1/2 times as large as before input, but this is not limited to this example. However, in the processing layer 401 at the forefront, the number of channels is not twice that before input.

Note that each processing layer 401 may include a plurality of convolutional layers 402 and perform convolution processing multiple times, but here, it is assumed that there is only one convolutional layer 402. Although two processing layers 401 are shown in FIG. 4, in this embodiment, there are four processing layers 401. However, the number of processing layers 401 is not limited to four. Furthermore, the above-described post-processing may be performed after the convolution processing.

The FC layer 404 converts the feature map into a one-dimensional output vector by performing full connection processing on the feature map output from the processing layer 401 at the last stage, and outputs the resultant one-dimensional output vector. Each element of the output vector corresponds to a transformation parameter of the transformation model. Therefore, the number of elements of the output vector is the same as the number of transformation parameters of the transformation model, and here, since an affine transformation model is used as the transformation model, the number of elements of the output vector is six.

The conversion parameter regression model 202 and feature relationship map calculation unit 203 described above constitute a parameter estimation unit that estimates conversion parameters based on the designated feature map generated by the image feature extraction model.

<Execution part>
The execution unit 204 performs image registration processing by performing affine transformation on either of the two designated images based on the transformation parameters corresponding to the elements of the output vector from the transformation parameter regression model 202. .

<Operation of image registration section>
FIG. 5 is a diagram for explaining an example of the operation of the image registration section 107, and FIG. 6 is a flowchart for explaining an example of the operation of the image registration section 107.

First, the image registration unit 107 receives two specified images 501a and 501b specified by the user, inputs the specified image 501a into the image feature extraction model 201a, and inputs the specified image 501b into the image feature extraction model 201b. (Step S600). It is assumed that designated images 501a and 501b are similar to each other. "w", "h", and "c" in FIG. 5 represent the horizontal width, vertical width, and number of channels of the designated images 501a and 501b, respectively.

Subsequently, the image feature extraction models 201a and 201b repeatedly perform convolution processing by the convolution layer 302 and pooling processing by the pooling layer 303 on the designated images 501a and 501b four times to generate designated feature maps 502a and 502b. (Step S601). In FIG. 5, rectangular parallelepipeds 512a and 512b represent a process in which each processing layer 301 doubles the number of channels of designated images 501a and 501b while reducing the image size by half.

Then, the feature relevance map calculation unit 203 performs inner product processing to calculate the inner product of the designated feature maps 502a and 502b output from the image feature extraction models 201a and 201b as the feature relevance map 503 (step S602).

Here, the specified feature map 502a is f _a , the specified feature map 502b is f _b , and the indices representing the pixel positions of the specified feature maps f _a and f _b are (i, j) and (i _k , j _k ),
f _b (i, j) is the feature vector (vector consisting of color components) of the (i, j) component of the specified feature map f _b , f _a (i _k , j _k ) is the ( _{i k} , j _k ) components, c _ab (i, j, k), which is the feature association map 503, can be calculated from the following equation (2).

In Equation (2), " ^T " indicates the transposition of the vector, and "." indicates the inner product of the vector. Therefore, in the feature relationship map c _ab (i, j, k), the feature vector of the (i, j) component of the specified feature map f _b and _the (i _k , j _k ) of the specified feature map fa are expressed as This shows that the inner product of the pixel position with the feature vector becomes the (i, j, k) component. In other words, the feature vector of the (i, j) component of the feature relationship map c _ab (i, j, k) is the pixel area of the specified image 501a to which the (i, j) component of the specified feature map f _b corresponds. indicates the relationship with the entire designated image 502b.

For example, if the specified image 501a is an RGB image with dimensions (w, h, c) = (240, 240, 3), in the image feature extraction model 201a, the image size of the feature map is changed by the four processing layers 301 to the original size. The image size is 1/16 of the designated image 501a. Further, assuming that the number of channels of the output tensor of the processing layer 301 at the forefront is 64, the number of channels changes from 64 to 128 to 256 to 512 in the four processing layers 301. Therefore, the feature maps f _a and f _b have dimensions (w, h, c) = (15, 15, 512), respectively, and the feature association map c _ab has (15, 15, 15 x 15) = (15 , 15, 225) dimensions.

After calculating the feature relevance map 503, the image registration unit 107 inputs the feature relevance map 503 into the conversion parameter regression model 400 (step S603). The conversion parameter regression model 400 repeatedly executes the convolution process by the convolution layer 402 and the pooling process by the pooling layer 403 on the input feature relationship map 503 four times, generates an image feature map 504, and further calculates the image features. A full connection process 510 is performed on the map 504 by the FC layer 404 to generate six transformation parameters 505 (step S604). Note that the processing from steps S600 to S604 is image registration model processing.

Further, the execution unit 204 executes the image registration process by performing affine transformation on either of the images 501a and 501b based on the six transformation parameters 505 (step S605), and ends the process.

Next, the image difference extraction process by the image difference extraction unit 108 will be described in more detail.

<Image difference extraction model>
FIG. 7 is a diagram showing an example of an image difference extraction model used by the image difference extraction unit 108. The image difference extraction model 700 shown in FIG. 7 includes image feature extraction models 701a and 701b, a similarity calculation section 702, and an image processing section 703. The image feature extraction models 701a and 701b are the same models as the image feature extraction models 201a and 201b, which are image registration models used by the image registration unit 107, and are feature maps indicating the respective features of the two input target images. A feature extraction unit that generates a target feature map is configured.

The similarity calculation unit 702 generates a similarity map that indicates the similarity for each position (specifically, pixel position) between the two target feature maps calculated by the image feature extraction models 701a and 701b. In this embodiment, the similarity is a cosine similarity.

The image processing unit 703 performs image processing to generate a difference extraction image according to the similarity map calculated by the similarity calculation unit 702. Here, the image processing is a heat mapping process that generates a heat map image representing each value of the similarity map by color or shading as a difference extraction image. Further, the image processing may include processing to enlarge the difference extracted image to the same size as the target image.

<Operation of image difference extraction unit>

FIG. 8 is a diagram for explaining an example of the operation of the image difference extraction unit 108, and FIG. 9 is a flowchart for explaining an example of the operation of the image difference extraction unit 108.

First, the image difference extraction unit 108 receives two target images 801a and 801b specified by the user, inputs the target image 801a into the image feature extraction model 701a, and inputs the target image 801b into the image feature extraction model 701b. (Step S900). The target images 801a and 801b are designated images that have been subjected to image registration processing by the image registration unit 107.

Subsequently, the image feature extraction models 701a and 701b repeatedly perform convolution processing by the convolution layer 402 and pooling processing by the pooling layer 403 on the target images 801a and 801b four times to generate target feature maps 802a and 802b. (Step S901). In FIG. 5, rectangular parallelepipeds 812a and 812b represent a process in which each processing layer 301 doubles the number of channels while reducing the size of the target images 801a and 801b by half.

The similarity calculation unit 702 performs similarity calculation processing to calculate a similarity map 803 indicating the similarity between the target feature maps 802a and 802b calculated by the image feature extraction models 701a and 701b (step S902).

Here, the target feature map 802a is g _a , the target feature map 802b is g _b , and the indices representing the pixel positions of the target feature maps g _a and g _b are (i, j) and (i _k , j _k ), If the feature vectors of the (i, j) components of the target feature maps g _a and g _b are g _a (i, j) and g _b (i, j), the similarity map 803 C _sim (i, j) can be calculated from the following equation (3).

In equation (3), " ^T " indicates the transpose of the vector, "." indicates the inner product of the vector, and "||" indicates the Euclidean distance of the vector. Therefore, the similarity map C _sim (i, j) shows the cosine similarity of the target feature maps g _a and g _b for each pixel position (i, j). Note that since the cosine similarity is a scalar value, the similarity map is a one-channel tensor (image). Further, cosine similarity is an index indicating how much two vectors point in the same direction, and takes a value from -1 to 1. The closer the cosine similarity value is to 1, the more similar the two vectors are, and the closer the value is to -1, the less similar the two vectors are.

After calculating the similarity map, the image processing unit 703 performs heat mapping processing on the similarity map 803 to convert the similarity map 803 into a heat map image 804 (step S903). Here, it is assumed that the heat map image is an RGB image that becomes redder as the value of the similarity map approaches 1 and becomes bluer as it approaches -1, but the heat map image is not limited to this example.

The image processing unit 703 then enlarges the heat map image 804 to the same size as the target image (step S904), and ends the process.

<Overall flow>
FIG. 10 is a flowchart for explaining an example of the operation of the entire similar image difference extraction system of this embodiment. 11 to 13 are diagrams showing specific examples of images handled by the similar image difference extraction system. Specifically, FIG. 11 is a diagram showing an example of the result of image search processing, FIG. 12 is a diagram showing an example of the result of image registration processing, and FIG. 13 is a diagram showing an example of the result of image difference extraction processing. It is a figure showing an example.

First, the input/output unit 101 of the user terminal 100 receives an operation from the user to select a query image to be searched for (step S1000). Here, it is assumed that image 1100 shown in FIG. 11 is selected as the query image.

Subsequently, the input/output unit 101 receives an image search command from the user. The terminal management unit 102 adds information indicating a query image to the image search command, and transmits the image search command to the image processing server 104 via the network unit 103 (step S1001).

The server management unit 105 of the image processing server 104 receives the image search command via the network unit 109, and instructs the image search unit 106 to execute image search processing in accordance with the image search command. The image search unit 106 searches the image database 110 for similar images similar to the query image based on the image search command. The server management unit 105 transmits the searched similar images as a search result to the user terminal 100 via the network unit 109 (step S1002).

Here, it is assumed that images 1102 to 1105 included in the similar image group 1101 shown in FIG. 11 are searched as similar images. Image 1102 has a positional shift with respect to image 1100 which is a query image, image 1103 has a different size from image 1100, and image 1104 has a different angle from image 1100. Image 1105 also matches image 1100 with respect to geometrical characteristics such as position, size, and angle.

The terminal management unit 102 of the user terminal 100 receives the search results via the network unit 103 and displays the search results on the input/output unit 101. The input/output unit 101 receives an operation from the user to designate two images to be compared with each other from among the similar images included in the search results and the query image (step S1003). Here, it is assumed that image 1100, which is a query image, and image 1102, which is a similar image, are specified. Note that two similar images may be specified in this operation.

Subsequently, the input/output unit 101 receives an image registration command from the user. The terminal management unit 102 adds information indicating the two designated images to the image registration command, and transmits the image search command to the image processing server 104 via the network unit 103 (step S1004).

The server management unit 105 of the image processing server 104 receives the image registration command via the network unit 109, and instructs the image registration unit 107 to execute image registration processing in accordance with the image registration command. The image registration unit 107 executes image registration processing based on the image registration command. The server management unit 105 transmits the registration result, which is the result of the image registration process, to the user terminal 100 via the network unit 109 (step S1005). The image registration process is specifically the process described using FIGS. 5 and 6. Here, it is assumed that affine transformation is performed on image 1100 of the two images 1100 and 1102 shown in FIG. 12, and image 1200 is obtained as a registration result.

Thereafter, the terminal management unit 102 of the user terminal 100 receives the registration result via the network unit 103 and displays the registration result on the input/output unit 101. The input/output unit 101 receives a difference extraction command from a user who has confirmed the registration result. The terminal management unit 102 adds information indicating images 1100 and 1200 as target images to the difference extraction command, and transmits the difference extraction command to the image processing server 104 via the network unit 103 (S1006).

The server management unit 105 of the image processing server 104 receives the difference extraction command via the network unit 109, and instructs the image difference extraction unit 108 to execute image difference extraction processing in accordance with the difference extraction command. The image difference extraction unit 108 executes image difference extraction processing on the two target images based on the difference extraction command to generate a heat map image (step S1007). The image difference extraction process is specifically the process described using FIGS. 7 and 8. Here, it is assumed that image difference extraction processing is performed on the two images 1100 and 1200 shown in FIG. 13, and image 1300 is generated as a heat map image.

The server management unit 105 transmits the heat map image, which is the processing result of the image difference extraction process, to the user terminal 100 via the network unit 109. The terminal management unit 102 of the user terminal 100 receives the heat map image via the network unit 103, displays the heat map image on the input/output unit 101 (step S1008), and ends the process. By checking the heat map images, the user can check which parts of the two images differ and to what extent.

<Divided image difference extraction processing>
Next, divided image difference extraction processing, which is a modification of the image difference extraction processing performed by the image difference extraction unit 108, will be described. FIG. 14 is a flowchart for explaining an example of divided image difference extraction processing. FIG. 15 is a diagram illustrating an example of the results of the divided image difference extraction process.

In the divided image difference extraction process, first, the processes of steps S1000 to S1005 in FIG. 10 are executed. Subsequently, the terminal management section 102 of the user terminal 100 receives the registration result via the network section 103 and displays the registration result on the input/output section 101. The input/output unit 101 receives, from the user who has confirmed the registration result, a grid division operation for dividing the target image, which is the designated image on which the image registration process has been performed, into a plurality of partial images into a grid (step S1400). The grid division operation may specify division details (for example, the number of partial images, aspect ratio, division position, etc.).

The input/output unit 101 receives a difference extraction command from a user who has confirmed the registration result. The terminal management unit 102 adds information indicating the target image and the division details according to the grid division operation to the difference extraction command, and transmits the difference extraction command to the image processing server 104 via the network unit 103 (step S1401 ).

The server management unit 105 of the image processing server 104 receives the difference extraction command via the network unit 109, and instructs the image difference extraction unit 108 to execute image difference extraction processing in accordance with the difference extraction command. Based on the difference extraction command, the image difference extraction unit 108 grid-divides the target image into a plurality of partial images according to the specified division details (step S1402). Here, assume that images 1500 and 1502 are divided into grids as target images, as shown in FIG. Note that although grid lines 1501 are shown in FIG. 15 for the sake of explanation, it is not actually necessary to draw the grid lines 1501 on the image.

The image difference extraction unit 108 executes a loop process (A) in which steps S1403 to S1404 are repeated for each grid-divided partial image.

In loop processing (A), the image difference extraction unit 108 inputs partial images at mutually corresponding positions of each target image to the feature amount extraction models 701a and 701b of the image difference extraction model 700 (step S1403). The feature extraction models 701a and 701b calculate partial feature maps that are feature maps of the input partial images (step S1404).

After the image difference extraction unit 108 performs steps S1403 to S1404 on all partial images, the process exits from the loop process (A). Then, the similarity calculation unit 702 combines each partial feature map to generate a target feature map so that the positional relationship of each partial feature map is the same as the positional relationship of each partial image corresponding to each partial feature map. (Step S1405).

Subsequently, the similarity calculation unit 702 of the image difference extraction unit 108 performs similarity calculation processing to calculate a similarity map indicating the similarity between each target feature map (step S1406). The image processing unit 703 performs heat mapping processing on the similarity map and converts the similarity map into a heat map image (step S1407). The image processing unit 703 enlarges the heat map image to the same size as the input image (step S1408), and ends the process. As a result, an image 1504 shown in FIG. 15 is generated as a heat map image. Note that after the process in step S1408, the process in step S1008 in FIG. 10 is executed.

<Example of learning process>
FIG. 16 is a flowchart for explaining an example of a learning process performed on the image registration model 200 and the image difference extraction model 700.

In the learning process, first, when the input/output unit 101 receives a learning command from the user, the terminal management unit 102 transmits the learning command to the image processing server 104 via the network unit 103. The server management unit 105 of the image processing server 104 receives the learning command via the network unit 109 (step S1600).

The server management unit 105 executes a loop process (B) in which steps S1601 to S1607 are repeated until a predetermined condition is satisfied. The predetermined conditions include that the number of learning times (the number of loops) reaches a predetermined number of times, or that the learning time reaches a predetermined time.

In loop processing (B), the server management unit 105 acquires an arbitrary image as a first learning image from the image database 110 (step S1601). The image acquired here is called image A.

The server management unit 105 generates a first conversion parameter that is a random conversion parameter (step S1602). The number of first conversion parameters is predetermined according to the conversion model. For example, when the transformation model is an affine transformation model, the number of first transformation parameters is six.

The server management unit 105 converts image A into image B, which is a second learning image, using the conversion model in which the first conversion parameter is set (step S1603). The server management unit 105 inputs images A and B as designated images to the image registration model 200 (step S1604). The image registration model 200 estimates a transformation parameter based on images A and B as a second transformation parameter (step S1605).

The server management unit 105 uses a predetermined loss function to calculate a loss value indicating the difference between the second transformation parameter estimated in step S1605 and the first transformation parameter generated in step S1602 (step S1606). Note that the loss function is, for example, MSE (Mean Square Error).

The server management unit 105 adjusts the weight value of the image registration model 200 so that the loss value decreases (step S1607).

Then, when the predetermined condition is satisfied, the server management unit 105 exits the loop process (B) and ends the process. As a result, the image registration model 200 is constructed as a learning model. By learning the image registration model 200, internal image feature extraction models 201a and 201b are learned. Accordingly, in this embodiment, since the same model as the image feature extraction models 201a and 201b is used as the image feature extraction models 701a and 701b inside the image difference extraction model 700, the image feature extraction models 201a and 201b are also learned. The Rukoto.

<Other examples of learning processing>
FIG. 17 is a flowchart for explaining another example of the learning process performed on the image registration model 200 and the image difference extraction model 700. By using this learning process, it is possible to construct a more accurate model than the learning process shown in FIG. 16.

First, the server management unit 105 of the image processing server 104 registers, in the image database 110, a set of two images that are similar in geometrical characteristics such as position, size, and angle (step S1700). An example of a set of two similar images is the set of image 1100 and image 1105 in FIG. 11. The set of images may be created by the user. Further, the set of images may be generated using an existing program such as a general image processing library. Alternatively, the configuration may be such that the user selects a set of images from images generated using an existing program.

Subsequently, when the input/output unit 101 receives a learning command from the user, the terminal management unit 102 transmits the learning command to the image processing server 104 via the network unit 103. The server management unit 105 of the image processing server 104 receives the learning command via the network unit 109 (step S1701).

The server management unit 105 executes a loop process (C) that repeats the processes of steps S1702 to S1708 for each set of images.

In loop processing (C), the server management unit 105 selects two images included in an arbitrary image set from the image database 110 as a first learning image and a corresponding similar image corresponding to the first learning image. Acquire (S1702). The first learning image acquired here is called image C, and the corresponding similar image is called image D.

The server management unit 105 generates a first conversion parameter that is a random conversion parameter (step S1703).

The server management unit 105 converts the image D into the image D' which is the second learning image using the conversion model in which the first conversion parameter is set (step S1704). The server management unit 105 inputs images C and D' into the image registration model 200 as designated images (step S1705). The image registration model 200 estimates a transformation parameter based on images C and D' as a second transformation parameter (step S1706).

The server management unit 105 uses a predetermined loss function to calculate a loss value indicating the difference between the second transformation parameter estimated in step S1706 and the first transformation parameter generated in step S1703 (step S1707).

The server management unit 105 adjusts the weight value of the image registration model 200 so that the loss value decreases (step S1708).

After executing the processes of steps S1702 to S1708 for all sets of corresponding similar images, the server management unit 105 exits the loop process (C) and ends the process. As a result, the image registration model 200 is constructed as a learning model.

As described above, in this embodiment, the feature extraction unit including the image feature extraction models 701a and 701b generates a target feature map that is a feature map indicating the respective features of two target images. The similarity calculation unit 702 generates a similarity map that indicates the degree of similarity for each position of each target feature map. The image processing unit 703 generates a difference extraction image according to the similarity map.

Therefore, since a difference extraction image is generated according to a similarity map indicating the degree of similarity between the features of the two target images, the user can visually check the difference extraction image without visually comparing the two target images. It becomes possible to understand the difference between the two target images just by Therefore, it becomes possible to efficiently understand the difference between images. Furthermore, since the user does not need to specify the type of binary descriptor for each image, it is possible to efficiently grasp the differences between images from this point of view as well.

Furthermore, in this embodiment, since the target feature map is generated using the image feature extraction models 701a and 701b, which are learning models, it is possible to generate a difference extraction image that appropriately reflects the similarity between the target images. becomes.

Furthermore, in this embodiment, the image registration unit 107 performs image registration processing on two designated images to generate a target image, so a difference extraction image that appropriately reflects the similarity between the target images is generated. It becomes possible to do so.

Furthermore, in this embodiment, since the conversion parameters of the conversion model for image registration processing are estimated based on two specified feature maps, which are feature maps of two specified images, appropriate It becomes possible to estimate the transformation parameters.

Furthermore, in this embodiment, the designated feature map is generated using the image feature extraction models 201a and 201b, which are learning models, so it is possible to estimate transformation parameters that appropriately reflect the characteristics of each designated image. .

Furthermore, in this embodiment, since the image feature extraction models 201a and 201b used by the image registration unit 107 and the image feature extraction models 701a and 701b used by the image difference extraction unit 108 are the same model, efficient learning is possible. It becomes possible.

Further, in this embodiment, the server management unit 105 uses the randomly generated first conversion parameter and the image registration when image A and image B obtained by converting image A using the first conversion parameter are designated images. Image feature extraction models 201a and 201b are constructed based on the loss value indicating the difference from the second transformation parameter estimated by the conversion unit 107. Therefore, it becomes possible to construct highly accurate image feature extraction models 201a and 201b.

Further, in this embodiment, the server management unit 105 uses a randomly generated first conversion parameter, and an image D obtained by converting an image C and an image D that is associated with the image C in advance using the first conversion parameter. Image feature extraction models 201a and 201b are constructed based on a loss value indicating the difference between the second transformation parameter and the second transformation parameter estimated by the image registration unit 107 when the specified image is . Therefore, it becomes possible to construct more accurate image feature extraction models 201a and 201b.

Furthermore, in the present embodiment, the feature extraction unit generates, for each of the two target images, a partial feature map that is a feature map indicating the characteristics of each of the plurality of partial images obtained by dividing the target image, and generates a partial feature map for each of the partial images. A target feature map is generated by combining the partial feature maps so that the positional relationship of the maps is the same as the positional relationship of each partial image corresponding to each partial image. Therefore, a highly accurate target feature map can be generated.

Furthermore, in this embodiment, the difference extracted image is a heat map image in which each value of the similarity map is represented by color or shading, so it is possible to quickly and intuitively grasp the difference between the two images.

The embodiments of the present disclosure described above are examples for explaining the present disclosure, and are not intended to limit the scope of the present disclosure only to those embodiments. Those skilled in the art can implement the present disclosure in various other ways without departing from the scope of the disclosure.

100: User terminal, 101: Input/output unit, 101: Process flow creation server, 102: Terminal management unit, 103: Network unit, 104: Image processing server, 105: Server management unit, 106: Image search unit, 107: Image registration section, 108: Image difference extraction section, 109: Network section, 110: Image database, 111: Network, 112: Memory, 201: Specified feature extraction section, 201a to 201b: Image feature extraction model, 202: Conversion parameter Regression model, 203: Feature relevance map calculation unit, 204: Execution unit, 300: Image feature extraction model, 301: Processing layer, 302: Convolution layer, 303: Pooling layer, 400: Transformation parameter regression model, 401: Processing layer , 402: Convolution layer, 403: Pooling layer, 404: FC layer, 700: Image difference extraction model, 701: Feature extraction unit, 701a to 701b: Image feature extraction model, 702: Similarity calculation unit, 703: Image processing unit

Claims (11)

a feature extraction unit that generates a plurality of target feature maps that are feature maps indicating characteristics of each of the plurality of target images;
a similarity calculation unit that generates a similarity map indicating the similarity for each position of each target feature map;
an image processing unit that generates a difference extraction image according to the similarity map;
an image registration unit that performs image registration processing to correct geometric differences on a plurality of specified images to generate the target image ;
The image registration section includes:
a specified feature extraction unit that generates a plurality of specified feature maps that are feature maps of each of the plurality of specified images;
a parameter estimation unit that estimates transformation parameters of a transformation model for correcting geometric differences between the plurality of images based on each specified feature map;
A similar image difference extraction device , comprising: an execution unit that performs the image registration process using the conversion parameter . The similar image difference extraction device according to claim 1, wherein the feature extraction unit generates the target feature map using a learning model that generates a feature map of images. The similar image difference extraction device according to claim 1 , wherein the specified feature extraction unit generates the specified feature map using a learning model that generates a feature map of images. The feature extraction unit generates the target feature map using a learning model that generates a feature map of an image,
4. The similar image difference extraction device according to claim 3 , wherein the learning model used by the specified feature extraction section and the learning model used by the feature extraction section are the same model. A first transformation parameter that is the randomly generated transformation parameter, a first learning image, and a second learning image obtained by converting the first learning image using the first transformation parameter are combined into the plurality of further comprising a learning management unit that constructs the learning model based on a loss value indicating a difference between the second transformation parameter that is the transformation parameter estimated by the image registration unit when the specified image is The similar image difference extraction device according to claim 3 . A first transformation parameter that is the randomly generated transformation parameter, and a first learning image and a corresponding similar image that is associated in advance with the first learning image are transformed using the first transformation parameter. a second transformation parameter that is the transformation parameter estimated by the image registration unit when the second learning image is the plurality of designated images;
4. The similar image difference extraction device according to claim 3 , further comprising a learning management unit that constructs the learning model based on a loss value indicating a difference in the learning model. The feature extraction unit generates, for each of the plurality of target images, a partial feature map that is a feature map indicating the characteristics of each of a plurality of partial images obtained by dividing the target image, and determines the positional relationship of each partial feature map. The similar image difference extraction device according to claim 1, wherein the target feature map is generated by combining each partial feature map so that the positional relationship of each partial image corresponding to the partial feature map is the same. 2. The similar image difference extraction device according to claim 1, wherein the difference extraction image is a heat map image in which each value of the similarity map is expressed in color or shade. A similar image difference extraction method using a similar image difference extraction device, the method comprising:
Performs image registration processing to correct geometric differences on multiple designated images to generate multiple target images,
Generating a plurality of target feature maps that are feature maps indicating characteristics of each of the plurality of target images,
Generate a similarity map showing the similarity for each position of each target feature map,
Generating a difference extraction image according to the similarity map,
In generating the target image,
generating a plurality of designated feature maps that are feature maps of each of the plurality of designated images;
Based on each designated feature map, estimate transformation parameters of a transformation model for correcting geometric differences between multiple images;
A similar image difference extraction method , wherein the image registration process is performed using the conversion parameter . to the computer,
a step of generating a plurality of target images by performing image registration processing to correct geometric differences on the plurality of specified images;
a step of generating a plurality of target feature maps that are feature maps indicating characteristics of each of the plurality of target images;
a step of generating a similarity map indicating the degree of similarity for each location of each target feature map;
performing a step of generating a difference extraction image according to the similarity map ;
The procedure for generating the target image is as follows:
a step of generating a plurality of designated feature maps that are feature maps of each of the plurality of designated images;
estimating transformation parameters of a transformation model for correcting geometric differences between the plurality of images based on each designated feature map;
A program comprising: performing the image registration process using the conversion parameters . A recording medium recording the program according to claim 10 .

Images