JP7439953B2 - Learning device, processing device, learning method, processing method and program - Google Patents

Description

The present invention relates to a learning device, a processing device, a learning method, a processing method, and a program.

Technologies related to the present invention are disclosed in Non-Patent Document 1 and Patent Documents 1 and 2.

Non-patent Document 1 generates a feature value map, a repeatability map, and a reliability map based on an image, and based on these, characterizes the external appearance of a subject included in the image. discloses a technology (R2D2: Repeatable and Reliable Detector and Descriptor) that extracts key points with high precision. Patent Document 1 discloses a technique for extracting corresponding points between two images with high precision. Patent Document 2 discloses a technique for calculating feature amounts for each pixel and analyzing an image.

JP 2008-220617 International Publication No. 2020/100630

Jerome Revaud, 3 others, "R2D2: Repeatable and Reliable Detector and Descriptor", [online], [searched October 23, 2020], Internet <URL: https://papers.nips.cc/paper/ 9407-r2d2-reliable-and-repeatable-detector-and-descriptor.pdf>

By using the technology described in Non-Patent Document 1, it becomes possible to extract key points of the appearance of a subject included in an image with high precision. However, the key points extracted by the technique described in Non-Patent Document 1 have a problem in that they are susceptible to changes in illumination conditions. Therefore, for example, in an image taken under certain lighting conditions (e.g. daytime), the key points (pixels) can be distinguished from surrounding pixels, but in images taken under different lighting conditions (e.g. nighttime), the key points (pixels) can be distinguished from surrounding pixels. In some cases, a situation may occur in which a key point cannot be distinguished from surrounding pixels. Furthermore, even if the images contain the same subject, if the lighting conditions at the time each image was generated are different (e.g., one was taken during the day and the other at night), the key points extracted from each image will differ from each other. This situation may occur. None of the prior art documents discloses the problem and its solution.

An object of the present invention is to enable extraction of key points that are robust to changes in illumination conditions.

According to the invention,
Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. a storage means for storing the
acquisition means for acquiring a combination of a plurality of learning images that have different lighting conditions and include the same subject;
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. learning means for adjusting parameters of the learning model based on a loss function defined using the learning model;
A learning device having the following is provided.

Further, according to the present invention,
The computer is
Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. Remember,
Obtain a combination of multiple training images that have different lighting conditions and include the same subject,
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. A learning method is provided for adjusting parameters of the learning model based on a loss function defined using the learning model.

Further, according to the present invention,
computer,
Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. storage means for storing
acquisition means for acquiring a combination of a plurality of learning images that have different lighting conditions and include the same subject;
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. learning means for adjusting parameters of the learning model based on a loss function defined using the learning model;
A program is provided to enable this function.

Further, according to the present invention,
A processing device is provided that determines key points of an input image using a learning model generated by the learning device.

Further, according to the present invention,
A processing method is provided in which a computer determines key points of an input image using a learning model generated by the learning device.

Further, according to the present invention,
A program is provided that causes a computer to function as means for determining key points of an input image using a learning model generated by the learning device.

According to the present invention, key points that are robust to changes in lighting conditions can be extracted.

FIG. 2 is a diagram for explaining a learning model of this embodiment. It is an example of the hardware block diagram of the learning device and processing device of this embodiment. It is an example of the functional block diagram of the learning device of this embodiment. It is an example of the functional block diagram of the processing device of this embodiment. FIG. 3 is a diagram for explaining an example of usage of the processing device according to the present embodiment. FIG. 7 is a diagram for explaining another usage example of the processing device of the present embodiment.

Embodiments of the present invention will be described below with reference to the drawings. Note that in all the drawings, similar components are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

<First embodiment>
"overview"
First, an overview of the learning device of this embodiment will be explained using FIG. The learning device of this embodiment executes characteristic learning based on characteristic learning data and adjusts the parameters of the illustrated learning model (eg CNN: Convolutional Neural Network).

The illustrated learning model is based on an input image ("H x W image" in the figure) and a feature map ("H' x W' x C feature group" in the figure) that shows the feature values for each pixel. ) and a first weighting map ("Saliency map" in the figure) indicating a weighting value for each pixel.

Based on the feature amount map and the first weighting map, pixels to be key points are determined. Specifically, an evaluation value is calculated for each pixel based on the feature amount map and the first weighting map. Then, a pixel to be a key point is determined based on the evaluation value. Note that the illustrated learning model may further include a function to determine the key points, or another processing means that is physically and/or logically separate from the illustrated learning model may have this function. .

Although details will be described later, the learning device of this embodiment executes characteristic learning based on characteristic learning data and adjusts the parameters of the illustrated learning model. As a result, a feature map and a first weighting map are generated in which the evaluation value of pixels that are robust to changes in illumination conditions is relatively high. In other words, a feature map and feature map in which the evaluation value of a pixel that can be distinguished from surrounding pixels is relatively high not only when the image is taken under a specific lighting condition but also when the image is taken under various lighting conditions. A weighting map of 1 is now generated. As a result, pixels that are robust to changes in lighting conditions are more likely to be determined as key points.

"composition"
Next, the configuration of the learning device will be explained. First, an example of the hardware configuration of the learning device will be explained. Each functional part of the learning device consists of a CPU (Central Processing Unit) of any computer, a memory, a program loaded into the memory, and a storage unit such as a hard disk that stores the program (the program is stored in advance at the stage of shipping the device). (In addition to programs, it can also store programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet, etc.), and is realized by any combination of hardware and software, centering on network connection interfaces. . It will be understood by those skilled in the art that there are various modifications to the implementation method and device.

FIG. 2 is a block diagram illustrating the hardware configuration of the learning device. As shown in FIG. 2, the learning device includes a processor 1A, a memory 2A, an input/output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The learning device does not need to have the peripheral circuit 4A. Note that the learning device may be composed of a plurality of physically and/or logically separated devices, or may be composed of one physically and/or logically integrated device. When the learning device is composed of a plurality of physically and/or logically separated devices, each of the plurality of devices can be provided with the above hardware configuration.

The bus 5A is a data transmission path through which the processor 1A, memory 2A, peripheral circuit 4A, and input/output interface 3A exchange data with each other. The processor 1A is, for example, an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The input/output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., an interface for outputting information to an output device, an external device, an external server, etc. . Input devices include, for example, a keyboard, mouse, microphone, physical button, touch panel, and the like. Examples of the output device include a display, a speaker, a printer, and a mailer. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

Next, the functional configuration of the learning device will be explained. As shown in FIG. 3, the learning device 10 includes a storage section 11, an acquisition section 12, and a learning section 13.

The storage unit 11 stores parameters of a learning model (hereinafter sometimes simply referred to as a "learning model") that generates both a feature map and a first weighting map based on an input image. That is, the storage unit 11 stores the parameters of the learning model shown in FIG.

The feature amount map shows the feature amount for each pixel. The feature amount of each pixel is represented by C values (C is 2 or more). When the input image has H×W pixels, the feature amount map is expressed as H′×W′×C as shown in FIG. The type of feature amount and the means for generating the feature amount map are not particularly limited, and any conventional techniques can be employed.

Note that by adjusting the parameters of the learning model by the learning unit 13, which will be described later, a feature map is generated in which the evaluation value of pixels that are robust to changes in illumination conditions is relatively high.

The first weighting map is generated from the feature map, for example. Alternatively, the first weighting map may be generated using the output of the intermediate layer of the network. The first weighting map is generated, for example, by convolving the feature map. When the feature map is expressed as H'×W'×C, the first weighting map is H'×W'×1. That is, in the first weighting map, each pixel has one value. This one value for each pixel becomes a weighting value that is referenced in the process of determining a pixel to be a key point. The means for convolving the H′×W′×C feature map to generate the H′×W′×1 map is not particularly limited, and any conventional technique can be employed.

Note that by adjusting the parameters of the learning model by a learning means described later, a first weighting map is generated in which the evaluation value of pixels that are robust to changes in illumination conditions is relatively high.

The acquisition unit 12 acquires characteristic learning data. Specifically, the acquisition unit 12 acquires, as learning data, a combination of a plurality of learning images that have different lighting conditions and include the same subject. The number of learning images in each combination may be two, three or more.

The different lighting conditions mean that the lighting conditions at the time of photographing are different from each other. In other words, different lighting conditions mean that at least one of the conditions of natural light (sunlight, moonlight, etc.) and the conditions of artificial light (lights, candles, camera flash, etc.) at the time of shooting are different from each other. means. Note that in order to learn efficiently, it is preferable to use a plurality of learning images with sufficiently large differences in lighting conditions. For example, the plurality of learning images may be taken at different times (for example, one taken during the day and the other at night), or may be taken at different locations, whether outdoors or indoors. The weather at the time of shooting may be different from each other (for example, one shot was taken when it was sunny and the other was taken when it was raining), and the lighting (lights installed on buildings, lights prepared for shooting, camera flash) etc.) may be different from each other in their states (ON/OFF, intensity, etc.), or may be different from each other in other respects.

The learning unit 13 adjusts the parameters of the learning model stored in the storage unit 11 based on the characteristic loss function. By learning based on this characteristic loss function, a feature amount map and a first weighting map in which evaluation values of pixels that are robust to changes in illumination conditions are relatively high are generated. In other words, the loss function is designed so that a feature map and a first weighting map are generated in which the evaluation value of pixels that are robust to changes in illumination conditions is relatively high.

A pixel that is robust to changes in illumination conditions is a pixel that can be distinguished from surrounding pixels not only when photographed under a specific illumination condition but also when photographed under various illumination conditions. A pixel that can be distinguished from surrounding pixels is a pixel that has a sufficiently large difference in feature amount (feature amount shown in the feature amount map) from the surrounding pixels.

Here, the loss function will be explained in detail. The loss function of this embodiment is expressed, for example, by the following equation (1). Note that the following loss function may be changed as appropriate within the range that similar results can be obtained.

The loss function shows an example in which a combination of two learning images is acquired as learning data. Various parameters are defined as follows.

I indicates the first learning image.
I′ indicates the second learning image.
i and j indicate the coordinate values of pixels in the first learning image.
p _ij indicates a pixel in the area where the object exists in the first learning image. All pixels in the area where the subject exists may be shown, or some pixels picked up by any means may be shown. In this embodiment, information indicating an area where a subject exists in an image is used. Information indicating the area where the subject exists may be input to the learning device 10 from the outside, or the learning device 10 may analyze the learning image and identify the area where the subject exists.
S(p _ij ) indicates the state value of p _ij pixel of the first learning image. The status value will be described later.
U(i,j) indicates the pixel of the second learning image that corresponds to the (i,j) pixel of the first learning image. "Corresponding pixels" are pixels that indicate the same part of the same object. In this embodiment, information indicating corresponding pixels of a plurality of learning images is used. Information indicating the corresponding pixels of the plurality of learning images may be input to the learning device 10 from the outside, or the learning device 10 may analyze the learning images and identify the corresponding pixels of the plurality of learning images. .
p _{′ U (i, j)} indicates a pixel of the second learning image that corresponds to p _ij pixel of the first learning image.
S(p'U _(i,j)) indicates the state value of p'U _(i,j) pixel of the second learning image.
C _ij indicates the weighting value of the (i,j) pixel of the first weighting map generated based on the first learning image. As another example, C _ij is the weighting value of the (i,j) pixel of the first weighting map generated based on the first training image and the second weighting value generated based on the second training image. It may also show statistical values (average value, maximum value, minimum value, etc.) with weighting values of U(i,j) pixels of the map.
P indicates a patch group focused on p _ij pixels. A patch group is a set of pixels including a pixel of interest and pixels around it. What kind of positional relationship the pixels have with the pixel of interest to be included in the "surrounding pixels" can be arbitrarily determined based on required performance and the like.
|P| indicates the number of pixels included in the patch group.

Here, the state value of each pixel will be explained. The state value S(p _ij ) of the p _ij pixel of the first learning image is expressed by the following equation (2). Note that the state value of p _{′ U (i, j)} pixel of the second learning image is also obtained using a similar formula.

F _ij indicates a collection of a plurality of feature amounts (C values (C is 2 or more)) of the p _ij pixel.
var(F _ij ) indicates the unbiased variance of C values (C is 2 or more) of F _ij .
m and n indicate coordinate values of pixels other than p _ij included in the patch group focused on p _ij .
F _mn indicates a collection of a plurality of feature amounts (C values (C is 2 or more)) of pixels other than p _ij included in the patch group focused on p _ij .
var(F _mn ) indicates the unbiased variance of C values (C is 2 or more) of F _mn .
|p| indicates the number of patch groups minus 1.

In this way, the loss function is expressed by the state values of each pixel (S(p _ij ), S(p′ _{U(i, j)} )) calculated based on the feature map and the first weighting map. It is defined using the weighting value C _ij of each pixel. The state value of each pixel is calculated based on the feature amount of each pixel and the feature amounts of a plurality of pixels surrounding each pixel. Specifically, when the feature amount of each pixel is represented by C values (C is 2 or more), the state value of each pixel is determined by the unbiased variance of the C values of each pixel and the surroundings of each pixel. It is calculated based on the unbiased variance of C values of each of a plurality of pixels.

Here, an example of a method of calculating an evaluation value based on the feature amount map and the first weighting map, and a method of determining a key point based on the evaluation value will be described. In this embodiment, the weighting value of each descriptor (each pixel) indicated by the first weighting map becomes the evaluation value. Then, pixels with large evaluation values are determined as key points. For example, pixels with evaluation values equal to or higher than a reference value may be determined as key points, a predetermined number of pixels with larger evaluation values may be determined as key points, or key points may be determined based on other criteria. may be done.

As described above, by learning based on a characteristic loss function, a feature amount map and a first weighting map in which the evaluation value of pixels that are robust to changes in illumination conditions are relatively high are generated. By calculating evaluation values and determining key points using the above method, pixels that are robust to changes in illumination conditions can be determined as key points.

As described above, the learning device of this embodiment executes characteristic learning based on characteristic learning data (learning based on a characteristic loss function) and adjusts the parameters of a learning model. As a result, a feature amount map and a first weighting map are generated in which the evaluation value of pixels that are robust to changes in illumination conditions is relatively high. In other words, a feature map and a feature map in which the evaluation value of a pixel that can be distinguished from surrounding pixels is relatively high not only when the image is taken under a specific lighting condition but also when the image is taken under various lighting conditions. A weighting map of 1 is now generated. As a result, pixels that are robust to changes in lighting conditions are more likely to be determined as key points.

<Second embodiment>
"overview"
The processing device of this embodiment generates a feature map and a first weighting map from an input image using a learning model whose parameters have been adjusted by the learning device 10 described in the first embodiment, and based on them, Determine key points.

"composition"
First, an example of the hardware configuration of the processing device will be described. Each functional part of the processing device consists of a CPU of any computer, a memory, a program loaded into the memory, a storage unit such as a hard disk that stores the program (in addition to a program stored in advance at the stage of shipping the device, a CD It can also store programs downloaded from storage media such as , servers on the Internet, etc.), and is realized by any combination of hardware and software centered on a network connection interface. It will be understood by those skilled in the art that there are various modifications to the implementation method and device.

FIG. 2 is a block diagram illustrating the hardware configuration of the processing device. As shown in FIG. 2, the processing device includes a processor 1A, a memory 2A, an input/output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device does not need to have the peripheral circuit 4A. Note that the processing device may be composed of a plurality of physically and/or logically separated devices, or may be composed of one physically and/or logically integrated device. When the processing device is composed of a plurality of physically and/or logically separated devices, each of the plurality of devices can be provided with the above hardware configuration. Note that each element in FIG. 2 has been described in the first embodiment, and will therefore be omitted here.

Next, the functional configuration of the processing device will be explained. As shown in FIG. 4, the processing device 20 includes a storage section 21, an input section 22, an estimation section 23, and an output section 24.

The storage unit 21 stores a learning model whose parameters have been adjusted by the learning device 10 described in the first embodiment. Specifically, the storage unit 21 stores information (data) necessary for executing the learning model, such as parameters of the learning model.

The input unit 22 receives input of an input image.

The estimation unit 23 inputs the input image to the learning model stored in the storage unit 21 and obtains the estimation result.

When the learning model is configured to generate and output a feature map and a first weighting map based on the input image, the estimation unit 23 generates a feature map and a first weighting map based on the output feature map and first weighting map. An evaluation value of each pixel is calculated, and a pixel to be a key point is determined based on the evaluation value. The method for calculating the evaluation value and the method for determining pixels to be key points are as described in the first embodiment.

On the other hand, if the learning model is configured to generate a feature map and a first weighting map based on the input image, determine and output key points based on them, the estimation unit 23 Information indicating the output key point (information indicating the pixel determined as the key point) is obtained as the estimation result.

The output unit 24 outputs information indicating the pixels determined as key points.

Next, a usage example of the processing device 20 will be described using FIG. 5. In this example, the processing device 20 is used to search a database for images that include a subject similar to the subject included in the query image, and to output the searched images.

As illustrated, when a query image is input to the processing device 20, the processing device 20 executes the above-described processing and determines key points from the query image. Then, information indicating the pixels determined as key points is output. The output information is input to a similar image search device. The similar image search device searches a database for an image that includes a feature amount similar to the feature amount of a pixel determined as a key point, and outputs the searched image.

As described above, according to the processing device 20, pixels that are robust to changes in illumination conditions are determined and output as key points. By performing an image search using the features of such key points, you can search for images that include subjects with similar appearance to the subjects included in the query image, even if the lighting conditions at the time the images were captured were the same as when the query image was captured. It is possible to perform a search with high accuracy regardless of whether the lighting conditions are the same as the lighting conditions. That is, for example, by performing an image search based on a query image that includes building A that was photographed during the day, it is possible to search not only images that include building A that were photographed during the day but also images that include building A that were photographed at night with high accuracy. It becomes possible to do so.

Another usage example of the processing device 20 will be explained using FIG. 6. In this example, the processing device 20 is used to search a database for an image that includes a subject similar to the subject included in the query image, and to output position information associated with the searched image.

As illustrated, when a query image is input to the processing device 20, the processing device 20 executes the above-described processing and determines key points from the query image. Then, information indicating the pixels determined as key points is output. The output information is input to a similar image search device. The similar image search device searches a database for an image that includes a feature amount similar to the feature amount of the pixel determined as a key point. Note that in the database, position information indicating the position where each image was photographed is stored in association with each image. The similar image search device outputs location information linked to the searched images.

As described above, according to the processing device 20 of this embodiment, pixels that are robust to changes in illumination conditions can be determined as key points and output. By using such key points, image retrieval that is robust to changes in lighting conditions can be realized.

<Third embodiment>
In addition to the feature map and the first weighting map, the processing device 20 of this embodiment generates at least one of a reproducibility map and a reliability map based on the input image. Generation of the feature map and the first weighting map is realized by the method described in the second embodiment. Generation of a map regarding reproducibility and a map regarding reliability is realized by the method disclosed in Non-Patent Document 1.

Both the reproducibility map and the reliability map show a weighting value for each pixel. If the input image has H×W pixels and the feature map is expressed as H′×W′×C, both the reproducibility map and the reliability map are expressed as H′×W′×1. It will be done. That is, in the reproducibility map and the reliability map, each pixel has one value.

The processing device 20 then calculates an evaluation value for each pixel using at least one of a reproducibility map and a reliability map in addition to the feature map and the first weighting map, and based on the calculated evaluation value. Determine key points.

For example, for each pixel, the evaluation value is calculated by multiplying the weighting value shown in the first weighting map, the weighting value shown in the reproducibility map, and the weighting value shown in the reliability map. Good too.

In addition, for each pixel, the evaluation value is calculated by adding together the weighting value shown in the first weighting map, the weighting value shown in the reproducibility map, and the weighting value shown in the reliability map. Good too. In this case, the sum of (weighting value shown in the first weighting map) × α, (weighting value shown in the map related to reproducibility) × β, and (weighting value shown in the map related to reliability) × γ may be used as the evaluation value. α, β, and γ are weighting values for each map. Note that in the case of this method, pixels with high evaluation values calculated by evaluating the three maps collectively are extracted as key points. As another example, in addition to having high evaluation values calculated based on the three maps, pixels having high overall scores in each map may be extracted as key points. For example, the score of each map may be filtered using a predetermined condition (greater than a threshold). Specifically, when the reproducibility map is A, the reliability map is B, and the first weighting map is C, the pixel group M that satisfies the condition is M={(i,j∈R ² )|A _ij >thr1, B _ij >thr2, C _ij >thr3}. Then, the process of calculating the evaluation value described above may be performed for the pixel group M, and key points may be extracted from the pixel group M based on the evaluation value.

In addition, in the calculation method described above, the evaluation value may be calculated using only one of the weighting values shown in the map related to reproducibility and the weighting value shown in the map related to reliability, without using both.

The process of determining key points based on the calculated evaluation values is as described in the second embodiment.

The other configuration of the processing device 20 is the same as that of the second embodiment.

According to the processing device 20 of this embodiment, the same effects as in the second embodiment are realized. Further, according to the processing device 20 of the present embodiment, key points can be determined using at least one of a reproducibility map and a reliability map in addition to the feature map and the first weighting map. . As a result, it is possible to determine and output pixels that are robust to changes in lighting conditions and to changes in photographing angle as key points. By using such key points, image retrieval that is robust to changes in illumination conditions and changes in photographing angle can be realized.

In this specification, "acquisition" refers to "a process in which the own device retrieves data stored in another device or storage medium (actively)" based on user input or program instructions. (e.g., requesting or interrogating and receiving from other devices, accessing and reading other devices or storage media, etc.), and based on user input or program instructions. "Inputting data output from another device into one's own device (passive acquisition)," for example, receiving data that is distributed (or sent, push notification, etc.), and receiving received data or information. "Create new data by editing the data (converting it into text, sorting the data, extracting some data, changing the file format, etc.), and ``Obtaining data.''

（Ｉは、第１の学習画像を示す。Ｉ´は、第２の学習画像を示す。ｉ及びｊは、第１の学習画像の中のピクセルの座標値を示す。ｐ_ｉｊは、第１の学習画像の中の被写体が存在するエリアの中のピクセルを示す。Ｓ（ｐ_ｉｊ）は、第１の学習画像のｐ_ｉｊピクセルの状態値を示す。Ｕ（ｉ，ｊ）は、第１の学習画像の（ｉ，ｊ）ピクセルに対応する第２の学習画像のピクセルを示す。ｐ´_{Ｕ（ｉ，ｊ）}は、第１の学習画像のｐ_ｉｊピクセルに対応する第２の学習画像のピクセルを示す。Ｓ（ｐ´_{Ｕ（ｉ，ｊ）}）は、第２の学習画像のｐ´_{Ｕ（ｉ，ｊ）}ピクセルの状態値を示す。Ｃ_ｉｊは、第１の学習画像に基づき生成された第１の重み付けマップの（ｉ，ｊ）ピクセルの重み付け値を示す。Ｐは、ｐ_ｉｊに着目したパッチ群を示す。|Ｐ|は、パッチ群に含まれるピクセルの数を示す。Ｆ_ｉｊは、ｐ_ｉｊピクセルの特徴量（Ｃ個（Ｃは２以上）の値）を示す。ｖａｒ（Ｆ_ｉｊ）は、Ｆ_ｉｊのＣ個（Ｃは２以上）の値の不偏分散を示す。ｍ及びｎは、ｐ_ｉｊに着目したパッチ群に含まれるｐ_ｉｊを除くピクセルの座標値を示す。Ｆ_ｍｎは、ｐ_ｉｊに着目したパッチ群に含まれるｐ_ｉｊを除くピクセルの特徴量（Ｃ個（Ｃは２以上）の値）を示す。ｖａｒ（Ｆ_ｍｎ）は、Ｆ_ｍｎのＣ個（Ｃは２以上）の値の不偏分散を示す。|ｐ|は、パッチ群の個数から１を引いた数を示す。）
６． コンピュータが、
入力画像に基づき、ピクセル毎の特徴量を示す特徴量マップと、キーポイントとするピクセルを決定する処理で利用される重み付け値をピクセル毎に示す第１の重み付けマップとを生成する学習モデルのパラメータを記憶しておき、
照明条件が互いに異なり、かつ同一の被写体を含む複数の学習画像の組み合わせを取得し、
前記複数の学習画像各々から生成される複数の前記特徴量マップ各々に基づき算出される複数のパラメータと、前記複数の学習画像から生成される前記第１の重み付けマップに基づき算出されるパラメータとを用いて定義される損失関数に基づき、前記学習モデルのパラメータを調整する学習方法。
７． コンピュータを、
入力画像に基づき、ピクセル毎の特徴量を示す特徴量マップと、キーポイントとするピクセルを決定する処理で利用される重み付け値をピクセル毎に示す第１の重み付けマップとを生成する学習モデルのパラメータを記憶する記憶手段、
照明条件が互いに異なり、かつ同一の被写体を含む複数の学習画像の組み合わせを取得する取得手段、
前記複数の学習画像各々から生成される複数の前記特徴量マップ各々に基づき算出される複数のパラメータと、前記複数の学習画像から生成される前記第１の重み付けマップに基づき算出されるパラメータとを用いて定義される損失関数に基づき、前記学習モデルのパラメータを調整する学習手段、
として機能させるプログラム。
８． １から５のいずれかに記載の学習装置で生成された学習モデルを用いて、入力画像のキーポイントを決定する処理装置。
９． コンピュータが、１から５のいずれかに記載の学習装置で生成された学習モデルを用いて、入力画像のキーポイントを決定する処理方法。
１０． コンピュータを、１から５のいずれかに記載の学習装置で生成された学習モデルを用いて、入力画像のキーポイントを決定する手段として機能させるプログラム。 Part or all of the above embodiments may be described as in the following supplementary notes, but the embodiments are not limited to the following.
1. Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. a storage means for storing the
acquisition means for acquiring a combination of a plurality of learning images that have different lighting conditions and include the same subject;
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. learning means for adjusting parameters of the learning model based on a loss function defined using the learning model;
A learning device with
2. 2. The learning device according to claim 1, wherein the learning means adjusts parameters of the learning model that generates both the feature map and the first weighting map based on the input image, based on the loss function.
3. The loss function is defined using a state value of each pixel calculated based on the feature map and a weighting value of each pixel indicated by the first weighting map,
3. The learning device according to 1 or 2, wherein the state value of each pixel is calculated based on the feature amount of each pixel and the feature amounts of a plurality of pixels surrounding each pixel.
4. The feature amount of each pixel is represented by C values (C is 2 or more),
4. The state value of each pixel is calculated based on an unbiased variance of the C values of each pixel and an unbiased variance of the C values of each of a plurality of pixels surrounding each pixel. learning device.
5. 5. The learning device according to any one of 1 to 4, wherein the loss function is expressed by the following formula.

(I indicates the first learning image. I' indicates the second learning image. i and j indicate the coordinate values of pixels in the first learning image. p _ij indicates the first learning image. S(p _ij ) indicates the state value of the p _ij pixel in the first learning image. U(i, j) indicates the pixel in the area where the subject exists in the first learning image. p _{′ U(i, j)} is the pixel of the second learning image corresponding to the p _ij pixel of the first learning image. S(p′ _{U(i, j)} ) indicates the state value of p _{′ U(i, j)} pixel of the second learning image. Indicates the weighting value of the (i, j) pixel of the generated first weighting map. P indicates a patch group focused on p _ij . |P| indicates the number of pixels included in the patch group. F _ij indicates the feature amount (C values (C is 2 or more)) of the p _ij pixel. var (F _ij ) indicates the unbiased variance of the C values (C is 2 or more) of F _ij . _m and n indicate the coordinate values of pixels excluding _{p ij} included in the patch group focused on p _ij . F _mn is the feature amount ( var (F _mn ) indicates the unbiased variance of C values (C is 2 or more) of F _mn . |p| is calculated from the number of patch groups. (Indicates the number minus 1.)
6. The computer is
Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. Remember,
Obtain a combination of multiple training images that have different lighting conditions and include the same subject,
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. A learning method that adjusts parameters of the learning model based on a loss function defined using the learning model.
7. computer,
Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. storage means for storing
acquisition means for acquiring a combination of a plurality of learning images that have different lighting conditions and include the same subject;
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. learning means for adjusting parameters of the learning model based on a loss function defined using the learning model;
A program that functions as
8. A processing device that determines key points of an input image using a learning model generated by the learning device according to any one of items 1 to 5.
9. 6. A processing method in which a computer determines key points of an input image using a learning model generated by the learning device according to any one of 1 to 5.
10. A program that causes a computer to function as means for determining key points of an input image using a learning model generated by the learning device according to any one of items 1 to 5.

10 learning device 11 storage unit 12 acquisition unit 13 learning unit 20 processing unit 21 storage unit 22 input unit 23 estimation unit 24 output unit 1A processor 2A memory 3A input/output I/F
4A peripheral circuit 5A bus

Claims (10)

Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. a storage means for storing the
acquisition means for acquiring a combination of a plurality of learning images that have different lighting conditions and include the same subject;
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. learning means for adjusting parameters of the learning model based on a loss function defined using the learning model;
A learning device with The learning device according to claim 1, wherein the learning means adjusts parameters of the learning model that generates both the feature map and the first weighting map based on the input image, based on the loss function. The loss function is defined using a state value of each pixel calculated based on the feature map and a weighting value of each pixel indicated by the first weighting map,
The learning device according to claim 1 or 2, wherein the state value of each pixel is calculated based on the feature amount of each pixel and the feature amounts of a plurality of pixels surrounding each pixel. The feature amount of each pixel is represented by C values (C is 2 or more),
4. The state value of each pixel is calculated based on an unbiased variance of the C values of each pixel and an unbiased variance of the C values of each of a plurality of pixels surrounding each pixel. learning device. The learning device according to any one of claims 1 to 4, wherein the loss function is expressed by the following formula.

(I indicates the first learning image. I' indicates the second learning image. i and j indicate the coordinate values of pixels in the first learning image. p _ij indicates the first learning image. S(p _ij ) indicates the state value of the p _ij pixel in the first learning image. U(i, j) indicates the pixel in the area where the subject exists in the first learning image. p _{′ U(i, j)} is the pixel of the second learning image corresponding to the p _ij pixel of the first learning image. S(p′ _{U(i, j)} ) indicates the state value of p _{′ U(i, j)} pixel of the second learning image. Indicates the weighting value of the (i, j) pixel of the generated first weighting map. P indicates a patch group focused on p _ij . |P| indicates the number of pixels included in the patch group. F _ij indicates the feature amount (C values (C is 2 or more)) of the p _ij pixel. var (F _ij ) indicates the unbiased variance of the C values (C is 2 or more) of F _ij . _m and n indicate the coordinate values of pixels excluding _{p ij} included in the patch group focused on p _ij . F _mn is the feature amount ( var (F _mn ) indicates the unbiased variance of C values (C is 2 or more) of F _mn . |p| is calculated from the number of patch groups. (Indicates the number minus 1.) The computer is
Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. Remember,
Obtain a combination of multiple training images that have different lighting conditions and include the same subject,
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. A learning method that adjusts parameters of the learning model based on a loss function defined using the learning model. computer,
Parameters of a learning model that generates, based on an input image, a feature amount map showing feature amounts for each pixel and a first weighting map showing, for each pixel, weighting values used in the process of determining pixels as key points. storage means for storing
acquisition means for acquiring a combination of a plurality of learning images that have different lighting conditions and include the same subject;
a plurality of parameters calculated based on each of the plurality of feature maps generated from each of the plurality of learning images; and a parameter calculated based on the first weighting map generated from the plurality of learning images. learning means for adjusting parameters of the learning model based on a loss function defined using the learning model;
A program that functions as A processing device that determines key points of an input image using a learning model generated by the learning device according to any one of claims 1 to 5. A processing method in which a computer determines key points of an input image using a learning model generated by the learning device according to any one of claims 1 to 5. A program that causes a computer to function as means for determining key points of an input image using a learning model generated by the learning device according to any one of claims 1 to 5.

Images