JP7386630B2 - Image processing device, control method and program for the image processing device - Google Patents

Description

The present invention relates to an image processing apparatus, a method of controlling the image processing apparatus, and a program.

2. Description of the Related Art In recent years, systems have been proposed that search for a specific person in a video by photographing a predetermined area with a camera and analyzing the photographed video. Such a system is expected to be useful for early detection of suspicious persons by searching for people from surveillance cameras installed in public spaces using facial images registered in the system in advance. Face recognition technology is used to search for people by matching registered people with facial images of people in videos.

In face recognition technology, feature quantities representing facial features are extracted from each of the registered and input face images, and it is determined whether the two are the same person based on the similarity of the extracted feature quantities. In recent years, with the introduction of deep learning technology, it has been reported that such face recognition technology exceeds human recognition accuracy. In Non-Patent Document 1, a facial image that has been detected and aligned in advance from an image is compared using facial features extracted by a neural network.

Taigman et al. Deepface: closing the gap to human-level performance in face verification. in 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Girshick. Fast R-CNN. in 2015 IEEE International Conference on Computer Vision(ICCV)

However, in the conventional technology, it is necessary to detect feature points such as eyes and mouth from a face image in advance and perform alignment. Therefore, if the accuracy of alignment is low, the extracted feature amounts will shift, resulting in a problem that face matching accuracy will decrease.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a technique for accurately matching object images such as face images without requiring preprocessing such as alignment. do.

An image processing device according to the present invention that achieves the above object includes:
An image processing device that matches objects from images obtained by photographing with a photographing device, the image processing device comprising:
Feature map calculation means for calculating a plurality of feature maps indicating positions where feature points of the image are detected from the image using a convolutional neural network;
Estimating means for estimating the feature point position of the object in the image in the plurality of feature maps calculated by the feature map calculating means;
normalization processing means for extracting partial regions from the plurality of feature maps using the feature point positions estimated by the estimation means, and normalizing the partial regions to a predetermined size;
Extracting means for extracting features of the object including features corresponding to the feature point positions from the feature map normalized by the normalization processing means;
collation means for collating the features extracted by the extraction means and the features stored in advance;
It is characterized by having the following.

According to the present invention, it is possible to accurately match object images without requiring preprocessing such as alignment.

1 is a diagram showing an example of a functional configuration of an image processing apparatus according to an embodiment of the present invention. 5 is a flowchart illustrating a procedure of facial image matching processing performed by an image processing apparatus according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of a convolutional neural network according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of facial feature points according to an embodiment of the present invention. FIG. 4 is a diagram showing examples of three feature maps output by a convolutional neural network when a facial image is input, according to an embodiment of the present invention. 1 is a flowchart showing a procedure of facial image registration processing performed by an image processing apparatus according to an embodiment of the present invention. 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

<Hardware configuration of image processing device>
FIG. 7 is a diagram showing an example of the hardware configuration of the image processing apparatus according to this embodiment. In FIG. 7, an image processing device 200 includes an arithmetic processing device 1, a storage device 2, an input device 3, and an output device 4. Note that the devices are configured to be able to communicate with each other and are connected via a bus or the like.

The arithmetic processing device 1 controls the operation of the image processing device 200, executes programs stored in the storage device 2, etc., and is composed of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The storage device 2 is a storage device such as a magnetic storage device or a semiconductor memory, and stores programs read based on the operation of the arithmetic processing unit 1, data that must be stored for a long time, and the like. In this embodiment, the arithmetic processing device 1 performs processing according to the procedure of the program stored in the storage device 2, thereby realizing the functions of the image processing device 200 and the processing related to the flowcharts described below. The storage device 2 also stores images to be processed by the image processing device 200 and processing results.

The input device 3 is a mouse, keyboard, touch panel device, button, etc., and is used to input various instructions. The input device 3 may also include an imaging device such as a camera. The output device 4 is a liquid crystal panel, an external monitor, etc., and outputs various information.

Note that the hardware configuration of the image processing device 200 is not limited to the configuration described above. For example, the image processing device 200 may include an I/O device for communicating between various devices. For example, the I/O device is an input/output unit such as a memory card or a USB cable, or a wired or wireless transmission/reception unit.

<Functional configuration of image processing device>
FIG. 1 is a diagram showing the functional configuration of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 1, the image processing device 200 is connected to a camera (photographing device) 100. The image processing device 200 includes a parameter acquisition section 201, an image acquisition section 202, an image scaling section 203, a feature map calculation section 204, and a feature map storage section 205. The image processing device 200 also includes a facial feature point position estimation unit 206, a normalization processing unit 207, a facial feature extraction unit 208, a facial feature storage unit 209, a face size calculation unit 210, a facial feature selection unit 211, a facial feature matching unit 212, and an input section and a display section (not shown). Note that the image processing device 200 may include the camera 100.

The parameter acquisition unit 201 acquires photographing parameters of the camera 100. The image acquisition unit 202 acquires image data captured by the camera 100. The image scaling unit 203 scales the image data acquired by the image acquisition unit 202 into an image of a predetermined size. The image size to be scaled is set based on the imaging parameters acquired by the parameter acquisition unit 201.

The feature map calculation unit 204 calculates a plurality of feature maps by inputting the image data acquired through the image scaling unit 203 into a convolutional neural network and processing the data. The feature map storage unit 205 stores the processing results of the feature map calculation unit 204. The facial feature point position estimation unit 206 estimates the facial feature point positions of the person in the image from the plurality of feature maps obtained by the feature map calculation unit 204. The normalization processing unit 207 cuts out a partial region from the feature map stored in the feature map storage unit 205 using the facial feature point positions determined by the facial feature point position estimation unit 206, and normalizes it into a feature map of a predetermined size. . The facial feature extraction unit 208 extracts facial features of a person from the feature map output by the normalization processing unit 207. The facial feature storage unit 209 stores the processing results obtained by the facial feature extraction unit 208.

The face size calculation unit 210 calculates the size of the face from the facial feature point positions determined by the facial feature point position estimation unit 206. The facial feature selection unit 211 selects appropriate facial features based on the face size calculated by the face size calculation unit 210 from the facial features stored in the facial feature storage unit 209. The facial feature matching unit 212 matches the facial features obtained by the facial feature extracting unit 208 and the facial features selected by the facial feature selecting unit 211, and outputs the degree of similarity between the facial features.

<Face image matching process>
Next, with reference to the flowchart of FIG. 2, the procedure of face image matching processing performed by the image processing apparatus 200 according to an embodiment of the present invention will be described. Note that, below, a process of matching a registered face image, which is a search target specified by an input unit (not shown), with a face image photographed by the camera 100 will be explained.

In S100, the image acquisition unit 201 acquires image data from the video captured by the camera 100. The image data to be acquired is, for example, two-dimensional data consisting of 8-bit RGB pixels. Alternatively, brightness data consisting of 8-bit pixels may be used, and the number of bits of each pixel is not limited to 8 bits. At this time, the parameter acquisition unit 201 acquires the photographing parameters when the acquired image data was photographed. The photographing parameters to be acquired include the focal length of the photographing lens, the aperture value, the shutter speed, the gain of the image sensor, the compression rate of the photographed image, and the like. However, it is not necessary to acquire all of these photographing parameters, and it is sufficient to acquire at least one parameter that affects the image quality of the photographed image.

In S200, the image scaling unit 203. The image data acquired by the image acquisition unit 202 is scaled to an image of a predetermined size. The image size to be scaled is set based on the imaging parameters acquired by the parameter acquisition unit 201.

In the following, a method will be described in which the image size is determined based on the shutter speed, using the shutter speed as a photographing parameter. Faces in images taken of people walking in public spaces will be blurred if the shutter speed is slow, so if faces are matched using the images obtained at the time of shooting, the matching accuracy will drop. Therefore, in the case of a blurred image, it is necessary to reduce the size of the image to suppress the effects of blur. Therefore, the image size to be scaled is set according to the shutter speed. Specifically, the setting is such that the slower the shutter speed, the smaller the image size to be scaled. To set the image size, the relationship between shutter speed and image size may be stored in advance as a table, and the image size may be determined from the shutter speed by referring to the table.

A similar effect can be obtained by similarly setting the image size for other shooting parameters that affect the image quality of the shot image. For example, the gain of the image sensor and the compression rate of the captured image affect image noise, so similar effects can be obtained.

It goes without saying that similar effects can also be obtained by analyzing the image quality of a photographed image and setting an appropriate image size without acquiring photographic parameters. For example, image blur and noise can be estimated and applied by frequency analysis of a captured image.

In S300, the feature map calculation unit 204 calculates a plurality of feature maps by inputting the image data acquired via the image scaling unit 203 into a convolutional neural network and processing it.

<Example of convolutional neural network>
Here, FIG. 3 shows an example of a convolutional neural network. As shown in FIG. 3, this convolutional neural network is composed of 10 convolutional layers Conv1 to Conv10. The convolutional layers Conv1 to Conv6 are layers that constitute the functions of the feature map calculation unit 204, and output a plurality of feature maps by inputting image data. Further, the convolutional layers Conv7 to Conv10 constitute the function of a facial feature point position estimation unit 206, which will be described later. Each of the convolutional layers Conv1 to Conv10 performs, for example, 3×3 convolution operations for the number of output channels of each layer. In addition, in the convolutional neural network shown in Figure 3, convolution operations are performed by skipping every other pixel (stride number is 2) in convolution layers Conv1 to Conv4, and the resolution of the output feature map is reduced to half the size vertically and horizontally. ing. Furthermore, convolutional operations are performed in the convolutional layers Conv5 to Conv6 so as to output a feature map of the same size as the feature map input from the previous convolutional layer.

In such a multilayer convolutional neural network, feature maps from low to high order can be obtained by repeating convolution operations. A high-order feature has a wider receptive field because the convolution operation is performed on the input image more times than a low-order feature. Here, the receptive field is an area of the input image that contributes to one feature in the map, and higher-order features are the result of extracting features from a wider area of the image.

The processing results of the feature map calculation unit 204 are stored in the feature map storage unit 205. The feature map storage unit 205 stores at least a plurality of feature maps output from the convolutional layer Conv6, which is the final layer of the feature map calculation unit 204, and stores feature maps of a plurality of convolutional layers and different resolutions. You may also do so. For example, the feature maps of convolutional layers Conv1 to Conv3 and Conv6 of the convolutional neural network in FIG. 3 are stored. All feature maps of each convolutional layer may be stored, or some feature maps may be stored. The larger the number of feature maps stored, the higher the face matching accuracy can be expected, but since the capacity of feature maps stored in the feature map storage unit 205 increases, the number of feature maps is optimized depending on the required accuracy. may be selected.

In S400, the facial feature point position estimation unit 206 estimates the facial feature point positions of the person in the image from the plurality of feature maps obtained by the feature map calculation unit 204. Here, FIG. 4 shows an example of facial feature points to be extracted. The facial feature points to be estimated are three points indicated by LE (Left Eye), RE (Right Eye), and N (Nose) in FIG. 4. Here, the positions of three points, the centers of the left and right eyes and the apex of the nose, are estimated, but the positions of other points, such as the mouth, may be estimated.

Referring again to FIG. 3, the convolutional layers Conv7 to Conv10 forming the facial feature point position estimating section 206 will be described. The convolutional layers Conv7 to Conv10 are composed of deconvolutional neural networks. In a deconvolutional neural network, the feature map input from the previous convolutional layer is enlarged to twice the size vertically and horizontally, and then a convolution operation is performed to obtain a feature map that is twice the size vertically and horizontally of the feature map input from the previous convolutional layer. do. Note that the number of output channels of the convolutional layer Conv10, which is the final layer of the facial feature point position estimation unit 206, is three, and the three feature maps are designed to obtain high output values at the positions of the left eye, right eye, and nose, respectively. It has been learned in advance. The learning method of the neural network will be described later. FIGS. 5(a) to 5(c) show output examples of three feature maps when a facial image is input. However, in FIGS. 5(a) to 5(c), the output of the feature map is superimposed on the original face image. In the figure, the distribution of brightness and darkness superimposed on the positions of the left eye (Fig. 5 (a)), right eye (Fig. 5 (b)), and nose (Fig. 5 (c)) are the positions of the left eye, right eye, and nose, respectively. This is an estimation result, indicating that the darker the position, the higher the reliability. The facial feature point position estimating unit 206 outputs the position coordinates of the positions where the value of each feature map of the convolutional layer Conv10 is the highest as the estimated positions of the left eye, right eye, and nose, respectively.

Note that although the example in FIG. 5 shows an output example when there is one face in the image, facial feature point positions can be similarly estimated even when there are multiple faces in the image.

The method for estimating the facial feature point positions from the feature map output by the deconvolutional neural network has been described above, but the feature map output by the feature map calculation unit 204 is input to the fully connected layer and the position coordinates of the facial feature points are directly estimated. may be estimated by regression.

In S500, the normalization processing unit 207 cuts out a partial region from the feature map stored in the feature map storage unit 205 using the facial feature point positions determined by the facial feature point position estimation unit 206, and normalizes it into a feature map of a predetermined size. become The RoI Pooling layer described in Non-Patent Document 2 is used for the feature map normalization process. The RoI Pooling layer projects any rectangular area of the input feature map onto a rectangular area of a predetermined size, and outputs the maximum value of the feature map for each element within the output rectangular area. The rectangular region of the input feature map is a partial region cut out using the facial feature point positions determined by the facial feature point position estimation unit 206.

Here, a method for determining partial regions from facial feature point positions will be explained. If the positional coordinates of the left eye LE, right eye RE, and nose N obtained by the facial feature point position estimation unit 206 are (xL, yL), (xR, yR), and (xN, yN), respectively, then the central coordinate of the partial region (x , y), width W, and height H are determined by the following (Equation 1).

Note that in the above description, partial regions are determined from the positional coordinates of the left eye, right eye, and nose, but the positions of other facial feature points such as the mouth, ear, etc. may also be used. In that case, the facial feature point position estimating unit 206 is configured to be able to estimate the facial feature point positions necessary to find the partial region.

The normalization processing unit 207 performs the normalization processing described above on each of the feature maps stored in the feature map storage unit 205, and outputs the processing results.

In S600, the facial feature extraction unit 208 extracts the facial features of the person from the feature map output by the normalization processing unit 207. The facial feature extraction unit 208 receives the plurality of feature maps output by the normalization processing unit 207 as input and acquires feature quantities (facial features) of a predetermined number of dimensions using a fully connected layer. A plurality of fully connected layers may be provided to extract facial features. The larger the number of output dimensions, the more accurate matching can be expected, but the amount of calculation will also increase, so an appropriate number of dimensions may be selected depending on the required accuracy. The processing results obtained by the facial feature extraction unit 208 are stored in the facial feature storage unit 209.

In S700, the facial feature selection unit 211 selects appropriate facial features from the facial features stored in the facial feature storage unit 209 based on the size of the face in the input image. A plurality of facial features of a person to be matched are stored in advance in the facial feature storage unit 209 at the time of registration according to the resolution of the facial image. Note that the process when registering facial features will be described later. The face size calculation unit 210 calculates the size of the face from the facial feature point positions determined by the facial feature point position estimation unit 206. The size S of the face is obtained, for example, by the following (Formula 2) using the width W and height H obtained by (Formula 1).

The facial feature selection unit 211 selects an appropriate facial feature from the plurality of facial features stored in the facial feature storage unit 209 based on the size S of the face.

The facial features are selected by comparing the face size S calculated by the face size calculation unit 210 with the face sizes of face images of multiple resolutions obtained when registering the facial features of the person to be matched. done by. For example, if a face image of a certain person is registered at two resolutions, high and low, and the face sizes S1 and S2 are obtained at the time of registration, the facial feature with the smaller difference from the input face size S is selected. Make it. Selecting a facial feature whose registered face size is closer to the input face size means selecting a facial feature whose receptive fields of surrounding pixels that contribute to facial feature extraction are closer. The receptive field for facial features extracted from small faces is wider than the receptive field for facial features extracted from large faces.

In S800, the facial feature matching unit 212 matches the facial features obtained by the facial feature extracting unit 208 and the facial features selected by the facial feature selecting unit 211, and outputs the degree of similarity between the facial features. The degree of similarity between facial features is calculated using the L ² norm of the difference between two facial features. In addition to this, a calculation method using parameters obtained by machine learning, such as cosine similarity and support vector machine, may be used as the similarity. In the facial feature matching unit 212, since the facial feature selecting unit 211 selects the facial feature of the registered face that is closer to the receptive field of the facial feature of the input face, highly accurate matching can be performed.

If the output of the facial feature matching unit 212 exceeds a predetermined threshold, a display unit (not shown) of the image processing device displays the facial image as a search result.

In S900, the facial feature matching unit 212 determines whether processing has been completed for all facial images of the current frame. If the processing has not been completed for all face images, the process returns to S500. That is, if the feature point positions of multiple people's faces have been acquired in the facial feature point position estimation process of S400, the processes of S500 to S800 are repeated for each face image. On the other hand, if the processing has been completed for all face images, the process advances to S1000.

In S1000, the facial feature matching unit 212 determines whether there is an image of a subsequent frame captured by the camera 100. If an image of a subsequent frame exists, the process returns to S100 and repeats the process. If the image of the subsequent frame does not exist, the series of processing ends.

<Face image registration processing>
Next, with reference to the flowchart of FIG. 6, a procedure for facial feature registration processing performed by the image processing apparatus according to an embodiment of the present invention will be described. In S110, the image acquisition unit 201 acquires a registered face image that is a search target specified by an input unit (not shown).

In S210, the image scaling unit 203 scales the image data acquired by the image acquisition unit 120 into an image of a predetermined size. A predetermined size is used as the image size to be scaled. When it is assumed that a face image as shown in FIG. 4 is to be registered, the scaling factor is set to have a resolution of, for example, 100×100 pixels and 50×50 pixels. The higher the number of images with different scaling factors, the more accurate matching can be expected, but the amount of calculations during registration and the capacity of facial features to be registered will increase, so the number of images with different scaling factors depends on the required precision. It is better to choose appropriately.

In S310, the feature map calculation unit 204 calculates a plurality of feature maps by inputting the image data acquired via the image scaling unit 203 to a convolutional neural network and processing the image data. The feature map calculation unit 204 performs the same process as the process at the time of matching shown in S300 of FIG.

In S410, the facial feature point position estimating unit 206 estimates the facial feature point position of the person in the image from the plurality of feature maps obtained by the feature map calculating unit 204. The facial feature point position estimation unit 206 performs the same process as the process at the time of matching shown in S400 of FIG.

In S510, the normalization processing unit 207 cuts out a partial region from the feature map stored in the feature map storage unit 205 using the facial feature point positions determined by the facial feature point position estimation unit 206, and converts the partial region into a feature map of a predetermined size. Normalize. The normalization processing unit 207 performs the same process as the process at the time of verification shown in S500 of FIG.

In S610, the facial feature extraction unit 208 extracts the facial features of the person from the feature map output by the normalization processing unit 207. The facial feature extraction unit 208 performs the same process as the process at the time of matching shown in S600 of FIG. The processing results obtained by the facial feature extraction unit 208 are stored in the facial feature storage unit 209. At this time, the face size calculation unit 210 calculates the size of the face using (Equation 2) using the width W and the height H, and stores it in the facial feature storage unit 209 in association with the extracted facial feature.

In S710, the image scaling unit 203 determines whether all the images to be scaled have been processed. Here, if it is necessary to process images with a plurality of variable magnifications in S210, it is determined whether or not the images with all variable magnifications have been processed. If all images have been processed, the series of processing ends. On the other hand, if there is an unprocessed image, the process returns to S210 and the processes of S210 to S610 are repeated for each magnification ratio.

Through a series of processes from S110 to S710, facial features for multiple resolutions of the registered facial image can be acquired.

<Neural network learning method>
Next, a learning method of the neural network used in the feature map calculation unit 204, facial feature point position estimation unit 206, and facial feature extraction unit 208 of this embodiment will be described.

First, a learning method of the neural networks of the feature map calculation unit 204 and the facial feature point position estimation unit 206 shown in FIG. 3 will be described. Facial image data for the number of images used for learning and coordinate values of facial feature point positions in the images are prepared in advance. As a guideline for training samples, tens of thousands to hundreds of thousands of samples are used. The coordinate values of the facial feature point positions are manually specified for each facial image. Next, a reliability map is obtained for each facial feature point from the coordinate values of the facial feature point position. The reliability map is a map similar to the feature map output by the facial feature point position estimation unit 206 shown in FIG. 5, and is a map in which reliability values are assigned based on a Gaussian distribution around the facial feature point positions. be. When estimating the facial feature point positions of the left eye, right eye, and nose, three reliability maps corresponding to each are created. Next, using the prepared face image and reliability map, the parameters of the neural network that minimize the loss function L1 shown in (Equation 3) below are determined.

、および、

は夫々ニューラルネットワークが推定した信頼度マップと学習データとして準備した信頼度マップである。

はＬ²ノルムである。Σ_n、Σ_f、Σ_pは夫々学習データ、顔特徴点、信頼度マップの要素の総和を表す。パラメータの学習は確率的勾配降下法等を用いて行う。 however,

,and,

are the reliability map estimated by the neural network and the reliability map prepared as learning data, respectively.

is the ^L2 norm. Σ _n , Σ _f , and Σ _p represent the sum of learning data, facial feature points, and reliability map elements, respectively. Parameter learning is performed using stochastic gradient descent, etc.

Learning of the neural network used by the facial feature extraction unit 208 is performed using the feature map output by the learned convolutional neural network of the feature map calculation unit 204. Facial image data for the number of images used for learning and coordinate values of facial feature point positions in the images are prepared in advance. As a guideline for training samples, tens of thousands to hundreds of thousands of samples are used. A person ID (personal identification number) is associated with the face image of the learning sample. In learning the neural network, a pair of arbitrary two face images is sampled. Then, using the feature maps output by the convolutional neural network of the feature map calculation unit 204, the facial features output by the neural network used by the facial feature extraction unit 208 are extracted. When extracting facial features, the normalization processing unit 207 performs normalization using the coordinate values of facial feature point positions prepared as learning data from the feature map output by the convolutional neural network of the feature map calculation unit 204. Next, using the prepared face image, facial feature point positions, and person ID, the parameters of the neural network that minimize the loss function L2 shown in (Equation 4) below are determined.

However, d _n is the L ² norm of the difference in facial features calculated from two facial images. y _n takes a value of 1 if the two face images are of the same person, and 0 if the two face images are of different people. Margin is a parameter that takes a predetermined value. Σ _n represents the sum of the number of pairs of two face images sampled as learning data. Learning of the parameters of the neural network used in the facial feature extraction unit 208 is performed using stochastic gradient descent or the like.

As described above, in this embodiment, facial feature point positions are estimated from a feature map extracted from a facial image by a neural network, and facial features are extracted from the feature map using the estimated facial feature point positions. As a result, faces can be matched with high accuracy without requiring preprocessing such as alignment.

In addition, the facial features acquired from multiple facial images of registered people with different resolutions are registered in advance, and the facial features of the facial images that are similar in size are selected and compared, resulting in even higher performance. Accurate matching can be performed. Note that even if the facial features of a registered person are obtained from a facial image with one resolution, the same effect can be achieved by comparing the facial features obtained from multiple facial images with different resolutions from the input facial image. Obtainable. Furthermore, it goes without saying that the same effect can be obtained even if facial features are acquired from facial images of a plurality of resolutions for both registered and input facial images and compared.

Further, since face matching is performed by changing the size of the face image in consideration of the image quality when photographed with a camera, it is possible to perform matching with even higher accuracy. By acquiring photographic parameters and using a face image with an appropriate resolution for matching, it is possible to more easily perform matching that takes into account the image quality of the face image.

Note that although this embodiment has been described using the comparison of face images as an example, the present invention can also be applied to the comparison of a whole body image of a person (that is, the entire person). For example, a feature map is obtained using a convolutional neural network, the positions of feature points such as joint positions of a person are estimated from the feature map, and the whole body features of the person are extracted from the feature map that is extracted and normalized based on the feature point positions. All you have to do is to check it. If the resolution of the image taken by the camera 100 is low and it is difficult to identify a person's face, matching based on body features becomes useful for searching for the person. Furthermore, it goes without saying that the present invention is widely applicable not only to the verification of people but also to the verification of other objects (for example, animals, cars, etc.).

(Other embodiments)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

100: Camera, 200: Image processing device, 201: Parameter acquisition unit, 202: Image acquisition unit, 203: Image scaling unit, 204: Feature map calculation unit, 205: Feature map storage unit, 206: Facial feature point position estimation 207: Normalization processing unit, 208: Facial feature extraction unit, 209: Facial feature storage unit, 210: Face size calculation unit, 211: Facial feature selection unit, 212: Facial feature matching unit

Claims (10)

An image processing device that matches objects from images obtained by photographing with a photographing device, the image processing device comprising:
Feature map calculation means for calculating a plurality of feature maps indicating positions where feature points of the image are detected from the image using a convolutional neural network;
Estimating means for estimating the feature point position of the object in the image in the plurality of feature maps calculated by the feature map calculating means;
normalization processing means for extracting partial regions from the plurality of feature maps using the feature point positions estimated by the estimation means, and normalizing the partial regions to a predetermined size;
Extracting means for extracting features of the object including features corresponding to the feature point positions from the feature map normalized by the normalization processing means;
collation means for collating the features extracted by the extraction means and the features stored in advance;
An image processing device comprising: Calculating means for calculating the size of the object using the feature point positions estimated by the estimating means;
further comprising a selection means for selecting a feature of the object to be used for matching from a plurality of features based on the size of the object calculated by the calculation means,
2. The image processing apparatus according to claim 1, wherein the verification means performs the verification using the features selected by the selection means. 3. The image processing apparatus according to claim 2, wherein the selection means selects features that are similar in size to the object. The image processing according to claim 2 or 3, wherein the selection means selects the feature of the object to be used for the matching from among a plurality of features obtained in advance from a plurality of images of different resolutions. Device. acquisition means for acquiring imaging parameters of the imaging device;
Magnification changing means for changing the size of the image photographed by the photographing device based on the photographing parameters acquired by the acquisition means;
The image processing device according to any one of claims 1 to 4, further comprising: 6. The image processing apparatus according to claim 5, wherein the photographing parameters include parameters that affect the quality of images photographed by the photographing device. analysis means for analyzing the image quality of the image;
A scaling unit for scaling an image obtained by photographing with the photographing device based on the image quality analyzed by the analyzing unit;
The image processing device according to any one of claims 1 to 4, further comprising: 8. The image processing apparatus according to claim 1, wherein the object is an entire person or a face of a person. A method of controlling an image processing device that performs object matching from an image obtained by photographing with a photographing device, the method comprising:
a feature map calculation step of calculating a plurality of feature maps indicating positions where feature points of the image are detected from the image by a convolutional neural network;
an estimation step of estimating the feature point position of the object in the image from the plurality of feature maps calculated in the feature map calculation step;
a normalization processing step of extracting partial regions from the plurality of feature maps using the feature point positions estimated in the estimation step, and normalizing the partial regions to a predetermined size;
an extraction step of extracting features of the object including features corresponding to the feature point positions from the feature map normalized in the normalization processing step;
a matching step of matching the features extracted in the extraction step with features stored in advance;
A method for controlling an image processing device, comprising: A program for causing a computer to function as the image processing device according to any one of claims 1 to 8.

Images