JP7077046B2 - Information processing device, subject identification method and computer program - Google Patents

Description

The present invention relates to an information processing technique for discriminating a subject reflected in an image.

In the device that recognizes the subject (person, object, etc.) reflected in the image, various measures are taken to improve the recognition accuracy. For example, in the technique disclosed in Patent Document 1, the scene in which an image is taken when recognizing an object is classified. That is, the appearance positions of objects on the image are aggregated and classified for each scene. Then, the score representing the prior knowledge of the object is adjusted according to the result of the classification. As a result, erroneous detection is reduced by performing object detection based on prior knowledge that, for example, the frequency of appearance of "cars" is low in the upper part of the screen of the "street corner" scene.

WO2012 / 046426

In the method premised on the classification of scenes shown in Patent Document 1, if the classification accuracy is not appropriate, the recognition accuracy is conversely lowered. In addition, since the distribution of the appearance position of the object changes when the direction and elevation angle of the camera are different, it is necessary to prepare the classification results of many scenes in order to improve the recognition accuracy. Therefore, there is a problem that the effect can be expected only for some images.

An object of the present invention is to provide a technique capable of accurately discriminating a subject in an acquired image without any particular restriction.

The information processing apparatus according to one aspect of the present invention is based on the base feature generation means for generating the base feature which is the base for extracting the feature of the image and the first feature of the first resolution regarding the local region of the image based on the base feature. Based on the base feature, the first feature generating means and the second feature of the image relating to a wide area wider than the local region and having a second resolution coarser than the first resolution are used. A second feature generation means for generating the second feature, a setting means for setting a discrimination parameter for discriminating a subject existing in the image for each partial region of the image based on the second feature, and the first feature. It is characterized by comprising a discriminating means for discriminating the subject based on the discriminating parameter.

According to the present invention, it is possible to provide an information processing apparatus capable of accurately discriminating a subject in an image captured without any particular restriction.

The hardware block diagram of the information processing apparatus which concerns on 1st Embodiment. The functional block block diagram of the information processing apparatus which concerns on 1st Embodiment. The procedure explanatory diagram of the subject discriminating method in 1st Embodiment. Detailed procedure explanatory diagram of the generation process of the base feature. (A) is a procedure explanatory diagram of a local feature generation process, and (B) is a wide area feature generation process. Detailed procedure explanatory diagram of the process of estimating the discrimination parameter. Detailed processing procedure explanatory diagram of the subject discrimination operation. Explanatory drawing which shows the generation process of a feature map. Explanatory diagram showing the generation process of wide area features in connection with camera information. (A) to (C) are explanatory views showing the relationship between the input image and various camera information. A block diagram showing a functional configuration during learning processing. The procedure explanatory diagram of the learning process in the discrimination parameter setting part. An explanatory diagram of the procedure of the regression device learning process in the discrimination parameter setting unit. (A) and (B) are explanatory diagrams showing an example of learning data used for learning processing. An explanatory diagram showing the flow of discriminant parameter setting learning. The functional block diagram of the information processing apparatus of 2nd Embodiment. (A) and (B) are procedure explanatory views of the subject discrimination method in 2nd Embodiment. The block diagram which shows the functional structure of the information processing apparatus of 3rd Embodiment. The procedure explanatory diagram of the subject discriminating method in 3rd Embodiment. (A) to (D) are schematic diagrams of discriminant parameter learning.

Hereinafter, examples of embodiments of the information processing apparatus to which the present invention is applied will be described.
The information processing device can be implemented using a computer having storage and a computer program.
[First Embodiment]
FIG. 1 is a hardware configuration diagram of the information processing apparatus according to the first embodiment. This information processing device includes a computer 10 and its peripheral devices. The computer 10 has a CPU 11, a GPU 12, a ROM 13, a RAM 14, and an external storage device 15 connected to the system bus 19. An input device interface 16, an output device interface 17, and a video equipment interface 18 are also connected to the system bus 19.

The CPU (Central Processing Unit) 11 controls the entire computer 10. The GPU (Graphics Processing Unit) 12 functions as an arithmetic unit that performs high-load arithmetic such as image processing. The ROM (Read Only Memory) 13 stores control programs, parameters, and the like that do not require changes. The RAM (Random Access Memory) 14 is a work memory of the CPU 11, and temporarily stores programs, data, and the like. As an example of storage, the external storage device 15 has a storage medium such as a semiconductor memory, a hard disk, a magneto-optical disk, and a detachable memory card. The external storage device 15 stores the computer program of the present invention, images (including video), various feature maps described later, discrimination parameters, predetermined rules used for feature conversion, camera information, and the like. In addition, a learning image used for learning feature generation, a teacher value used for subject discrimination, a trained neural network model, and the like are also stored.

The input device interface 16 is an interface with an input device 21 such as a pointing device or a keyboard. The output device interface 17 is an interface with the monitor 22 for displaying data. The video device interface 18 is an interface with an image pickup device such as a camera 23.

The computer 10 operates as an information processing device suitable for implementing a method for discriminating a subject by having the CPU 11 read and execute the computer program of the present invention. The GPU 12 processes learning, parameter setting, discrimination, image processing, etc. by a neural network or the like. Of course, if the CPU 11 has sufficiently high performance, the GPU 12 may be omitted.

FIG. 2 shows an example of a functional block configuration when the computer 10 operates as an information processing device. Further, FIG. 3 shows an example of a processing procedure of a subject discrimination method executed by the information processing apparatus of the present embodiment. In the following description, each step of the process will be abbreviated as "S" below.
The image input unit 101 acquires an image (S10). In the present embodiment, it is assumed that the captured image taken by the camera is captured. The captured image is called an "input image". The camera information input unit 110 performs camera information input processing indicating conditions at the time of shooting and the like. The contents of the camera information will be described in detail later.

The base feature generation unit 108 generates and stores a base feature that is a base for feature extraction in the input image (S11). The contents of the base information and the generation process thereof will be described in detail later.

The first feature generation unit 102 generates the first feature including the feature of the local region of the input image by converting the base feature according to a predetermined rule, and stores the generated first feature (S12). The characteristics of the local region are hereinafter referred to as "local characteristics". The second feature generation unit 103 generated and generated a second feature including a feature in a wide area of the input image by converting the base feature according to a predetermined rule and also considering the camera information as necessary. The second feature is stored (S13). "Wide area" refers to a feature that covers a wider area than the local area. The characteristics of this wide area are hereinafter referred to as "wide area features".

For example, when the face of a person is targeted for detection among the subjects and the task is to detect the face, if the person and an artificial object or a natural object are randomly reflected in the input image, many false detections or undetections occur. There is. In the present embodiment, in order to suppress erroneous detection and non-detection, the discrimination parameter setting unit 104 estimates the discrimination parameter used when discriminating the subject according to the input image. Then, this is set for each area block constituting the input image (S14). The "region block" refers to the region of the pixel group specified by the coordinates. For the discrimination parameter, for example, if the likelihood of the subject candidate is higher than that, a threshold value (θ) for discriminating that the candidate is the subject is used. This threshold value (θ) is an estimated value determined based on the second feature, and is a threshold value determined in advance by learning so that the discrimination error is equal to or less than a predetermined value.

The subject discrimination unit 106 discriminates the subject in the input image based on the first feature and the discrimination parameter (S15). In the present embodiment, in order to discriminate a person (or a person's face) to be detected, the subject discrimination unit 106 has three types of likelihood generation units 106a, 106b, 106c and a threshold processing unit 106d. And. The likelihood generation units 106a, 106b, 106c generate a likelihood score based on the first feature generated by the first feature generation unit 102. For this likelihood score, the likelihood scores indicating the certainty (likelihood) that there are a person with a small face, a person with a medium face, and a person with a large face are arranged for each area block of the input image. It is a thing. The subject discrimination unit 106 determines a candidate for a subject based on these likelihood maps. Then, the threshold value processing unit 106d performs threshold value processing on the threshold value (θ) estimated according to the input image and the candidate of the subject, so that whether or not the candidate is a person (or a person's face) to be detected is determined. Determine. The result output unit 107 outputs the determination result to the monitor 22 or the like (S16).

<Generation of base features>
The contents of the process of S11 in FIG. 3 will be described in detail with reference to FIGS. 4 and 8. FIG. 4 is a detailed procedure explanatory diagram of the processing executed by the base feature generation unit 108, and FIG. 8 is a conceptual explanatory diagram of the processing. The base feature can be generated using, for example, a CNN (Convolutional Neural Network, hereinafter abbreviated as CNN). However, in the treatment of S11, a layer called a fully bonded layer of CNN is not used, and only a layer of a type called a convolutional layer is used. In FIG. 8, the number with “ch” is the number of feature maps.

First, the base feature generation unit 108 prepares an empty array for storing the base feature F (x, y) (S1201). That is, the above array is initialized. (X, y) is a subscript representing the coordinates of the pixel (X, Y coordinate system). After initialization, the base feature generation unit 108 generates a multi-layer feature map by repeating the convolution operation shown in FIG. 8 a plurality of times by the CNN (S1202 to S1207 in FIG. 4). In the example of FIG. 8, the base feature generation unit 108 performs a convolution operation 402a on the RGB (red, green, blue) 3ch input image I (x, y) 401, and the 64ch feature map 403a. To generate. Further, the feature map 403b of 128ch is generated by performing the convolution calculation 402b again for the feature map 403a and performing the 1/2 pooling 404a for the calculation result. Similarly, the convolution calculation 402c is performed again on the feature map 403b, and the 1/2 pooling 404b is performed on the calculation result to generate the feature map 403c of 256ch. The 1/2 pooling 404a and 404b are processes for reducing the map size by representing the feature maps 403a and 403b with representative values for each predetermined local region. This process has the effect of making the CNN recognition performance robust (however, the resolution decreases as the calculation progresses). In the 1/2 pooling 404a and 404b of the present embodiment, the feature maps are integrated for each (2 × 2 pixels) and reduced to a feature map having a resolution of 1/2. The convolution operation of the Lth layer and the processing of 1/2 pooling are expressed by mathematical expressions as follows.

[Number 1]
f ^L (x, y, z)
= Θ (Σ _CHIN Σ _{Δx, Δy = -K to +} Kw ^L (Δx, Δy, CHIN, CHOUT)
× f ^L-1 (x + Δx, y + Δx, CHIN) + ^BL _CHOUT )

Here, f ^L (x, y, z) is an output result of the feature map output by the Lth layer, and represents z sheets of feature maps. x and y represent the positions (coordinates) of the pixels. Further, θ (・) is an activation function (ReLU (Rectified Linear Unit) function) consisting of half-wave rectification, and when the input value is 0 or less, it becomes 0, and when it is larger than 1, the input is output as it is. Further, w ^L (Δx, Δy, CHIN, CHOUT) (where Δx, Δy ∈ {−K, ..., 0, ..., K}) is a convolution weight parameter of the Lth layer. BL is the bias term of the ^Lth layer. CHIN represents the identification number of the feature map output by the L-1st layer, and CHOUT represents the identification number of the feature map output by the Lth layer.
In the above equation, the input image I (x, y) is treated as the feature map f ⁰ (x, y, z). Here, in order to prevent the size of the feature map in the x and y directions from changing before and after the convolution calculation, the pixels around the feature map fL-1 are filled with 0 values before the convolution calculation, and then the convolution is performed. In this way, feature maps 403a, 403b, and 403c, which are multiple layers, are generated in each layer of the CNN.

In CNN, weight parameters are learned in advance by an image classification task using a large-scale database. As a result, the feature map 403a of a predetermined number of layers (low layers) 109l close to the input image 401 (= I (x, y)) of the CNN reacts well to a simple pattern such as the inclination of a line segment in the image. It can be a feature map. Further, the feature map 403c of a predetermined number of layers (high layers) 109h farthest from the input image 401 of the CNN can be a feature map that aggregates a wider range of image patterns and reacts to a complicated pattern shape. The feature map 403b of the middle layer (middle layer) 109m can be a feature map that reacts to the pattern shape between the feature map 403a and the feature map 403c.

The base feature generation unit 108 determines whether or not the layer being processed in the CNN is a predetermined layer to be feature extracted (S1204). In the example of the present embodiment, the predetermined layer is any of a low layer, a middle layer, and a high layer representing a predetermined number of sheets (ch number). If it is not a predetermined layer (S1204: No), the process proceeds to S1207. If it is a predetermined layer (S1204: Yes), the feature map is upsampled to a predetermined size (S1205), and the feature map is added to the array of base features F (x, y) (S1206). Such processing is repeated until the loop condition (whether or not the feature map reaches n (natural number)) is satisfied (S1207), and finally the base feature F (x, y) in which the n feature maps are concatenated is repeated. ) Is generated. The process of generating the base feature described above is expressed as follows in the mathematical formula.
[Number 2]
F (x, y)
= [F ¹ (x, y) ^T , f ² (x, y) ^T , ..., f ⁿ (x, y) ^T ] ^T

Here, f ¹ , f ² , ..., F ⁿ are the extracted feature maps, and the base feature F (x, y) integrates the feature maps extracted as described above in the z-dimensional direction. It will be a thing. Since CNN performs 1/2 pooling 404a and 404b, the resolution of the feature map differs depending on the layer. Therefore, the base feature generation unit 108 performs a process of matching each feature map to a predetermined resolution, for example, the resolution of the input image I (x, y) before the above integration (S1205). Specifically, double upsampling (X2 up-sample) 405a is performed on the feature map 403b. Further, the feature map 403c is subjected to quadruple upsampling (X4 up-sample) 405b. Changes to match the resolution can be made by general methods such as copying pixel values and linear interpolation.

As described above, the base feature F (x, y) of 448ch in which the low layer 109l, the middle layer 109m and the high layer 109h of CNN are integrated is generated. As a result, as the first property, information on various scales and variations of various subjects is included in the base feature F (x, y). The second property is that a high-dimensional feature vector having three or more dimensions corresponds to each region (x, y) of the two-dimensional pixel group.
Due to these two properties, the base feature F (x, y) can be effectively used for various subject detection or recognition tasks.

In this embodiment, the resolutions of the input image I (x, y) and the base feature F (x, y) are matched, but this is not the case. By changing the magnification of the upsampling process according to the detection target, it is possible to generate a base feature F'(x', y') with a coarser resolution or a finer resolution than the input image I (x, y). can. Therefore, in the present specification, the above-mentioned "region block" is used as a generalized name representing the region of the pixel group of the base feature F (x, y).

<Generation of the first feature>
Next, a detailed procedure of the process (S12) for generating the first feature by the first feature generation unit 102 will be described with reference to FIG. 5 (A). This process becomes a loop from S1208 to S1211. As an example of a predetermined rule, the first feature generation unit 102 extracts the low-layer feature 109l having the first resolution from the base feature F (x, y), and extracts the region block xy (coordinates (x, y) from the low-layer feature 109l. The pixel group at the position of y), the local feature F _xy for each) is generated and stored. The local feature F _xy is whether or not the feature of the local part such as the face of the person exists in each area block x, y in the subsequent stage when the detection target of discrimination, for example, a person is reflected in the input image 401. It is used to distinguish. The feature amount is a vector that compactly expresses the data necessary for discrimination in the latter stage.

The first feature generation unit 102 acquires the base feature F (x, y), which is a multidimensional feature map, and the features in the vicinity of 8 (S1209). Specifically, the feature vector corresponding to the region block xy of the base feature F (x, y) and the feature vector of the eight region blocks around each region block are extracted. Then, these are connected one-dimensionally and stored as a local feature F _xy (S1210). It is expressed by a mathematical formula as follows. In the following equation, T is the transpose of the vector.
[Number 3]
F _xy = [F (x-1, y-1) ^T , F (x, y-1) ^T , F (x + 1, y-1) ^T ,
F (x-1, y) ^T , F (x, y) ^T , F (x + 1, y) ^T ,
F (x-1, y + 1) ^T , F (x, y + 1) ^T , F (x + 1, y + 1) ^T ,] ^T

<Generation of second feature>
Next, the details of the process (S13) for generating the second feature by the second feature generation unit 103 will be described with reference to FIGS. 5 (B) and 9. This process is performed in the order of S1301 to S1307, but S103 to 1305 form a loop. The second feature is a feature including a wide area feature of the input image, and is used as a clue when estimating the threshold value (θ) which is a discrimination parameter. The second feature is also generated based on the base feature F (x, y) like the first feature.

The second feature generation unit 103 first extracts the high-rise feature 109h having a second resolution coarser than the first resolution from the base feature F (x, y) (S1301). Then, the extracted high-rise feature 109h is aligned one-dimensionally (S1302). That is, rearrange. This corresponds to an arrangement of feature maps 403c in the example of FIG. At the time of alignment, the feature map 403c is stored in the RAM 14 so that it can be read out and used at any time.

Next, the second feature generation unit 103 performs feature conversion for the high-rise feature 109h in the loop from S1303 to S1305 by arithmetic processing of the fully connected layer of the neural network. In the arithmetic processing of the fully connected layer, weighting is performed for the feature 109h of the high layer arranged in one dimension. In the present embodiment, the weighting of the fully connected layer is learned in advance so that the image classification task can be determined by using the output layer shown as the image classification category 208 in FIG. Such a learning form is called "multi-task learning". Here, for example, learning of about 1000 classes of classification tasks is performed. After performing this multi-task learning, by performing feature conversion using the intermediate layer in front of the output layer, the features of the person and the person's eye size, hair color, and other rough image features to be detected are expressed. Image classification feature 115 can be obtained.

The second feature generation unit 103 next generates the wide area feature G by connecting the image classification feature 115 and the camera information input through the camera information input unit 110 (S1306). The camera information is information representing the conditions at the time of capturing the input image. As an example of the camera information, as shown in FIG. 10A, in the present embodiment, the focusing information value 1502, which is a numerical value of the focusing information 1501 indicating which region block the camera is focused on, is used. The in-focus information value 1502 is 0 in the focused area block, and is a + value according to the depth of focus otherwise. By adding this in-focus information value 1502 to the threshold value (θ) for each area block, for example, "the threshold value (θ) is raised because there are many false positives of the subject in the out-of-focus area". The threshold value (θ) can be set. After that, the second feature generation unit 103 stores the connected features as the wide area feature G, and finishes the second feature generation process (S1307).

As the camera information, in addition to the focusing information value 1502, the horizontal line information value 1504 relating to the positional relationship between the horizontal line estimation result 1503 and each area block as shown in FIG. 10B can also be used. If the camera has a gravity sensor, the horizon can be estimated from the detection information of the gravity sensor. Then, as a numerical value to be added to the threshold value (θ) for each region block, the region block above the horizon is set to -1, and the other region blocks are set to a positive value as they are closer. As another type of camera information, a photometric information value 1505 or the like related to the physical photometric value shown in FIG. 10C can also be used. It should be noted that FIGS. 10A to 10C are examples of camera information, and any kind of information can be used as a clue when estimating and setting an appropriate discrimination parameter (threshold value (θ)) for a subject. Camera information can be used. The camera information is stored in the external storage device 15 and is provided to the second feature generation unit 103 from the camera information input unit 110 at any time.

<Discrimination parameter setting>
Next, the process of setting the discrimination parameter shown in S14 of FIG. 3 will be described in detail. Here, an example will be described in which the discrimination parameter setting unit 104 calculates a threshold value (θ) used when detecting a person's face as a discrimination parameter and sets it for each area block.
The discrimination parameter setting unit 104 has three parameter regression devices shown in FIG. 9 so that an appropriate threshold value (θ) is set for each face size s and for each area block (i, j) of the input image. It includes 104a, 104b, 104c. The parameter regression device 104a is a regression device for a small face threshold value (face (small) threshold value). The parameter regression device 104c is a regression device for a large face threshold value (face (large) threshold value). The parameter regressor 104b is a regressor for an intermediate size face threshold (face (medium) threshold). The subscript of the area block is (i, j) in order to make the resolution lower than that of the area block (x, y) of the local feature.

For local features, position accuracy is important in addition to the likelihood score of the face, so it is necessary to perform discrimination in area block units (x, y) with higher resolution, but the threshold value (θ) by the discrimination parameter setting unit 104. ), The position accuracy is not so important. Rather, it is preferable to perform the estimation in units of region blocks (i, j) having a coarser resolution. The reason is that in the present embodiment, the threshold value (θ) is learned by a parameter regression device that is different for each region block of the input image. Therefore, if the resolution of the region block is too high, it is disadvantageous in terms of processing amount and memory consumption. .. Another reason is that overfitting is likely to occur because the number of cases during learning is reduced. One of the features of this embodiment is that the likelihood discrimination of the subject and the setting of the threshold value (θ) have a complementary relationship in this way.

The specific processing procedure by the discrimination parameter setting unit 104 is as follows. As shown in FIG. 7, the discrimination parameter setting unit 104 repeats processing for the face size s (where s ∈ {small, medium, large}) and each area block (i, j) on the input image (S1401 to S1401 to). S1405). The discrimination parameter setting unit 104 calculates a threshold value (θ _ijs ) for each face size s and each area block (i, j) using the extracted wide area feature G (S1403). The calculation is performed by the calculation of a general logistic regression device shown in the following equation.
[Number 4]
θ _ijs = 1 / (1 + exp {-W _ijs ^TG + b _ijs })

However, W is a weight parameter composed of a vector having the same length as the wide area feature G, and b is a bias value. Also. It is assumed that the value of the weight W and the bias value b are obtained by learning in advance. By the above calculation, the threshold value (θ _ijs ), which is a discrimination parameter, is calculated and set for each face size s and the area block (ij) of the input image.

An example of the threshold value (θ _ijs ) set by the processing of the discrimination parameter setting unit 104 is shown in FIG. In FIG. 9, the area block whose threshold value (θ _ijs ) is closer to 0 is shown white, and the area block whose threshold value (θ _ijs ) is closer to 1 is shown black. That is, the black region block whose threshold value (θ _ijs ) is closer to 1 suppresses the detection, and the detection is not performed unless the likelihood score is high.

<Subject discrimination>
Next, a detailed procedure example of the discrimination process shown in S15 of FIG. 3 will be described with reference to FIGS. 7 and 9. In FIG. 7, the subject discrimination unit 106 performs loop processing from S1501 to S1509 for each size s of a person's face. That is, the subject discrimination unit 106 generates a likelihood generation unit 106a that generates a face (small) likelihood score, a likelihood generation unit 106b that generates a face (medium) likelihood score, and a face (large) likelihood score. It functions as a likelihood generation unit 106c.
Each likelihood score is evaluation information indicating the probability that a face of size s exists in the region block as a candidate for a subject by inputting a local feature F _xy for each region block (x, y) (S1501 to S1504). .. Specifically, using a local feature F _xy as an input variable and using, for example, a support vector machine (SVM), a face likelihood score L _s (size s) for each region block (x, y) ( x, y) is calculated by the following formula.
[Number 5]
L _s (x, y) = R (Σ _k α _sk v _sk · F _xy + b _s )

v _sk is the k-th support vector for SVM to discriminate a face of size s, α _sk is the weighting coefficient of the support vector, and b _s is the bias term. It is assumed that these parameters are learned and stored in advance for each face size s by the method described later. R (・) is a normalized function for converting the output of SVM into likelihood. Here, the SVM score is simply standardized to 0 to 1 by a function like the following equation. Note that τ is a constant.
[Number 6]
Definition of normalization function z'= R (z):
z'= 1 if z ≧ τ
z'= z / τ if 0≤z <τ
z'= 0 if z <0

The SVM is one of two classes of pattern classifiers using a linear input element. The likelihood generation units 106a, 106b, 106c perform condition determination processing for all region blocks (x, y) and all face sizes s. Therefore, it is preferable to use a discriminator for light processing such as SVM. In other words, any discriminator such as a decision tree or a multi-layer perceptron can be used if the processing is light.

The likelihood generators 106a, 106b, 106c also compare the extracted L _s (x, y) values with the threshold (θ _ijs ). However, since the resolution of the area block (x, y) for which the likelihood score of the subject is calculated and the area block (i, j) for which the threshold value (θ _ijs ) is set are different, the coordinates are set as shown in the following formula. Compare with the threshold value (θ _i'j's ) of the area block obtained by conversion. In the following equation, δ and γ are the parameters of the coordinate transformation between the two region blocks.
[Number 7]
i'= δ _1x + γ ₁
j'＝ δ _2y ＋ γ ₂

Then, the conditions shown in the following formula are determined.
[Number 8]
L _s (x, y) ≧ θ _i'j's

In this way, the likelihood generation units 106a, 106b, 106c generate a face (small) likelihood score, a face (medium) likelihood score, and a face (large) likelihood score, as shown in FIG.
When each likelihood score is generated, the subject discrimination unit 106 makes a condition determination for each area block (x, y) by the threshold value processing unit 106d (S1506). That is, the threshold value processing unit 106d determines whether or not the likelihood score L _s (x, y) is the maximum value among the likelihood scores in the vicinity of 8. In addition, it is determined whether or not it is equal to or greater than the threshold value (θ _i'j's ). If it is affirmative (S1506: Yes), it is determined that a face of size s exists at a position centered on the coordinates (x, y) of the region block (i-th region block) satisfying the condition (S1507). That is, the face of that size s is determined to be a candidate for the subject shown in FIG. After that, the process returns to S1505.
On the other hand, if the result of S1506 is negative (S1506: No), the process immediately returns to the process of S1505. In S16 of FIG. 3, the result of the determination by the subject discrimination unit 106 is output to the result output unit 107 as the final detection result 121.

<Learning process>
Next, the learning process performed by the information processing apparatus will be described with reference to FIGS. 11 to 14. FIG. 11 is a functional block configuration diagram of the information processing apparatus when performing learning processing. A camera information holding unit 130, a learning image holding unit 131, a teacher value holding unit 132, a likelihood totaling unit 133, and a discrimination parameter teacher value holding unit 134 are added to the functional block configuration diagram of FIG. The base feature generation unit 108 is not shown.

With reference to the procedure explanatory diagram of the learning process shown in FIG. 12, the image input unit 101 acquires the learning image from the learning image holding unit 131. Further, the likelihood generation units 106a, 106b, and 106c acquire the teacher value of the face corresponding to each image from the teacher value holding unit 132. Here, the teacher value of the face of the person corresponding to each image is used, but it is not necessarily limited to the face. FIG. 14A shows the learning image set X, and FIG. 14B shows the learning image set Y. Each of the training image sets X and Y consists of a training image shown in row (1) of each figure and a set of facial teacher values shown in row (2) of the figure. The face teacher value is a value indicating whether the area block includes, or does not include, a small-sized face, a medium-sized face, or a large-sized face. Specifically, the area block including the reference point of the face is set as the teacher value of the positive case (with a face) “1”, and the other area blocks are set as the teacher value of the negative case (without a face) “0”. Each teacher value may be conveniently given by the operator via the input device 21 for each face size, but it may be performed by an automatic recognition process. The reference point of the face is the center of gravity of the face area, but this is not the case.

The first feature generation unit 102 generates a local feature from each learning image (S112). The content of the process for generating the local feature is the same as the process for S12.
First, the likelihood generation units 106a, 106b, 106c learn the likelihood determination SVM as follows so that the subject (face) can be correctly determined based on the local feature (S113). That is, the likelihood generation units 106a, 106b, 106c use the teacher values "0" and "1" for a given face as the target variable, and the connected layer feature amount F _xy of the corresponding region block as the explanatory variable. , Learn SVM so that positive and negative cases of the face can be discriminated. Learning is performed for each size s, and SVM parameters ([v _sk , α _sk , b _s ]) are obtained. The above is the learning procedure of the likelihood generation unit 106a.

After that, the information processing apparatus learns the regression units 104a, 104b, 104c of the discrimination parameter setting unit 104 (S114). The regressors 104a, 104b, 104c are parameter regressors Φ _ij provided for each region block of the image. Each of the parameter regression units Φ _ij is a logistic regression unit according to the above-mentioned equation (2). The purpose of the learning is to obtain the weight parameters ([W _ij , bi _j ]) of the regressor Φ _ij so that these logistic regression devices can estimate an appropriate threshold (θ _ij ) according to the wide area characteristics of the input image. That is. Here, for the sake of simplicity of explanation, the face size s will not be considered from now on, and all of them will be treated as the same face. The learning operation is essentially the same even when the size s is taken into consideration, and the learning procedure described below may be simply performed for each size.

FIG. 13 is an explanatory diagram of the learning procedure, and executes a loop of processing from S1141 to S1149. FIG. 15 is a conceptual explanatory diagram of the learning process. Referring to FIGS. 13 and 15, the image input unit 101 randomly samples n images from a plurality of images held in the learning image holding unit 131 and selects a batch set 501 (S1142).
The likelihood generation unit 106a uses SVM for this batch set 501 to generate a likelihood map in which the likelihoods of the subjects of each learning image are arranged (S1143).
The likelihood aggregation unit 133 extracts and stores the position of a local peak in the likelihood map (S1144). Then, the likelihood aggregation unit 133 aggregates the peaks and sets the score of the peak likelihood of the positive case (there was a face at the peak position) or the negative case (there was no face at the peak position) for each region. Aggregate to generate a distribution of likelihood scores (S1145). The results aggregated in this way are the likelihood score distributions 503a and 503b shown in FIG. As shown as the likelihood score distributions 503a, 503b, the likelihood scores are aggregated for each position in the image, that is, for each region block (i, j). The thin line curve of the likelihood score distributions 503a and 503b is the score distribution of each image, and the thick line curve is the average score distribution of the batch.

Generally, when a feature is discriminated from an image pattern, the accuracy is relatively good at the center of the image, and the recognition accuracy is lowered because a part of the pattern is hidden at the edge of the image. That is, as shown in FIG. 15, the score distribution of the positive case and the negative case changes depending on the location of the image or the area block. In the present embodiment, an appropriate target threshold value (θ _ij ) is set for each region block (i, j) of the image of each batch set 501. For example, a threshold value that can reject 95% of negative cases in the batch set 501 is set as a target threshold value (θ ^) so that the false detection rate of face detection can be suppressed to less than a certain level. For example, a relatively low threshold of θ ^ _3,2 = 0.4 for the parameter regressors Φ 3, ₂ near the center of the image, and θ ^ ₄ for the parameter regressors Φ ₄ , 3 at the corners of the image. _{, 3} = 0.7, which is a high threshold value, is given as the target value for regression estimation (S1146).

Next, the second feature generation unit 103 generates a wide area feature 505 of the image of each batch of the batch set 501 (S1147). Next, the second feature generation unit 103 uses the wide area feature 505 as an explanatory variable, and the weight parameter ([] so that the threshold value (θ _ij ) close to the target threshold value (θ ^ _ij ) can be calculated by the above-mentioned equation of equation 4. W _ij , b _ij ]) is obtained. To do this, the total value E of the errors between the target value θ ^ _ij and the estimated value θ _ij may be calculated for each batch, and the weight parameter may be updated in the gradient direction to reduce this. Specifically, the weight value is minutely updated as shown in the following equation using a gradient method widely known in machine learning (S1148).
[Number 9]
W _ij ^{t + 1} = η∂E / ∂W _ij ^t + W _ij ^t
b _ij ^{t + 1} = η∂E / ∂b _ij ^t + b _ij ^t

However, η is a minute coefficient. The above update operation is repeated a predetermined number of times, and the obtained weight parameter ([W _ijm , _bijm ]) is used as the parameter of the parameter regressionr Φ _ij .
The parameter regressor Φ _ij learned in this way is trained so that an appropriate target threshold value can be set for each region block according to the wide area characteristics of the image.

As described above, the above learning process is performed as a parameter regression device Φ _ijs for each face size s. As a result, for example, in an image having a messy background, a threshold value (θ) is set so as to suppress the detection of a small-sized face that is easily erroneously detected. In addition, for images with image features such as portrait photographs, it is considered that there are few cases of failure of a large-sized face near the center, so the threshold value near the center is set low and the large-sized face is set. Is more likely to be detected.
The above is the content of the learning process of the subject discrimination unit 106 and the discrimination parameter setting unit 104.

<Modification example>
As another derivative of the present embodiment, the first feature is generated using only the low-rise feature map 403a of the CNN, and the second feature is generated using only the high-rise feature map 403c. You may. Further, as a form of features other than the neural network, conventional image features such as a color histogram and SIFT features can also be applied. Further, the feature of the present embodiment is calculated by using a rectangular block area as an element unit, but the generation unit of the first feature is not limited to the rectangular block shape. For example, the likelihood score of the subject may be calculated for each region in a region called a super pixel obtained by grouping the pixels according to the closeness of color tones.
Further, in the present embodiment, the discrimination parameter setting unit 104 sets an appropriate threshold value for each area block and for each face size, but it is also conceivable that the threshold value learning / estimation is not performed for each area block. That is, one threshold may be obtained for each face size for the entire image, not for the area block. Furthermore, a derivative form such as not dividing into face size is also conceivable. Further, the target value of the threshold value given to the discrimination parameter setting unit 104 is determined on the basis that the false detection rate satisfies a predetermined condition, but this is not the case. As another derivation, it is conceivable to adopt a form such as a standard that the precision is equal to or higher than a predetermined value or a standard that minimizes Bayesian error.

[Second Embodiment]
In the second embodiment, it will be described that it can be applied to other tasks such as semantic region division. In this case, as the discrimination parameter, the weighting coefficient β of the score balance when integrating the recognition results is set instead of the threshold value (θ) as in the first embodiment. The information processing apparatus of the second embodiment has the same hardware configuration as that of the first embodiment, but the functional block configuration realized by the computer program is different from that of the first embodiment.

FIG. 16 is a functional block configuration diagram of the information processing apparatus of the second embodiment. In the information processing apparatus of the second embodiment, the image input unit 201 captures a necessary input image from the captured image. In addition, the first feature generation unit 202 generates the first feature including the feature of the local region of the captured input image. In addition, the second feature generation unit 203 generates a second feature including a wide area feature of the captured input image. Further, the discrimination parameter setting unit 204 sets the discrimination parameter from the second feature. Further, the subject discrimination unit 206 discriminates the subject in the image based on the first feature and the set discriminant parameter. Further, the result output unit 207 outputs the discrimination result. Further, the base feature generation unit 208 generates and stores the base feature information for generating the first feature and the second feature. Further, the camera information input unit 210 performs a process of inputting camera information to the base feature generation unit 208. In this embodiment, a region category determination unit 211 that newly determines for each category of the region block is provided.

As shown in FIG. 16, the base feature generation unit 208 generates an RGB histogram 208a and a SIFT (Scale-Invariant Feature Transform) feature 208b as base features. Further, from the camera information input unit 210, camera information that is effective when determining the category of the area block, such as the horizon information value 208c and the photometric information value 208d, is also taken in as a base feature. Here, the focusing information value 1502, the horizontal line information value 1504, and the photometric information value 1505 shown in FIGS. 10A to 10C are used.

The area category determination unit 211 calculates the likelihood score of the area category of each area block of the input image. Area categories represent, for example, the attributes of the sky, ground, buildings, and other objects. The teacher values of the area category are as shown in line (3) of FIGS. 14 (A) and 14 (B). It is the purpose of the task according to the second embodiment to determine such an area category.

In the second embodiment, unlike the first embodiment, the base feature generation unit 208 does not use the neural network. Further, the area category determination unit 211 is composed of an empty area determination unit 211a, a ground area determination unit 2011b, a building area determination unit 211c, and an object area determination unit 211d. These determination units 211a to 211d generate a map in which region categories are scored based on the first feature generated by the first feature generation unit 202, that is, a region category score map. This area category score map is used by the subject discrimination unit 206 and the second feature generation unit 203.

An example of the processing procedure of the subject discrimination method in the second embodiment will be described with reference to FIGS. 17A and 17B. First, similarly to the first embodiment, the base feature generation unit 208 generates a base feature from the input image captured by the image input unit 201 (S21, S22). The first feature generation unit 202 extracts the local feature and the wide area feature from the base feature and generates the first feature (S23).
The area category determination unit 211 generates an area category score map (S24). The detailed processing procedure of S24 is shown in FIG. 17 (B). That is, the area category determination unit 211 calculates the likelihood score of the category of the area block representing the four types of attributes of the sky, the ground, the building, and other objects by using, for example, the above-mentioned SVM. Then, these likelihood scores are arranged to generate a region category score map (S2401 to S2405). The second feature generation unit 203 generates the second feature from the base feature and the area category score map (S25).

The discrimination parameter setting unit 204 estimates the parameter β = [β ₁ , β ₂ , β ₃ , β ₄ ] of the weighting coefficient of the region category (S26). Here, a logistic regression device is prepared for each region category, and the explanatory variables are used as the wide area feature G, and the coefficient β of the balance between appropriate categories is estimated by the following equation.
[Number 10]
β _c = 1 / (1 + exp {-W _c ^T · G + b _c }) (c = 1, ..., 4)

In the subject discrimination unit 206, the likelihood generation unit 206a multiplies the estimated weight β = [β ₁ , β ₂ , β ₃ , β ₄ ] by the region category score map of each category to adjust the likelihood. Calculate the score (S27).
[Number 11]
L' _c (x, y) = β _c · L _c (x, y) (c = 1, ..., 4)

In the above logistic regression device, the weight parameters [W _c , b _c ] are adjusted in advance so that an appropriate coefficient βc can be obtained. Specifically, the adjusted likelihood L' _c (x, y) is adjusted by the gradient method or the like so as to reduce the error with the teacher value (binary value of [0, 1]) of each category on average. do it. At this time, unlike the first embodiment, it is conceivable to connect the score map of the area category to the wide area feature and use it. By adding such a device, it becomes possible to learn in consideration of the susceptibility to false detection between each area category. For example, co-occurrence between categories, such as buildings and objects being easily mistaken for each other, can be considered as a kind of information, and appropriate weighting coefficient β can be learned. For example, if both the building and the object have high likelihood scores, learn to lower both weighting factors.
The threshold processing unit 206d performs threshold processing for the area category (S28). The result output unit 207 outputs the determination result to, for example, the monitor 22 and displays it (S29).

<Modification example>
As a modification of the second embodiment, instead of multiplying the weighting coefficients for each region category, for example, using a matrix βMAT having 4 × 4 parameters, the results of the score maps of all categories are mixed and adjusted. You may ask for a degree score. The likelihood L (x, y) of the four categories of the area block (x, y) in this case can be calculated by the following equation. This equation corresponds to the likelihood of four categories of region blocks (x, y) arranged as a vector of 4 × 1 size.
[Number 12]
L'(x, y) = β _MAT · L (x, y)

[Third Embodiment]
A third embodiment of the information processing apparatus will be described. In this embodiment, it will be described that it is applicable not only to the fixed area block as described above but also to the task based on the irregular area block. The hardware configuration of the information processing apparatus of the third embodiment is the same as that of the first embodiment, and the functional block configuration realized by the computer program is different. FIG. 18 shows a functional block configuration diagram of the information processing apparatus of the third embodiment. The difference from the first and second embodiments is that the subject is discriminated by focusing on the region of interest (ROI) in which the human pays attention to the image, instead of focusing on the region block.

As shown in FIG. 18, the information processing apparatus of the third embodiment includes a feature generation unit 302, a ROI selection unit 304, an ROI feature extraction unit 306, a subject likelihood generation unit 307, a region likelihood generation unit 309, and a discrimination parameter. It has the functions of the setting unit 311 and the threshold processing unit 315. It is considered that the type of the subject to be detected and the appropriate threshold value differ depending on the imaging mode, and these are functions for dealing with this.

The information processing apparatus of the third embodiment also has an image pickup mode instruction unit 312 which is an interface on the camera side such as a landscape mode and a tracking image pickup mode and instructs an image pickup mode selected by the user. Further, it has an SVR coefficient holding unit 313 that holds the coefficient of the support vector regression device (SVR) used in the discrimination parameter setting unit 311.

The method of discriminating the subject in the third embodiment will be described with reference to FIG. Here, an example is shown in which the subject to be detected is a specific person. The information processing apparatus first captures an input image and performs processing for generating local features of the input image (S31 to S32). The contents of these processes are the same as those in the first embodiment.
The ROI selection unit 304 selects a plurality of ROIs having a high “personality” and generates candidate regions corresponding to each (S33). The ROI feature extraction unit 306 detects the position (coordinates (x, y)) and size s for each ROI, and calculates and aggregates the feature amount of each ROI (S34, called ROI pooling process). The subject likelihood generation unit 307 calculates the subject likelihood (likelihood of a person) by the same method as in the first and second embodiments, and generates detection candidates based on the calculation result (S35).

Further, the region likelihood generation unit 309 estimates the likelihood of region categories (attributes) such as lawn, crowd, and ball for each region block using the local features generated by the feature generation unit 302, and the region category likelihood. Generate degrees (S36). The discrimination parameter setting unit 311 obtains the SVR coefficient, which is a weight parameter that has been learned in advance for each mode, from the SVR coefficient holding unit 313 according to the mode instructed by the operator of the information processing device in the image pickup mode instruction unit 312, for example. Read (S37). The instructed mode includes, for example, a landscape mode, a macro mode, a tracking mode, a portrait mode, and the like.

The discrimination parameter setting unit 311 has the ROI size s extracted by the ROI feature extraction unit 306, the coordinates (x, y) of the ROI, and the likelihood 310 of the region category in the ROI generated by the region likelihood generation unit 309. A discriminant parameter is generated based on (S38). The discrimination parameter of the present embodiment is a threshold value (θ _ROI ) used for discrimination of each ROI (S38). The threshold value processing unit 315 compares the threshold value (θ _ROI ) with the likelihood of the detection candidate generated in S35, that is, performs threshold value processing, determines whether the detection candidate is a subject (person) (S39), and determines the result. Output (S40). The information processing apparatus determines a person by such a form of processing in consideration of information such as "a small-sized person around the ball is likely to be a subject to be detected". Can be done.

SVR is a method of performing regression learning based on maximizing the margin of the support vector. In order to estimate the optimum threshold value (θ) by SVR, first, a target threshold value (θ ^) is prepared for each learning case. First, as shown in FIG. 20 (A), consider the distribution of positive and negative cases of a person in a feature space whose feature dimension is the position and size of the ROI. In each case, the value (L) of the likelihood of the person generated by the subject likelihood generation unit 307 is attached in advance.

The discrimination parameter setting unit 311 sets an optimum threshold value (θ ^), which is a teacher value of SVR, for each case as follows. Specifically, an appropriate threshold value (θ ^) for discriminating a case in the vicinity of k around the case (a region of k (predetermined constant) adjacent to the region of the case) is set for each case. Here, for the positive case 801 in FIG. 20 (A), the likelihood scores of the surrounding cases are referred to, and θ is taken so that there is no negative case that exceeds the threshold value (θ ^) in any of the neighboring cases. ^ = 0.5 is set as the target threshold value. Similarly, a threshold value (θ ^) is set for all cases. An example of the set result is shown in FIG. 20 (B).
The set of threshold values (θ ^) thus obtained becomes the teacher value at the time of learning SVR. In the SVR, as shown in FIG. 20C as the input example 803, the input example enters at a position where the example does not exist. Therefore, the coefficient of SVR is learned based on the standard of maximizing the margin so that an appropriate threshold value can be estimated with robustness. The above is the learning method of SVR.

Further, the same method can be used when estimating a feature in a higher dimension as a feature amount of an explanatory variable as shown in FIG. 20 (D). Also. As a method for estimating such a threshold value, a method such as kernel density estimation may be used in addition to SVR, and the method is not limited to a specific method.
It is also possible to prepare a plurality of sets of learning data and teacher values according to the imaging mode selected by the operator and learn each. This makes it possible to perform learning such that the detection of a large-sized person is suppressed in the tracking mode as compared with the other modes.

In the third embodiment, unlike the first embodiment, it can be applied to a recognition method that is not based on a fixed area block. In particular, regarding the setting of the discrimination parameter, the discrimination parameter can be set in consideration of the continuous value such as the position and the size by using a method such as SVR.

The present invention is also realized by executing the following processing. That is, a computer program that realizes the functions of each of the above-described embodiments is supplied to the system or device via a network or various storage media. The present invention can also be carried out by a process in which a computer (or CPU, etc.) of the system or device reads out and executes a computer program. In this case, the computer program and the recording medium in which the computer program is stored constitutes the present invention.

Claims (18)

A base feature generation means that generates a base feature that is the basis for image feature extraction,
A first feature generating means for generating a first feature of the first resolution with respect to a local region of the image based on the base feature .
A second feature generation means for generating a second feature of the image, which is a wide area wider than the local region and has a second resolution coarser than the first resolution, based on the base feature .
Based on the second feature, a setting means for setting a discrimination parameter for discriminating a subject existing in the image for each partial region of the image, and a setting means.
A discriminating means for discriminating the subject based on the first feature and the discriminating parameter, and
Characterized by
Information processing equipment. The discriminating means is characterized in that the likelihood that the subject is included in each of the partial regions is generated based on the first feature, and the subject is discriminated based on the likelihood and the discriminating parameter. do,
The information processing apparatus according to claim 1. A first feature generating means for generating a first feature relating to a local region of an image,
A second feature generation means for generating a second feature for a wide area of the image, which is wider than the local area.
Based on the second feature, a setting means for setting a plurality of discrimination parameters corresponding to each of the plurality of types of subjects existing in the image for each partial region of the image, and a setting means for discriminating the plurality of types of subjects.
A plurality of likelihoods corresponding to each of the plurality of types of subjects are generated for each of the partial regions based on the first feature, and the plurality of types are based on the plurality of likelihoods and the plurality of discrimination parameters. It is characterized by comprising a discriminating means for discriminating a subject.
Information processing equipment. The setting means sets a plurality of discrimination parameters corresponding to each of the plurality of sizes for each of the partial areas.
The discrimination means generates a plurality of likelihoods corresponding to each of the plurality of sizes for each of the partial regions based on the first feature, and the discrimination means is based on the plurality of likelihoods and the plurality of discrimination parameters. Characterized by discriminating the subject,
The information processing apparatus according to claim 1 or 2. The base feature generation means generates a feature map of a plurality of layers by performing a plurality of convolution operations on the image with a neural network, and integrates the feature maps of these layers to generate the base feature. Characterized by generating,
The information processing apparatus according to claim 1 or 2 . The base feature generation means is characterized in that the resolutions of the feature maps of the plurality of layers are matched and the feature maps of the plurality of layers are integrated.
The information processing apparatus according to claim 5. The first feature generation means generates the first feature by converting the feature map of the layer having the first resolution among the feature maps of the plurality of layers.
The second feature generation means is characterized in that the second feature is generated by converting the feature map of the second resolution layer.
The information processing apparatus according to claim 5 or 6. Further provided with an acquisition means for acquiring camera information representing the conditions at the time of capturing the image,
The second feature generation means is characterized in that the second feature is generated based on the camera information.
The information processing apparatus according to any one of claims 1 to 7. The second feature generation means is characterized in that the second feature is generated by linking the image classification feature, which is a preclassified image feature, with the camera information.
The information processing apparatus according to claim 8. The camera information is characterized by being in-focus information regarding an in-focus region, horizon information regarding a horizon estimation result, or photometric information regarding a photometric value.
The information processing apparatus according to claim 8 or 9. The setting means is characterized in that the discrimination parameter is set for each of a plurality of sizes by using the image classification feature.
The information processing apparatus according to claim 9. The discriminant parameter is a threshold value determined in advance by learning.
The information processing apparatus according to any one of claims 1 to 11. A determination means for determining the attribute of the area is provided for each of the partial areas.
The second feature generation means is characterized in that the second feature is generated based on the determination result by the determination means.
The information processing apparatus according to any one of claims 1 to 3. The setting means is characterized in that the determination parameter is set based on the determination result by the determination means.
The information processing apparatus according to claim 13. A determination means for determining the likelihood of the attributes of each of the plurality of region blocks constituting the image based on the first feature.
Specific means for identifying the partial area, which is a region of interest, and
Further provided with a feature extraction means for detecting the position and size of each of the partial regions specified by the specific means and extracting the features of each of the partial regions.
The setting means is characterized in that the discrimination parameter is set for each partial region based on the position and size of each partial region and the determination result by the determination means.
The information processing apparatus according to claim 1. Steps to generate the base features that are the basis for image feature extraction,
A step of generating a first feature of the first resolution for a local region of the image based on the base feature .
A step of generating a second feature of the image, which is a wide area wider than the local region and has a second resolution coarser than the first resolution , based on the base feature, and the second feature. A step of setting a discrimination parameter for discriminating a subject existing in the image for each partial region of the image based on the above.
It is characterized by having a step of discriminating the subject based on the first feature and the discriminating parameter.
How to identify the subject. The steps to generate the first feature for the local area of the image,
A step of generating a second feature of the image for a wide area wider than the local area.
A step of setting a plurality of discrimination parameters corresponding to each of the plurality of types for discriminating a plurality of types of subjects existing in the image based on the second feature for each partial region of the image.
A plurality of likelihoods corresponding to each of the plurality of types of subjects are generated for each of the partial regions based on the first feature, and the plurality of types of subjects are generated based on the plurality of likelihoods and the plurality of discrimination parameters. It is characterized by having a step of discriminating.
How to identify the subject. A computer program for operating a computer as the information processing apparatus according to any one of claims 1 to 15.

Images