JP2012128638A - Image processing device, alignment method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable extraction of an image of a deformable target and correction of a position of the target based on an extraction result.SOLUTION: An image processing device comprises: image acquisition means that acquires an image; feature extraction means that extracts a feature amount from the image acquired by the image acquisition means; likelihood calculation means that calculates a likelihood of a target based on a model of the target and the feature amount of the image extracted by the feature extraction means; detection means that detects a detection area of the target in the image based on the likelihood calculated by the likelihood calculation means; discrimination means that discriminates between a part in which the target varies and a part in which the target does not vary based on the model and the detection area detected by the detection means; and alignment means that performs alignment between the model and the detection area based on the part in which the target does not vary discriminated by the discrimination means. Thereby the problem is solved.

Description

  The present invention relates to an image processing apparatus, an alignment method, and a program.

  In Patent Document 1, an image serving as a template is prepared, and points are extracted from the image. Then, by extracting points in the same way for another image, matching with the points of the prepared template image to obtain a coefficient for converting the image, normalizing the image, and taking the difference between the images to determine whether they are the same A method for collecting images is disclosed.

JP 2006-309259 A

  However, the technique disclosed in Patent Document 1 normalizes an image using three or more sets of corresponding points. Therefore, it can be applied to an image of a target such as a face that does not deform so much, but when applied to an image that includes a displacement such as a hand or a leg, the object in the image is deformed by normalization processing. However, there is a problem that the region of the target object cannot be extracted from the image.

  The present invention has been made in view of such problems, and an object of the present invention is to extract a target image to be deformed and to correct the position of the target based on the extraction result.

  Therefore, the image processing apparatus of the present invention is extracted by an image acquisition unit that acquires an image, a feature extraction unit that extracts a feature amount from the image acquired by the image acquisition unit, a target model, and the feature extraction unit. A likelihood calculating means for calculating the likelihood of the object based on the feature amount of the obtained image, and a detecting means for detecting a detection region of the object in the image based on the likelihood calculated by the likelihood calculating means. Based on the model and the detection area detected by the detection means, an identification means for identifying a portion where the object is changing and a portion where the object is not changing, and the object identified by the identification means is changing Alignment means for aligning the model and the detection region based on a portion that is not.

  According to the present invention, it is possible to extract an image of an object to be deformed and correct the position of the object based on the extraction result.

It is a figure which shows an example of an image processing apparatus. 1 is a diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment. It is a figure which shows an example of the processing process by a learning method. It is a figure which shows an example of the processing process by a learning method. It is a figure which shows an example of the processing process by a learning method. It is a figure which shows an example of the processing process by a learning method. It is a figure which shows an example of the processing process by a learning method. It is a figure which shows an example of the processing process by a learning method. 6 is a flowchart illustrating an example of a processing process according to the alignment method and the learning method of the first embodiment. 6 is a diagram illustrating a functional configuration of an image processing apparatus 100 according to Embodiment 2. FIG. It is a figure which shows the result of the alignment with the area | region of each detection result, and a model. It is a figure which shows the result of the alignment with the area | region of each detection result, and a model. It is a figure which shows the result of the alignment with the area | region of each detection result, and a model. It is a figure which shows the result of the alignment with the area | region of each detection result, and a model. 10 is a flowchart illustrating an example of a processing process according to the alignment method and the learning method of the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Embodiment 1>
The learning method according to the present embodiment is a learning method for extracting an object appearing in an image or video, collecting a suitable image for learning the extracted object, and calculating a position. It can be a camera or network camera, and can also be used for pre-captured images / videos.
In the present embodiment, a case will be described in which an image is captured in an environment where an object is present, and a person to be detected is detected from the obtained image. Although the detection target according to the present embodiment is a person, the present invention is not limited to this. This method can be used if an object is detected from a video in another environment. Moreover, although it is set as the image in this embodiment, you may apply to an image | video other than that.

  FIG. 1 is a diagram illustrating an example of the image processing apparatus 100. As shown in FIG. 1, the image processing apparatus 100 includes a CPU 1, a RAM 2, a ROM 3, and an HD 4. The CPU 1 executes processing based on a program stored in the ROM 3 or the HD 4 or the like, thereby realizing functions of the image processing apparatus 100 described later and processing related to flowcharts described later. The RAM 2 stores data used when the CPU 1 executes a program. The ROM 3 stores a boot program that is read when the image processing apparatus 100 is activated. The HD 4 stores a program and the like according to the present embodiment. Although omitted for simplification of description, the image processing apparatus 100 includes hardware related to an interface that controls communication with an imaging unit described later.

FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus 100 according to the first embodiment.
As illustrated in FIG. 2, the image processing apparatus 100 includes an image acquisition unit 101, a feature extraction unit 102, a likelihood calculation unit 103, a target detection unit 104, a corresponding point extraction unit 105, and an alignment unit 106. And a learning unit 107.
The image acquisition unit 101 acquires an image from a camera or a previously captured video. The acquired image 210 is output to the feature extraction unit 102.
The feature extraction unit 102 acquires the image obtained from the image acquisition unit 101 and extracts a feature amount from the acquired image. In the present embodiment, HOG is used for feature extraction, but pixel values are extracted as they are from an image, but regions may be extracted as templates.
Regarding HOG,
Reference: see "histograms of orientated gradients for human detection, N. Dalal, CVPR 2005".
Further, the feature extraction unit 102 may extract a region using a known technique such as Haar-Like or Edgelets.
Regarding Haar-Like,
Reference: See “P. Viola and M. Jones,“ Rapid Object Detection using Boosted Cascade of Simple Features ”, CVPR2001”.
And for Edgelets,
References: “See Bo Wu and Ram Nevaia,“ Segmentation of Multiple, Partially Occluded Objects by Grouping, Merging, Assigning Part Detection Responses, ”7 IJC.
The feature amounts extracted by the feature extraction unit 102 for the object 230 in the image are 231 to 235. FIG. 3 is a diagram illustrating an example of a processing process according to a learning method. Similarly, the feature amounts extracted from the other objects 240, 250, and 260 are 241 to 245, 251 to 255, and 261 to 265, respectively. The extracted feature quantities 231 and the like are sent to the likelihood calculating unit 103.

The likelihood calculating unit 103 calculates the likelihood by comparing with a target model (target model) 220 to be detected based on the feature quantity 231 of the image obtained from the feature extracting unit 102 or the like. In this embodiment, the likelihood is calculated using the target model 220 previously learned by Boosting as a method for identifying the target.
Regarding Boosting,
References: See “Y. Freund and Roberts E. Schapier,“ A decision-theoretic generalization of on-line learning and an application to boosting, 1997 ”. The vessel has a feature that represents a part of the object. A plurality of weak classifiers have different characteristics, thereby forming a target model. The image processing apparatus 100 may create a target model from an image using an arbitrary template, or prepare an image of a target to be detected and learn using a known learning technique such as Support Vector Machine. You may create a model.
Regarding Support Vector Machine,
Reference: “N. Christianiani and J. Shawe-Talor,“ An Induction to Support Vector Machines, University University Press, 2000 ””
At this time, the model may be an image or an image feature amount. In the likelihood calculation method, the likelihood calculation unit 103 slides the model and compares a part of the features representing the model of the weak classifier with the features extracted from the images at the position of each image. The likelihood calculating unit 103 outputs a high value if they are the same feature (a feature having the same gradient direction if HOG), and a low value if they are not the same, and is obtained by learning of the weak classifier Calculate the weighted value. The likelihood calculating unit 103 calculates a value for the feature extracted from the image by each weak classifier, and calculates the sum of the output values as the final likelihood. The calculated likelihood is sent to the target detection unit 104.
Based on the likelihood calculated from the likelihood calculation unit 103, the target detection unit 104 identifies a position and a region of a target having a higher likelihood than an arbitrary threshold value set in advance. As a result of the identification, the extracted target areas are indicated by 330 to 360. The obtained target detection results 330 to 360 are output to the corresponding point extraction unit 105. FIG. 4 is a diagram illustrating an example of a processing process according to a learning method.

The corresponding point extraction unit 105 extracts corresponding points corresponding to the feature of the model in the obtained target detection area (330 to 360). The corresponding point extraction unit 105 has the same features 431 to 435 and 441 as the features obtained by extracting the target model from the image based on the new features (corners and the like) 421 to 425 of the model region and the target detection region. ˜445, 451˜455, 461˜465 (corresponding points) are extracted.
In addition, the corresponding point extraction unit 105 stores the feature amounts 231 to 235, 241 to 245, 251 to 255, and 261 to 265 at the time of detection of each target extracted by the feature extraction unit 102 in the target detection region (330 to 360). You may make it match with the characteristic (421-425) which a model has and uses. The feature of the obtained model is associated with the feature of each detection area. The corresponding point extraction unit 105 uses the result of the likelihood calculation unit 103 for the feature association. The likelihood of each feature quantity (441 and 442 shown in FIG. 5) is calculated at the time of object detection, but the feature with a high likelihood at the time of detection does not vary much compared to the shape of the model, and the feature with a low likelihood is It is thought that it fluctuates from the shape of the model. The weak classifier having each feature (421 to 425) of the model has a weight. The corresponding point extracting unit 105 associates the model with the features (431 to 435) in the target detection region so that the feature position having a high likelihood and the feature position of the weak classifier having a high weight are preferentially matched. . Similarly, the corresponding point extraction unit 105 associates features (441 to 445, 451 to 455, 461 to 465) in a target detection region different from the model with features (421 to 425) in the model.

FIG. 5 is a diagram illustrating an example of a processing process according to a learning method.
Corresponding points obtained in the target detection area are output to the alignment unit 106.
The alignment unit 106 aligns the images of the regions 330, 340, 350, and 360 detected based on the corresponding points obtained from the corresponding point extraction unit 105 and the target model 220. The alignment unit 106 calculates the alignment based on the correspondence relationship output from the corresponding point extraction unit 105. For example, when using one obtained corresponding point, the alignment unit 106 performs alignment so that the corresponding points overlap. In addition, when using the two corresponding points obtained, the alignment unit 106 performs alignment so that corresponding points with high likelihood of the two points overlap preferentially, or the sum of the distances of the corresponding points is minimized. Align so that it becomes. In addition, when the alignment unit 106 performs alignment using three or more corresponding points, in addition to parallel movement between the model and the detection region, the detection unit 106 can correspond to rotation, scale change, and affine deformation of the detection region. An affine transformation matrix is calculated based on the positional relationship of the obtained corresponding points. The alignment unit 106 solves the following equation when calculating the transformation matrix.

In the above equation, x ′ and y are feature point position coordinates in the model, and x and y are feature point position coordinates in the target detection region.
The alignment unit 106 may output the calculated conversion matrix as an alignment result. Results of alignment (results of superimposing the detected image and the target model) are shown in 436, 446, 456, and 466 in FIG. The aligned results 436, 446, 456, 466 are output to the learning unit 107. In FIG. 5, the model and each detection area are shown to be individually overlapped, but the model and all detection area results may be overlapped.

The learning unit 107 extracts the target detection result from the image based on the results 436, 446, 456, and 466 of the alignment unit 106, and learns by matching the position of the model obtained by the learning so far. For learning the model, the learning unit 107 extracts an image feature amount from the obtained target detection region, and learns the image feature in the newly obtained detection target region with respect to the feature of the model. This makes it possible to generate a model in which the probability distribution of the weak classifier having a feature that changes due to learning is updated while maintaining the feature whose shape of the target does not change with respect to the model learned so far. For example, as a result of learning 738, models with different body orientations are generated, and at 748, models with different leg positions are generated. The learning unit 107 may perform learning using another known learning technique such as SVM or a sequential learning method.
The above is the configuration part related to the present embodiment.

Next, processing performed by the image processing apparatus 100 according to the present embodiment will be described using the flowchart shown in FIG.
FIG. 6 is a flowchart illustrating an example of a process performed by the alignment method and the learning method according to the first embodiment.
When the entire process is started, first, in step S101, the image acquisition unit 101 acquires an image from a camera or a previously captured video. The acquired image is sent to the feature extraction unit 102, and the process proceeds to step S102.
In step S <b> 102, the feature extraction unit 102 extracts an image feature amount from the image acquired by the image acquisition unit 101. A known technique is used as an image feature amount extraction method. The extracted features are sent to the likelihood calculating unit 103, and the process proceeds to step S103.
In step S103, the likelihood calculating unit 103 calculates the likelihood of a target to be detected from the image based on the feature extracted in step S102. At this time, for example, the likelihood calculating unit 103 creates a model to be detected in advance using a known learning technique such as SVM or Boosting. The calculated likelihood is sent to the target detection unit 104, and the process proceeds to step S104.
In step S104, the target detection unit 104 identifies a position / region where the target is in the image based on the likelihood in the image calculated in step S103. If it is identified that there is a target, the target position / region, image, and feature are sent to the corresponding point extraction unit 105, and the process proceeds to step S105. If no target is detected, the entire process ends.

In step S105, the corresponding point extraction unit 105 first extracts a feature point from the detection target model. Next, the corresponding point extraction unit 105 extracts a feature point corresponding to a point that is a feature in the model with respect to an area identified as having a target in the image by the target detection unit 104, and calculates a point-to-point Take correspondence. The obtained corresponding points are sent to the alignment unit 106, and the process proceeds to step S106.
In step S <b> 106, the alignment unit 106 calculates a parameter for calculating a position where the image position and the image of the current frame are overlapped based on the corresponding points obtained from the corresponding point extraction unit 105. The alignment unit 106 only needs to have one or more corresponding points for parameter calculation. In the case of one point, the corresponding points are overlapped so that they are at the same position, and in the case of two or more points, many points are overlapped. The position of overlapping may be determined using the geometric relationship. The alignment unit 106 may describe translation / rotation / enlargement / reduction / affine transformation as conversion parameters corresponding to images. The obtained conversion parameter is sent to the learning unit 107, and the process proceeds to step S107.
In step S <b> 107, the learning unit 107 learns a model based on the model sent from the alignment unit 106 and the detection region alignment result. As a result, it is possible to incorporate into the model features that change due to learning while retaining strong features representing the shape of the object compared to the models learned so far. The learning unit 107 may use SVM or Boosting as a learning method, or may use a sequential learning method such as online boosting.
With the above processing, the image processing apparatus 100 performs detection processing on the image using a model learned in advance, and when learning the extracted target image, detection is performed based on the likelihood of each feature at the time of detection. Align using the position of the contributing feature and the position of the salient feature in the model. The image processing apparatus learns a model based on the alignment. This makes it possible to newly create a model whose target posture or shape has changed.

<Embodiment 2>
Next, Embodiment 2 will be described. FIG. 7 is a diagram illustrating a functional configuration of the image processing apparatus 100 according to the second embodiment. The image processing apparatus 100 according to the second embodiment includes an image acquisition unit 101, a feature extraction unit 102, a likelihood calculation unit 103, a target detection unit 104, a model part position calculation unit 605, a registration unit 606, and a learning unit 107. Yes. Hereinafter, the configuration and processing of the learning method according to the present embodiment will be described with reference to the drawings. In addition, the same code | symbol is attached | subjected to the same location as Embodiment 1, and description is abbreviate | omitted.
Based on the likelihood calculated from the likelihood calculation unit 103, the target detection unit 104 identifies a position and a region of a target having a higher likelihood than an arbitrary threshold value set in advance. As a result of the identification, the extracted target areas are indicated by 330 to 360. The obtained target detection results 330 to 360 are output to the model part position calculation unit 605.
The model part position calculation unit 605 receives the position and area of the object sent from the object detection unit 104, and detects the position where the specific part (model part) 217 of the model for alignment is located from within the area. In the present embodiment, the model part 217 is a head, but may be a body, a limb or the like. The model part position calculation unit 605 calculates the likelihood of the model part using the feature obtained from the feature extraction unit 102, and identifies the part position. FIG. 8 shows part positions 737, 747, 757, and 767 in the obtained target images.

FIG. 8A to FIG. 8D show the result of alignment between each detection result region and the model. This alignment result (737, 747, 757, 767) and each target detection area (330, 340, 350, 360) are sent to the alignment unit 506.
Based on the part position (737, 747, 757, 767) obtained from the model part position calculation unit 605, the alignment unit 506 extracts the model part 217 and the part position (737, 747, 757, 767) of the extracted image. And match. The result of the alignment (738, 748, 758, 768) is sent to the learning unit 107.
The learning unit 107 learns a model using the alignment results (738, 748, 758, 768) of the alignment unit 606. For learning the model, the learning unit 107 extracts an image feature amount from the obtained target detection area, and learns the image feature in the newly obtained detection target area for the feature of the model. As a result, a model in which the probability distribution of the weak classifier having a feature that varies due to learning is updated is generated while maintaining the feature whose shape of the target does not vary with respect to the model learned so far. For example, as a result of learning 738, models with different body orientations are generated, and at 748, models with different leg positions are generated. The learning unit 107 may perform learning using another known learning technique such as SVM or a sequential learning method.
The above is the configuration of the image processing apparatus 100 according to this embodiment.

Next, processing performed by the image processing apparatus 100 according to the present embodiment will be described using the flowchart shown in FIG.
FIG. 9 is a flowchart illustrating an example of a process performed by the alignment method and the learning method according to the second embodiment.
The processing from step S601 to step S603 is the same as the processing from step S101 to step S103 of FIG.
In step S <b> 604, the target detection unit 104 identifies a position / region where the target exists in the image based on the likelihood in the image calculated by the likelihood calculation unit 103. If it is identified that there is a target, the target position / region, image, and feature are sent to the model part position calculation unit 605, and the process proceeds to step S605. If no target is detected, the entire process ends.
In step S <b> 605, the model part position calculation unit 605 calculates the part position of the target model within the target position / region output from the target detection unit 104. The part position, image, and feature calculated at this time are sent to the alignment unit 606, and the process proceeds to step S606.
In step S606, the alignment unit 606 aligns the model part position with the position of the target part in the model detection area extracted from the image based on the part position calculated by the model part position calculation unit 605. The result of the alignment is sent to the learning unit 107, and the process proceeds to step S507.
In step S607, the learning unit 107 learns the model again based on the model sent from the result of the alignment unit 606 and the detection region alignment result. As a learning method, SVM or Boosting may be used, or a sequential learning method such as online boosting may be used.
With the above processing, the image processing apparatus 100 can learn the feature of the changing portion according to the image while maintaining the feature of the part portion that does not change.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

  As described above, according to each of the above-described embodiments, the image processing apparatus 100 identifies a portion where the target is changing and a portion where the target is not changing based on the model and the detection region, and the identified target is changed. The model and the detection area are aligned based on the part that is not. As a result, it is possible to extract the target image to be deformed and correct the target position based on the extraction result.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

100 Image processing apparatus

Claims (4)

Image acquisition means for acquiring images;
Feature extraction means for extracting feature quantities from the image acquired by the image acquisition means;
Likelihood calculation means for calculating the likelihood of the target based on the model of the target and the feature amount of the image extracted by the feature extraction means;
Detecting means for detecting a target detection area in the image based on the likelihood calculated by the likelihood calculating means;
Identification means for identifying a portion where the object is changing and a portion which is not changing based on the model and the detection region detected by the detection means;
An alignment means for aligning the model and the detection area based on a portion where the object identified by the identification means has not changed;
An image processing apparatus.   The image processing apparatus according to claim 1, further comprising a learning unit that learns the model based on a result of alignment by the alignment unit. An alignment method executed by an image processing apparatus,
An image acquisition step of acquiring an image;
A feature extraction step of extracting a feature amount from the image acquired in the image acquisition step;
A likelihood calculating step for calculating the likelihood of the target based on the model of the target and the feature amount of the image extracted in the feature extracting step;
A detection step of detecting a target detection area in the image based on the likelihood calculated in the likelihood calculation step;
An identification step for identifying a portion where the object is changing and a portion where the object is not changing based on the model and the detection region detected in the detection step;
An alignment step of aligning the model and the detection area based on a portion where the object identified in the identification step is not changed;
Including an alignment method. On the computer,
An image acquisition step of acquiring an image;
A feature extraction step of extracting a feature amount from the image acquired in the image acquisition step;
A likelihood calculating step for calculating the likelihood of the target based on the model of the target and the feature amount of the image extracted in the feature extracting step;
A detection step of detecting a target detection area in the image based on the likelihood calculated in the likelihood calculation step;
An identification step for identifying a portion where the object is changing and a portion where the object is not changing based on the model and the detection region detected in the detection step;
An alignment step of aligning the model and the detection area based on a portion where the object identified in the identification step is not changed;
A program that executes

Images