JP2007141107A - Image processor and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor and that achieves shape representation with a less number of dimensions for a model image referred to when an image is composed, and also to provide a method therefor. <P>SOLUTION: The model image referred to when a two-dimensional image of an object present in an image is composed, is composed, based upon composition parameters. At this time, efficient dimension compression of the composition parameters is performed by taking a main-component analysis of information representing a feature point of the model image, based upon local coordinate systems in a plurality of areas, such as an eye area of, for example, a face image, obtained by sectioning the model image. Thus, shape representation of the model image using a less number of dimensions is achieved. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method, and more particularly to an image processing apparatus and method for performing object recognition processing in an image.

  In recent years, digital cameras, mobile phones, digital video camcorders and the like have been put on the market at low prices. In addition, with the advent of a television tuner card, an encoder and a decoder card in which video compression techniques represented by MPEG2 and MPEG4 are implemented in hardware, a personal computer (PC) has realized a function as a video deck. Furthermore, such a PC can easily obtain a large amount of digital moving images / images such as moving image distribution via the Internet.

  Large volumes of hard disks and optical disks can be used to store a large amount of movies and images on popular price range PCs, but I would like to search for similar parts and images of videos from the stored content. Needs are increasing. In particular, there is a high search need to search for images with the same subject.

  In order to search for an image with the same subject, it is first necessary to detect the subject, particularly the face area, from the image. As a face detection technique, a technique using a neural network proposed by Rowley et al. Of Carnegie Mellon University is known. According to this technique, the position, size, rotation angle, etc. of a face in an image can be obtained. It is done.

  As a face recognition technique for determining who the detected face is, there is a face recognition descriptor that is a part of MPEG7 (Moving Pictures Experts Group-7) as an international standard. This uses luminance information or information converted into a frequency space with respect to a luminance image of a face image obtained by cutting out and normalizing a face region from the image. That is, by combining principal component analysis and linear discriminant analysis, a base vector for extracting a feature amount of a face image is obtained, and the face feature amount is extracted by linear projection. In this method, the face is cut out based on both eyes, and it is possible to cope with a slight positional shift by converting to the frequency space, but there is a problem of performance deterioration due to the correspondence between each organ of the face not being taken. is there.

  As a technique for deforming so as to match the correspondence of each organ of the face, there are AAM (active appearance model) proposed by Patent Document 1 and Mr. Cootes of Manchester University.

  In the former, a grid-like reference point is set for a face image, and a variation matrix of the reference point is modified while being updated by a genetic technique to match the reference image with the input image. The amount of variation at each reference point is the feature amount of the face image.

  The latter prepares a large amount of a two-dimensional face image database, extracts shape information and texture information from the coordinates and luminance values of a plurality of feature points for each face in the image, An average face is obtained by obtaining an average of texture information. Then, a principal component analysis is performed on the difference from the average face of each face image to obtain a partial space corresponding to a change in shape and expression. Then, the face image is synthesized by changing the synthesis parameter along each coordinate axis of the obtained partial space. When this model is used as an image recognition method, an image is synthesized by moving the synthesis parameters in the previously obtained partial space, and the synthesis is performed when the difference from the input face image to be recognized is the smallest. Find the synthesis parameters for the image.

The composite parameter of the composite face image highly correlated with the input face image is obtained by the following method. That is, a matrix for projecting the parameter correction amount from the difference information between the input face image and the combined face image is obtained in advance, and the difference evaluation, parameter update, and face recombination are repeated. At this time, a multi-resolution technique is used to prepare a multi-stage resolution in which the resolution ratio is halved in one stage, and a projection matrix is obtained at each resolution. Then, the synthesized face image is approximated to the input image with a coarse resolution, and the approximation is repeated while increasing the resolution step by step to obtain a synthesized parameter having a high correlation with the input image at the final resolution.
JP 2000-113197 A

  In the conventional face expression model, the principal component analysis is performed using the coordinates of the feature points of the face and the coordinates divided in a grid pattern in order to represent the shape of the face. These fluctuations in physical characteristics have non-linear elements, which cause the number of dimensions to not decrease even when principal component analysis is performed.

  The present invention has been made to solve the above-described problems, and provides an image processing apparatus and method capable of expressing a shape with a smaller number of dimensions in a model image referred to in image synthesis. With the goal.

  As a technique for achieving the above object, the image processing method of the present invention includes the following steps.

  That is, an image processing method for constructing a model image that is referred to when a two-dimensional image of an object existing in an image is synthesized, the setting step for setting a synthesis parameter of the model image, and the setting in the setting step A synthesis step of synthesizing the model image based on the synthesized parameters, wherein the synthesis parameter is characterized by a feature of the model image based on a local coordinate system in a plurality of regions into which the model image is partitioned. Dimensional compression is performed by principal component analysis of information indicating points.

  According to the present invention having the above-described configuration, the dimensional compression effect in the model image that is referred to during image synthesis is increased, and shape representation with a smaller number of dimensions is possible.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

<First Embodiment>
Apparatus Configuration FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to the present embodiment. In the image processing apparatus of the present embodiment, 201 is a CPU that executes instructions according to programs stored in the ROM 202 and the RAM 203. Reference numeral 202 denotes a ROM that stores a program for realizing the operation of the present embodiment and other programs and data necessary for control, and 203 denotes a RAM that is used as a work area for temporarily storing data. Reference numeral 204 denotes a drive I / F that realizes an interface with an external storage device such as IDE or SCSI, and 205 denotes an HDD as a storage device that stores an image, a feature amount for image search, a program, and the like. Reference numeral 206 denotes an image input unit that inputs an image from a device such as a digital camera or a scanner, and 208 denotes an operation input unit that receives an input from an operator, such as a keyboard or a mouse. Reference numeral 209 denotes a display unit such as a cathode ray tube or a liquid crystal display, and 210 denotes a network I / F such as a modem or a LAN for connecting to a network such as the Internet or an intranet. Reference numeral 211 denotes a bus which connects the above-described components to enable mutual data input / output.

  The image processing apparatus according to the present embodiment is installed as an application operating on a WINDOWS (registered trademark) XP (registered trademark) manufactured by Microsoft Corporation as an operating system on a PC.

FIG. 2 shows an outline of the face model in this embodiment and will be described. As shown in the figure, the face model includes shape information 301, texture information 302, constraint conditions 303, and parameter update table 304. The shape information 301 includes an average of shape vectors obtained from a large number of face images, and a shape base vector of a subspace obtained by principal component analysis with respect to a displacement from the average. The texture information 302 includes a luminance average of images obtained from a large number of face images and a texture basis vector of a subspace obtained by principal component analysis with respect to a displacement from the average. The constraint condition information 303 indicates a maximum value and a minimum value that are ranges that can be taken in each dimension of the shape vector. The parameter update table 304 has a table for each resolution of 256 × 256, 128 × 128, 64 × 64, 32 × 32, 16 × 16, and 8 × 8.

  Hereinafter, the construction procedure of each part of the face model will be described. Note that in order to statistically process a face image, an image including a large amount of face images is prepared in advance when the face model of this embodiment is constructed.

Facial Feature Points First, a method for setting facial feature points in the present embodiment, which is necessary for constructing the shape information 301, will be described with reference to FIGS. 3A to 3F.

  3A to 3F are diagrams illustrating an example of shape information on a face facing the front. In the present embodiment, a local coordinate system is defined by the face portion, a range of feature points that can be taken in the local coordinate system is limited, and information whose coordinates can be specified under this limited condition is a shape vector. To express the feature points of the face.

  FIG. 3A is a diagram illustrating an example of face coordinates. The face coordinates are a plane with the X axis as the line connecting the centers of both eyes and the Y axis as a straight line orthogonal to the center of both eyes, and the length from the origin to the center of the eyes is one unit.

  FIG. 3B is a diagram showing details of the eye portion. The center of the eye is the center of the line connecting the head and the corner of the eye. The length of this line segment in the left eye is L1, the angle between the line connecting the centers of both eyes is A1, and the length from the point obtained by dividing this line segment into four equal parts to the intersection of the line segment and the eyelid From L2 to L7. The same number of parameters are set for the right eye as for the left eye. That is, L14 corresponding to L7 and A2 corresponding to A1 are set from L8 corresponding to L1 in FIG. 3B.

  FIG. 3C is a diagram showing details in the vicinity of the nose. A position P1 at the lowest part of the nose in the face coordinate system is set as the origin, a line connecting the centers of the line segments connecting both eyes is defined as the Y axis, and the X axis is orthogonal to the Y axis at the origin. A nose coordinate system is defined in which the intersection point with the center of the line segment connecting both eyes is Y = 1, and the feature point of the nose in the nose coordinate system is represented by P2 and P3 relative to P2. Moreover, the right end is set to P4 toward the end of the nose, and the relative coordinates from P3 are expressed. If there is a nostril, P2 and P3 are set at the end of the nostril. If not, P1 and P3 are set along the edge of the nose at intervals of equal lengths P1-P2, P2-P3, and P3-P4. Similarly, P5 to P7 are set for the right side portion of the nose. Further, when there is a nostril, P8 and P9 are set on the edge of the hole, and when there is no nostril, the position between P2 and P3 is set to be divided into two on the nose edge.

  FIG. 3D is a diagram showing details in the vicinity of the eyebrows. The angle between the line connecting the eyebrow eye end and the center of the eye below and the line connecting the centers of both eyes is A3, and the line connecting the eyebrow eye end and the center of the eye below the eyebrow Let A4 be the angle formed by the line connecting the edge of the eye corner side and the center of the lower eye. Then, three lines that divide the angle A4 into four equal parts are drawn, and the intersection with the edge of the eyebrows is used as a feature point. Lengths L8 to L12 and eyebrows widths L13 to L15 from each feature point to the eye center are set. Similarly, A5, A6, and L16 to L23 are set for the right eyebrows. The expression of eyebrows is equivalent to the coordinates in the local coordinate system.

  FIG. 3E is a diagram showing details of the mouth region. A nose coordinate system is used for the mouth region. On the Y axis of the nose coordinate system, the upper lip upper part is represented as P11, the upper lip lower part P12 is represented relative to P11, the lower lip upper part P13 is represented relative to P12, and the lower lip lower part P14 is represented relative to P13. . Further, P16 to P18 are determined so that the right end toward the lips is P15 and the lip edges from P11 to P15 are equally divided into four. Similarly, P19 to P21 are set between P12 and P15, and P22 to P27 are similarly set for the lower lip. Here, it is assumed that the coordinates P16 to P27 are expressed as relative coordinates from the feature point on the left. Similarly, P28 to P40 are set for the left side of the lips.

  FIG. 3F is a diagram showing the outline of the face. The face contour is equivalent to a local coordinate system having the same origin as the nose coordinate system. In the local coordinates centered on P1 in the nose coordinate system (FIG. 3C), the direction of P11 is 0, the range of −5 / 8π to 5 / 8π radians is equally divided by 1/8 unit, and the center of the face contour ( The distance from P1) is indicated by L28 to L33.

  The facial feature points in this embodiment are set as described above.

Hereinafter, the construction method of each configuration of the face model shown in FIG. 2 will be described in detail. Forming procedure of shape information In order to build the shape information 301, the shape vector x is set as follows.

X = {A1, ..., A6, L1, ..., L33, P1x, P1y, P2x, P2y, ..., P40x, P40y}
Also, a feature point coordinate vector s consisting of each feature point on the face coordinate axis is set as follows.

s = {x1, x2, ......, y1, y2, ...}
Hereinafter, the construction procedure of the shape information 301 will be described using the flowchart of FIG. First, in step S501, a target image including a face image is displayed on the display unit 209. In step S502, the coordinates of both eyes in the face image in the image are designated from the input unit 208. The coordinate system specified here is an XY coordinate system with the origin at the upper left of the image. Next, affine transformation is performed in step S503, and a face image is cut out so that the coordinates of the eye designated in step S502 are in a predetermined position.

  Next, in step 504, each feature point coordinate on the face image is designated from the input unit 208. Then, the coordinates of the designated feature point are converted into the above-mentioned local coordinate system and converted into shape information. Here, for feature points that should be enlarged and set at equal angles and intervals, the input error can be reduced by displaying a guideline each time or performing fine adjustment of the position by calculation. .

  When the specification of all the feature points for one face image is completed, the process proceeds to step S505, and is recorded in the HDD 205 in association with the binocular coordinates on the image, the shape vector, and the image file name.

  Next, the process proceeds to step S506, where it is determined whether or not feature point designation has been completed for all face images. If there is an unset face image, the process returns to step S501. If all face images have been designated, step S507 is performed. Proceed to

  In step S507, the average coordinates (average shape) of each feature point are obtained for all the face images specified in the previous stage. In step S508, a difference shape vector is generated based on a difference from the average shape at each predetermined sample point. In step S509, the principal component analysis is performed on the shape vectors of all samples. As a result, a base matrix for the subspace is obtained, which is composed of principal components from the largest contribution rate until the total contribution rate reaches a predetermined ratio. That is, this basis matrix becomes a shape basis vector.

  In step S510, the obtained average shape and shape basis vector are recorded in the HDD 205 as shape information 301.

Texture Information Construction Procedure Hereinafter, the texture information 302 construction procedure will be described with reference to the flowchart of FIG.

  First, in step S601, the coordinates of both eyes corresponding to one face image recorded in the HDD 205 in step S505 and the cut shape vector are read, and the face image is cut out by affine transformation.

  In step S602, morphing is performed so that the feature point moves to the average position. Morphing is expressed as follows.

I _m = M (I, S _s , S _d ) (1)
Incidentally, it is I _m is synthetic face image is morphed, the face image I to cut out, S _s is the feature point coordinate sequence of the conversion source, S _d is a feature point coordinate sequence of the destination in (1). The feature point coordinate sequence is coordinates on the cut face image, and can be uniquely obtained from the shape vector.

  By such morphing, the positions of the organs of different faces become the same position in the face coordinate system, and the association is ensured.

  In step S603, masking is performed. That is, the outside of the face contour of the average face shape indicated by the feature point average coordinate sequence is masked, and the area other than the face is excluded from the processing target to be performed thereafter.

  In step S604, the luminance distribution is normalized. This ensures that the average luminance and luminance variance of the image are aligned to a predetermined value. In addition, a process for smoothing the luminance distribution by performing histogram smoothing is performed.

  In step S605, it is determined whether the morphing, masking, and normalization processes have been completed for all face images. If there are unfinished face images, the process returns to step S601, but all have been completed. If so, the process proceeds to step S606.

  In step S606, an average texture is obtained by obtaining the average luminance of all face samples for each pixel.

  In step S607, a difference from the average luminance obtained in the previous stage is obtained for each face sample, and a luminance difference vector including the number of non-masking pixels is generated. In step S608, the luminance difference vector of the total number of face samples is subjected to principal component analysis. In order of increasing contribution rate, a base matrix for a subspace consisting of principal components until the total contribution rate reaches a predetermined ratio is obtained. This basis matrix is a texture basis vector.

  In step S609, the obtained average texture and texture basis vector are recorded in the HDD 205 as texture information 302.

-Procedure for constructing constraint condition information The procedure for constructing constraint condition information 303 will be described below.

  The constraint condition that is a feature of the present embodiment is set in order to reduce the probability that a face that is unlikely to exist is synthesized. In the present embodiment, a range of possible values of each dimension of the shape vector x is set, and it is determined whether or not there is a low possibility of existence depending on whether or not this value is exceeded. The shape vector is corrected by clipping to fit. Specifically, determination and correction are performed according to the following equation (2).

When x _i ≤ x ^av _i -kσ _i ,
f (xi) = x ^av _i -kσ _i
When x _i ≧ x ^av _i + kσ _i ,
f (xi) = x ^av _i + kσ _i
At other times
f (xi) = x _i
... (2)
In equation (2), x ^av _i is the average of samples in the i-th dimension, and σ _i is the standard deviation of samples in the i-th dimension. Further, k is a predetermined constant and becomes a threshold value of the real probability. For each dimension of the feature vector of the sample face image, the distribution of values is approximated by a normal distribution. For example, if it is desired to cover 99.7%, k = 3.

That is, in this embodiment, x ^av _i -kσ _i in the equation (2) is stored in the HDD 205 as the constraint condition information 303 indicating the shape minimum value. Similarly,
x ^av _i + kσ _i is stored in the HDD 205 as constraint condition information 303 indicating the shape maximum value.

-Construction of Parameter Update Table The procedure for constructing the parameter update table 304 will be described below.

  The entity of the parameter update table 304 is a matrix for projecting to the correction amount of the synthesis parameter of the face image to be synthesized next in order to reduce the difference from the difference information between the inputted face image and the synthesized facial image. is there. This matrix is obtained at each resolution of 256 × 256, 128 × 128, 64 × 64, 32 × 32, 16 × 16, and 8 × 8 pixels for the face image.

  Hereinafter, the construction procedure of the parameter update table 304 at a certain resolution will be described with reference to the flowchart of FIG.

  First, in step S701, a first synthesis parameter is set. Here, the face image composition parameter is set randomly as the first composition parameter. However, the upper limit of displacement from the average model of each parameter must be experimentally determined from the approximate success probability.

  In step S702, the second synthesis parameter is set. The second synthesis parameter is obtained by changing one of the elements of the first synthesis parameter to positive or negative. The amount of change is determined based on the variance of the data of the parameter element, but it must be obtained experimentally.

  In step S703, two face images are synthesized based on the first and second synthesis parameters. At this time, the constraint condition that is a feature of the present embodiment is applied, and details thereof will be described later.

  In step S704, a difference vector between the two composite images is obtained. This difference vector is a vector having a dimension corresponding to the number of pixels and a luminance difference of each pixel as a value. The synthesized face outline and the average face have different face outlines and different numbers of pixels. Here, in order to collect the difference vectors and handle them as a matrix, it is necessary to make the number of pixels constant for each resolution. Therefore, by applying masking of the same shape as in step S603, a difference vector is formed by pixels in the area occupied by the average face.

  In step S705, it is determined whether a predetermined number of loops have been performed. The number of loops is at least the number of dimensions of the face difference vector. If it is necessary to further loop, the process returns to step S701. If the loop has reached a predetermined number of times, the process proceeds to step S706.

  In step S706, multivariable linear regression is obtained. Specifically, a projection matrix A such that the determinant (3) shown below is obtained.

ΔC = AΔI (3)
In Equation (3), ΔC is a set of differences between the first synthesis parameter and the second synthesis parameter, and ΔI is a set of difference vectors of the corresponding synthesized image.

  The projection matrix A can be derived from the determinant (4) shown below.

A = ΔCΔI ⁺ (4)
In Equation (4), ΔI ⁺ is a pseudo inverse matrix of ΔI.

  In step S707, the obtained projection matrix A is recorded in the HDD 205 as the parameter update table 304.

-Face Image Combining Method Hereinafter, the face image synthesizing method using the synthesis parameter in step S703 will be described with reference to the flowchart of FIG.

  First, in step S801, as shown in the following equation (5), a texture G is created by a linear sum of an average texture and a synthesis parameter that is the strength of each principal component.

G = g ^av + P ⁺ _g b _g (5)
In equation (5), g ^av is an average texture, and P ⁺ _g is a pseudo inverse matrix of texture basis vectors, which may be obtained in advance. b _g is a luminance synthesis parameter.

  In step S802, a shape vector X is created by a linear sum of an average shape vector and a synthesis parameter that is the strength of each principal component.

_{^{X = x av + P + x}} b x ··· (6)
In Equation (6), x ^av is an average shape, and P ⁺ _x is a pseudo inverse matrix of a shape basis vector, and may be obtained in advance. b _x is a shape synthesis parameter.

In step S803, it is determined whether or not the synthesized face image indicates a face that may actually exist. Here, as shown in Equation (7), it is determined whether or not the values of the dimensions of the combined shape vector X are within a predetermined range. At this time, the above-described constraint condition information 303 (in this case, kσ _i ) is referred to.

| X _i -x ^av _i | ≦ kσ _i (7)
If it is determined that there is no possibility that the face image actually exists, the process proceeds to step S804. If the face image can exist, the process proceeds to step S805.

  In step S804, the above equation (2) is applied to convert the face shape vector X ′ that is close to the synthesized face and may exist.

X '← f (X) (8)
In step S805, the shape vector X is converted into feature point coordinates on the face coordinates based on the equation (9).

S = S (X ') (9)
Further, the conversion from the average shape vector to the average coordinate value is performed in the same manner, but this may be obtained in advance based on the average shape vector x ^av as shown in the equation (10).

s ^av = S (x ^av ) (10)
In step S806, a composite face image _Im is obtained by performing morphing from an average face shape to a synthesized shape as shown in Expression (11).

I _m = M (G, s ^av , S) (11)
In order to make the explanation easier to understand, an example in which the principal component analysis is performed separately for the shape and texture components is shown. Can be reduced. However, when dimensions with greatly different dynamic ranges are mixed, the influence of calculation errors increases, so it is preferable to perform scaling adjustment to align the dynamic ranges.

Image Search System The face model construction method in the present embodiment has been described above. Hereinafter, an outline of image processing in the present embodiment using the face model will be described.

  In this embodiment, an image search system that focuses on a face area in an image and outputs an image showing a similar face to a given query image as a similar image will be described as an example. As image processing in this case, an image registration process for extracting and recording feature amounts from images necessary for search, and an image for obtaining the most similar image by comparing feature amounts when a search condition is given Consists of search processing. Hereinafter, each of the image registration process and the image search process will be described.

Image Registration Processing The image registration processing in this embodiment will be described using the flowchart of FIG.

  In step S901, an image is input from the image input unit 206. In step S902, a face area is detected from the input image. As the detection process, the above-described method by Rowley et al. Is applied. In step S903, the face size and inclination are corrected from the obtained coordinates of both eyes, and the face image is cut out in the same manner as in step S503. In step S904, the luminance is normalized. That is, processing such as histogram equalization is performed so that the luminance distribution is constant.

  In step S905, feature extraction is performed. Here, a composite parameter of a composite face model having the highest correlation with the input face image is determined, and this is used as a feature amount. Details of this feature extraction processing will be described later.

  In step S906, the composite parameter obtained by the feature extraction process is recorded in the HDD 205. In step S907, the image input unit 206 determines whether an unprocessed image remains. If not, the process proceeds to step S908. If an image to be processed exists, the process returns to step S901.

  In step S908, linear discriminant analysis is performed in order to calculate the distance between feature quantities at the time of search, and a basis vector of the discriminant space is obtained. The linear discriminant analysis classifies feature quantities and obtains a basis vector of a discriminant space that maximizes the ratio of the distance between feature quantities in the same class and the distance between classes. Depending on the purpose of the system, define a similar face, such as a similar face is a face of the same person, or a similar face is a face with a similar expression. By performing linear discriminant analysis after classifying, functions that meet the purpose are realized.

  As a method for projecting to the discriminant space, for example, a technique for projecting to a non-linear space as described in the following document is also known. Since this detail is described in the following document, detailed description is omitted here.

J. Lu, K. Plateniotis, and A. Venetsanopoulos; "Face recognition using kernel direct discriminant analysis algorithms". IEEE Transactions on Neural Networks, 14 (1), Jan. 2003
In the present embodiment, an example in which registration processing is performed on an image input from the image input unit 206 is shown, but the present invention is not limited to this. For example, an image to be registered may be stored in a predetermined folder, and the registration process may be performed for an image file under the folder by designating the folder. In this case, as a process of registration processing, when the target folder is newly added, all files are processed, but thereafter, only the updated file may be processed. It is also possible to start as a background process and monitor the update of the target folder.

Feature Extraction Processing Hereinafter, the feature extraction processing shown in step S905, that is, processing for determining the synthesis parameter of the synthetic face model having the highest correlation with the input face image will be described in detail with reference to the flowchart of FIG. .

  First, in step S1001, an initial resolution is set. Here, the resolution is set to 8 × 8 pixels. In step S1002, the loop counter n is initialized to 1, and in step S1003, the synthesis parameter is initialized. At the first time, the synthesis parameters are set so as to obtain an average shape and texture. In step S1004, a face image is synthesized from the synthesis parameters by the method described above.

  In step S1005, it is determined whether the loop counter n is 2 or more. If it is not the second time or more, the process proceeds to step S1007. If it is the second or more time, the process proceeds to step S1006.

  In step S1006, it is determined whether or not the synthesized face has converged. This is determined by whether or not the difference from the previous combined face is equal to or less than a threshold value. The difference is determined as a sum of squares of the luminance difference of each pixel, and this is compared with a predetermined threshold set in advance for each resolution. If the difference is equal to or smaller than the threshold, it is determined that the difference has converged. In this case, the processing is terminated as an approximation failure. On the other hand, if the difference is larger than the threshold value, it is determined that it has not converged, and the process proceeds to step S1007.

  In step S1007, a difference vector between the face image of the input image and the synthesized face image is obtained. Here, the difference vector is a luminance difference between corresponding pixels of the input image and the composite image, but has the same number of dimensions, that is, pixels, as the corresponding resolution of the difference vector obtained when the parameter update table recorded in step S707 is constructed. A number of things are needed. Therefore, as shown in the following equation (12), the difference image is obtained as an image that is morphed into an average face shape and subjected to the same masking as in step S603.

ΔI _rn = M (I _rs -I _rn , S _rn , s ^av ) (12)
In Expression (12), I _rs is a face image of the input image at resolution r, and I _rn is a synthesized face image at resolution r and loop counter n. S _rn is a face feature coordinate vector in the loop counter n, and s ^av is an average coordinate value vector.

  In step S1008, it is determined whether the difference E is equal to or less than a predetermined value using the following equation (13). The difference E is the L2 norm (Euclidean distance) of the difference vector.

E = | ΔI _rn | (13)
If the difference E is less than or equal to a predetermined value, the process proceeds to step S1012 on the assumption that the approximation of the composite image with respect to the input image is successful. If not, the process proceeds to step S1009.

  In step S1009, the loop counter n is evaluated, and if it has been repeated a predetermined number of times (L times), it ends as an approximation failure. If it is repeated a predetermined number of times or less, the process proceeds to step S1010.

  In step S1010, the composite parameter is updated by projecting the difference to the composite parameter change amount according to the following equation (14).

b _{r (n + 1)} = b _rn + A _r ΔI _rn (14)
In Equation (14), b _rn is a synthesis parameter when the resolution is r and the loop counter is n.

  Thereafter, the loop counter n is incremented in step S1011 and the process returns to step S1004.

  On the other hand, if it is determined in step S1008 that the approximation of the composite image has succeeded, it is determined in step S1012 whether the final resolution has been reached. Here, the final resolution is 256 × 256 pixels. If it is the final resolution, the final synthesis parameter is output as a feature value if the approximation is successful. If it is not the final resolution, the process proceeds to step S1013, the resolution is doubled, and the process returns to step S1002.

  In the feature extraction process of the present embodiment, there are cases where the combined face image does not change even if the synthesis parameter is changed because the constraint condition information 303 is imposed. Therefore, the synthesized image is evaluated to detect the convergence state at the extreme solution so that the convergence state can be escaped.

  If the approximation process fails, retry may be performed by setting the composite parameter setting value in step S1003 to a new value. As a new value setting method, it is simply performed at random. In addition, other parameter candidates that are highly correlated with the input image obtained during the search, or by using genetic methods, a bit string is created with the changed parameters and regarded as a gene, and the bit string of the parameter highly correlated with the input image is partially cut and pasted You may do it. Also, depending on the time remaining in the approximation and the number of retries, a limit range is set for the standardized Euclidean distance from the randomly generated synthesis parameter and the average face image or the previous change parameter. Random synthesis parameters may be set.

  Also, a plurality of face models may be prepared, and when approximation with a certain face model fails, retry with another model may be performed.

Image search processing Based on the feature amount extracted as described above, that is, the synthesis parameter, a synthetic image to be approximated is searched from the face model. Hereinafter, the image search process in the present embodiment will be described with reference to the flowchart of FIG.

  In step S1101, a query image is input from the image input unit 206. Alternatively, an image file in the HDD 205 may be designated. Next, proceeding to step S1102, it is determined whether or not there is a feature amount in the query image. That is, it is determined whether or not the image has already undergone registration processing. If there is a feature amount, the process proceeds to step S1108.

  On the other hand, if the feature amount does not exist in the CLIE image in step S1102, that is, if it is an unregistered image, the process proceeds to step S1103, and the process up to step S1107 is performed. Here, with respect to the processing from step S1103 to step S1106, the same processing as step S902 to step S905 in the image registration processing shown in FIG. 8 described above is applied to the query image. Therefore, detailed description here is omitted. In step S1107, the feature amount extracted in step S1106 is linearly projected onto the discrimination space.

  In step S1108 and step S1109, substantial search processing is performed. That is, first in step S1108, the L2 norm of the difference vector is obtained for the feature vector of the query image and the face image to be searched in the discrimination space, and this is used as the distance. In step S1109, image output is performed. For example, the search target images are rearranged in order of distance from the query image, and the reduced images are displayed as a list.

  As described above, according to the present embodiment, a model image indicating a face shape is treated as a face feature amount expressed by a combination of local coordinate systems for each facial organ. By performing this principal component analysis, the contribution rate is concentrated on the principal component side, and the dimensional compression effect is enhanced. Therefore, cost reduction in the entire system such as feature amount storage cost, feature matching cost, parameter update table size and learning cost can be realized.

Second Embodiment
Hereinafter, a second embodiment according to the present invention will be described.

  In the first embodiment described above, an example in which much of the shape information is expressed by relative coordinates or the like in the local coordinate system has been shown. In the second embodiment, this is represented by the face of each feature point as in the past. An example of coordinate values in the coordinate system is shown.

  An example of setting facial feature points in the second embodiment is shown in FIG. Here, when the constraint condition is a range that each feature value can take, this means that the maximum and minimum values of the X coordinate and the Y coordinate are defined, and the short shape including the side parallel to the coordinate axis including the feature point is defined. Will show the area. For this reason, the evaluation condition is loose. Therefore, in the second embodiment, more effective evaluation conditions are given.

  A construction procedure of constraint condition information when shape information is used as a coordinate value in the second embodiment will be described with reference to a flowchart of FIG.

  First, in step S1301, for each feature point of the face, a convex hull region of the corresponding feature point set of all face samples is obtained. In step S1202, the corresponding convex hull region is enlarged by a predetermined magnification with the average coordinate at the center of each feature point of the face. In step S1203, the convex hull region is recorded in the HDD 205 in association with each feature point of the face.

  Then, in the second embodiment, the face image is synthesized by the procedure shown in FIG. 7 as in the first embodiment. In step S803, the coordinates of the feature points of the face to be synthesized are within the convex hull region. The possibility of not being a face is determined by determining. In step S804, if the coordinates of each feature point exist outside the convex hull region, the process of converting the coordinates of the intersection of the convex region boundary with the line connecting this point and the average feature position is performed in all dimensions. It may be converted into a face that may exist.

  In addition, although the convex area | region shown in 2nd Embodiment is a polygon with the indefinite number of vertices, if this is approximated by a rectangle or an ellipse, processing cost can also be reduced.

  As described above, according to the second embodiment, even if the shape information is handled as coordinate values in the face coordinate system, the same effect as in the first embodiment can be obtained.

<Third Embodiment>
The third embodiment according to the present invention will be described below.

  FIG. 13 is a block diagram illustrating a configuration of an image processing apparatus according to the third embodiment. In the same figure, the same number is attached | subjected to the same structure as FIG. 1 shown in 1st Embodiment mentioned above, and description is abbreviate | omitted.

  According to FIG. 13, the third embodiment is characterized in that an optical disk 212 such as a DVD or CD in which a characteristic program of the present invention is recorded and its interface are added as components. That is, an external storage reading / writing device 213 such as a CD / DVD drive is connected to the drive interface 204.

  When the optical disk 212 on which the characteristic program of the present invention is recorded is inserted into the external storage reading / writing device 213, the CPU 201 reads the program from the optical disk 212 and develops it in the RAM 203. Thereby, the process similar to 1st and 2nd embodiment mentioned above is realizable.

<Other embodiments>
The present invention is not limited to the first to third embodiments described above, and various modifications can be made without departing from the spirit of the present invention. Hereinafter, various modifications will be described.

Depth information Each feature point may have depth information. In this case, the dimension of the shape information increases by the number of feature points, but this is significant information representing the facial features, and becomes a model independent of the influence of the facial orientation. It is within the scope of the gist of the present invention when an object is virtually synthesized three-dimensionally with feature points having depth information and further imaged with viewpoint and illumination method parameters. The depth information can be acquired by, for example, triangulation using two input devices spaced apart or a laser rangefinder.

Parameter update table The parameter update table construction method is not limited to the method shown in the first embodiment, and various modifications are conceivable. For example, in the first embodiment, a random synthesis parameter is set in step S701. However, this may be a synthesis parameter projected from the sample face image.

  In addition, new dimensions such as XY movement amount, enlargement ratio, aspect ratio, rotation, etc. are added as elements of the synthesis parameter in order to be able to cope with some movement, enlargement ratio, aspect ratio, rotation, etc. of the entire face. Also good. At this time, coordinate information on an image that has been normalized by enlargement / reduction and rotation based on at least two feature points in the target object may be used as the feature points.

  Also, the parameter update table changes the approximate success probability depending on the first synthesis parameter setting method. Therefore, using the approximate success probability as an evaluation function, a method of increasing the approximate success probability by mating the first synthetic parameter set when a parameter update table with a high evaluation value is obtained using a genetic method or the like. Is also effective.

-Time limit In the above-described first embodiment, only the loop counter n is evaluated in step S1009 of FIG. 9, but in addition to that, an evaluation as to whether it is within the time limit may be added. That is, if the time limit is exceeded, it can be considered that the approximation is terminated as a failure.

Difference Vector In the first embodiment described above, the difference vector between the synthesized face image and the face image in the input image has been described as the luminance difference of each pixel in the area occupied by the average face shape, but the present invention is not limited thereto. It is not something. For example, frequency conversion such as FFT or DCT may be performed, and intensity and phase information in each frequency component may be used, or an optical flow method may be used. Also, using all the pixels in the area occupied by the average face shape is not limited, but using some pixels improves the robustness when a part of the object of the input image is hidden.

  Further, convolution may be performed with a plurality of Gabor filters, and an intensity image may be used. The Gabor filter G (x, y) is shown in the following formula (15).

G (x, y) = exp [-π {(x-x0) ² / A + (y-y0) ² / B}] · exp [2πi {u (x-x0) + v (y-y0)} ]
... (15)
Where i is the imaginary unit, x = 0 to s-1, y = 0 to s-1, x0 = s / 2, y0 = s / 2, A is the horizontal influence range, and B is the vertical influence range. It is. Further, tan ⁻¹ (u / v) is the wave direction, and (u ² + v ² ) ^1/2 is the frequency. s is the vertical and horizontal size of the filter, and is a square here.

  In addition, 40 types of filters are generated, one for each frequency in the influence range in the wave direction 8 direction, frequency 5 types, and horizontal and vertical directions.

  The filter output value at an arbitrary position in the image is calculated by convolution between the filter and the image. In the case of a Gabor filter, there are a real number filter and an imaginary number filter (an imaginary number filter is a filter whose phase is shifted by a half wavelength from the real number filter), and the mean square value thereof is used as a filter output value. If the convolution between the real number filter and the image is Rc and the convolution between the imaginary number filter is Ic, the output value P is calculated by the following equation (16).

P = (Rc ² + Ic ² ) ^1/2 (16)
Alternatively, the number of dimensions may be reduced by principal component analysis of the difference vector obtained during the parameter update table construction, and the number of dimensions may be reduced by performing the same projection on the difference vector of the input image and the synthesized image. If a difference vector of a certain dimension is obtained at each resolution, it is within the scope of the present invention.

・ Constraining method For the synthesis parameter vector expressed in the subspace analyzed by principal component analysis, the variance of each dimension is obtained from a number of face images, standardized so that the variance of each dimension is equal, and the average shape The Euclidean distance may be obtained and whether the approximation is correct or not may be determined. The Euclidean distance obtained here is a standardized Euclidean square distance obtained by dividing the square error of each parameter element by the variance of the elements and taking the sum. If the obtained Euclidean distance is equal to or smaller than a predetermined value, it is determined that the composite image is successfully approximated with respect to the input image.

  In order to convert the synthesis parameter so as to obtain a face that may exist, the absolute value of each dimension value of the standardized shape vector is set within a predetermined range as shown in FIG. . That is, the shape vector when the Euclidean distance falls within a predetermined value while narrowing this range is output.

Hybridization The present invention can be used in combination as a pretreatment of the prior art. For example, a face deformed by the image processing apparatus of the present invention can be considered as an input image for the MPEG-7 feature extraction method. In this case, a face feature descriptor obtained by adding a synthesis parameter to the output face feature descriptor may be used as a face feature, or a matrix obtained by projecting these output matrices by the subspace method may be used as the face feature.

-Application to other than face images In addition, in the first embodiment described above, a system for performing similar image search has been shown. However, a face discrimination system for discriminating who is a face, and criteria for classification when constructing a discrimination space It is also easy to implement the present invention in a system for discriminating gender and age.

  The present invention can also be applied to any object other than the face. For example, it is conceivable to target the whole human body, living things, automobiles, and the like. Further, in the medical field, the present invention can be applied to position identification of bones and organs in X-ray photographs and CT scan images, and identification and inspection of industrial products, parts, and distribution articles in the industrial and distribution fields.

Others Although the embodiment has been described in detail, the present invention can take an embodiment as a system, apparatus, method, program, storage medium (recording medium), or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device. As a system composed of a plurality of devices, a system in which an image input device and an image storage device are combined or connected can be considered. Examples of the image input device include a camera and scanner using various CCDs such as a video camera, a digital camera, and a surveillance camera, and an image input device converted into a digital image by AD conversion from an analog image input device. Examples of the image storage device include an external hard disk and a video recorder. In such a system, the present invention also includes a case where the CPU or the like provided in all or any of the devices constituting the system performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. include.

  In the present invention, a software program for realizing the functions of the above-described embodiments is supplied directly or remotely to a system or apparatus, and the computer of the system or apparatus reads and executes the supplied program code. Is also achieved. The program in this case is a program corresponding to the flowchart shown in the drawing in the embodiment.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  Recording media for supplying the program include the following media. For example, floppy disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD- R).

  As a program supply method, the following method is also possible. That is, the browser of the client computer is connected to a homepage on the Internet, and the computer program itself (or a compressed file including an automatic installation function) of the present invention is downloaded to a recording medium such as a hard disk. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to make it. That is, the user can execute the encrypted program by using the key information and install it on the computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, and then executed, so that the program of the above-described embodiment can be realized. Function is realized. That is, based on the instructions of the program, the CPU provided in the function expansion board or function expansion unit can perform part or all of the actual processing.

It is a block diagram which shows the hardware constitutions of the image processing apparatus which is one Embodiment which concerns on this invention. It is a figure which shows the outline | summary of the face model in this embodiment. It is a figure which shows the setting method of the feature point of the face in this embodiment. It is a figure which shows the setting method of the feature point of the face in this embodiment. It is a figure which shows the setting method of the feature point of the face in this embodiment. It is a figure which shows the setting method of the feature point of the face in this embodiment. It is a figure which shows the setting method of the feature point of the face in this embodiment. It is a figure which shows the setting method of the feature point of the face in this embodiment. It is a flowchart which shows the construction procedure of the shape information in this embodiment. It is a flowchart which shows the construction procedure of the texture information in this embodiment. It is a flowchart which shows the construction procedure of the parameter update table in this embodiment. It is a flowchart which shows the synthesis method of the face image in this embodiment. It is a flowchart which shows the image registration process in this embodiment. It is a flowchart which shows the feature extraction process in this embodiment. It is a flowchart which shows the image search process in this embodiment. It is a figure which shows the setting method of the feature point of the face in 2nd Embodiment. It is a flowchart which shows the construction procedure of the constraint condition information in 2nd Embodiment. It is a figure which shows the hardware constitutions of the image processing apparatus in 3rd Embodiment. It is a conceptual diagram at the time of converting a synthetic parameter into a face that may exist.

Claims (14)

An image processing method for constructing a model image that is referred to when a two-dimensional image of an object existing in an image is synthesized,
A setting step for setting a synthesis parameter of the model image;
Synthesizing a model image based on the synthesis parameter set in the setting step, and
The synthesis parameter is characterized in that, based on a local coordinate system in a plurality of regions into which the model image is divided, information indicating feature points of the model image is subjected to dimension compression by principal component analysis. Image processing method.   2. The image processing according to claim 1, wherein the composite parameter is obtained by dimensionally compressing a shape feature of the model image quantified by coordinates of the feature point in the local coordinate system by principal component analysis. Method.   The image processing method according to claim 1, wherein the local coordinate system includes an orthogonal coordinate system.   The image processing method according to claim 3, wherein the local coordinate system includes a polar coordinate system.   5. The image processing method according to claim 2, wherein the coordinates of the feature points are relative coordinates from other feature points in the local coordinate system. Further, an evaluation step (S803) for evaluating the possibility of existence of the model image synthesized in the synthesis step;
A change step (S804) of changing the synthesis parameter so that the possibility of existence of the model image is high when it is determined that the possibility of existence is low in the evaluation step;
The image processing method according to claim 1, further comprising:   The image processing method according to claim 6, wherein in the evaluation step, if the element value of the synthesis parameter is outside a predetermined range, it is determined that the possibility of existence is low.   The image processing method according to claim 7, wherein the predetermined range is set based on a statistical distribution in a target object set.   9. The image processing method according to claim 7, wherein, in the changing step, an element value of the synthesis parameter is changed to a boundary value within the predetermined range.   The image processing method according to claim 1, wherein the object is a human face. An image processing apparatus for constructing a model image to be referred to when a two-dimensional image of an object existing in an image is synthesized,
Setting means for setting a synthesis parameter of the model image;
Synthesizing means for synthesizing the model image based on the synthesis parameter set in the setting step,
The synthesis parameter is characterized in that, based on a local coordinate system in a plurality of regions into which the model image is divided, information indicating feature points of the model image is subjected to dimension compression by principal component analysis. Image processing device.   An image processing system for approximating a composite image for an object existing in an image, using a model image constructed by the image processing method according to claim 1.   11. A program that realizes the image processing method according to claim 1 on a computer by operating on the computer.   A recording medium on which the program according to claim 13 is recorded.

Images