JP2006343859A - Image processing system and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing system and image processing method capable of precisely detecting information of a target object from a picked-up image with a small amount of data. <P>SOLUTION: The image processing system 11 detecting various information of the object from the pick-up image is provided with an imaging means 12, an information holding means 13a for holding a template comprising evaluation information and three-dimensional position information regarding a plurality of characteristic points on a two-dimensional image of the target object, an estimated value preparing means 13c for preparing a plurality of estimated values for positions and attitudes of the target object, a goodness-of-fit calculating means 13d for calculating goodness of fit of a plurality of estimated values prepared on the basis of respective templates regarding the picked-up image and held plurality of characteristic points, and a determining means 13e for determining an existence, the position, and/or attitude of the target object in the picked-up image on the basis of the goodness of fit of the calculated plurality of estimated values. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for detecting various information about an object from a captured image.

In image processing, a target object such as a face is detected from a captured image captured by a camera, and if an object can be detected, the position and orientation of the object are estimated. As such an image processing method, a large number of reference images obtained by imaging a target object from various viewpoints and positions are prepared in advance, and the input captured image and each reference image are matched, and similar. There is a technique for detecting a target object on a captured image by searching for a reference image. As another method, a reference image of a plurality of feature points extracted from an image obtained by capturing the target object is prepared in advance, and matching between the input captured image and the reference images of the plurality of feature points is performed. Search for a position similar to each reference image on the captured image, and determine whether there is a contradiction in the positional relationship of the plurality of feature points on the captured image. There is a method of detecting (see Patent Document 1).
JP 2004-252511 A

  However, in the case of the one eye method, the appearance of an object on the image changes greatly depending on the orientation, position, illumination conditions, etc. of the object, so prepare a reference image that covers all these variations in advance. Otherwise, the target object cannot be detected in the captured image in which such changes have occurred. When all variations of reference images are prepared, the amount of data becomes enormous, the processing load increases, and processing time is extremely long. Also, in the case of the second method, if an object orientation, position, or lighting condition occurs when searching for a feature point on an image, the feature point cannot be extracted or an incorrect point is extracted as a feature point is there. In addition, since a plurality of feature points are searched individually, the positioning accuracy of the feature points may deteriorate.

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method that can detect information about a target object from a captured image with a small amount of data with high accuracy.

  An image processing apparatus according to the present invention includes an imaging unit, an information holding unit that holds a template including evaluation information and three-dimensional position information about a plurality of feature points on a two-dimensional image of a target object, and a position of the target object or And / or an estimated value generating means for generating a plurality of estimated posture values, and an estimated value generating means based on each of the captured image captured by the imaging means and a plurality of feature points held by the information holding means. A fitness calculation means for calculating the fitness of a plurality of estimated values, and the presence / absence, position or / and / or position of the target object in the captured image captured by the imaging means based on the fitness of the plurality of estimated values calculated by the fitness calculation means And a judging means for judging the posture.

  In this image processing apparatus, templates for a plurality of feature points on a two-dimensional image of a target object are held in an information holding unit. The template is evaluation information (for example, luminance image (luminance pattern), luminance histogram, edge image, frequency obtained by Fourier transform on the luminance image) for evaluating the probability that each feature point is the point on the two-dimensional image. Characteristic) and three-dimensional positional information (for example, three-dimensional coordinates of a three-dimensional model) for indicating a positional relationship between the characteristic points in three dimensions. In the image processing apparatus, a two-dimensional image is acquired by imaging with an imaging unit. In the image processing apparatus, the estimated value generating means generates a plurality of estimated values of the position or / and orientation of the target object with respect to the imaging means. Then, in the image processing apparatus, the accuracy of the plurality of feature points is determined by taking the three-dimensional positional relationship into consideration based on the captured image and each template for the plurality of feature points for each estimated value by the fitness calculation means. And the degree to which the estimated value is adapted to the position or / and orientation of the target object in the captured image is calculated from the evaluation of the plurality of feature points. Whether or not the position and orientation of the estimated value match the actual position and orientation is determined by evaluating the likelihood of each feature point while maintaining the three-dimensional positional relationship of the plurality of feature points. Further, in the image processing apparatus, the presence / absence, position, and / or orientation of the target object in the captured image is determined by the determination unit based on the fitness of the plurality of estimated values. The higher the degree of matching, the higher the degree of matching of the position or orientation of the estimated value with the position or / and orientation of the target object in the captured image. The position or / and posture can be determined. In this way, the image processing apparatus holds evaluation information and three-dimensional position information for each feature point, and processes each feature point instead of the entire image. Light load and short processing time. In addition, since the image processing apparatus generates various estimated values for the position and orientation and obtains the fitness of each estimated value, it is robust against various variations of the target object, and the target object in the captured image Presence / absence, position, and orientation can be determined with high accuracy.

  In the image processing apparatus of the present invention, the estimated value generating means is a value related to the position or / and orientation of the target object in the captured image captured in the past by the imaging means, or a value related to the position or / and orientation that the target object can take structurally. The estimated value may be generated based on at least one of the position or / and orientation history of the target object.

  In the estimated value generation means of this image processing apparatus, the value related to the position or / and orientation of the target object in the captured image (for example, the captured image of the previous frame) captured in the past by the imaging means, The estimated value is generated on the basis of at least one of / and a value related to the posture, a position of the target object, and / or a history of posture. That is, the estimated value is not generated at random, but is generated based on the position or orientation that the target object is likely to take. As described above, in the image processing apparatus, by narrowing the range for generating the estimated value, it is possible to reduce the processing load without generating a useless estimated value.

  It should be noted that the value related to the position or / and orientation of the target object in the captured image captured in the past by the imaging means is the position or orientation determined from the captured images sequentially obtained in the process of imaging by the imaging means, or the position or position thereof. Other values indicating the posture, for example, the position and posture of the target object determined from the captured image of the previous frame. The position and / or orientation values that the target object can take structurally are the position and orientation of the target object that can be physically determined from the environment in which the target object is placed and the positional relationship between the target object and the imaging means. It is a range. The history of the position or orientation of the target object accumulates the position and orientation that the target object has taken in the past, and shows the tendency of the position and orientation of each target object. It will indicate your posture.

  In the image processing apparatus of the present invention, the estimated value generation means includes at least one of a value related to the position or / and orientation of the target object in the captured image captured in the past by the imaging means, and a history of the position or / and orientation of the target object. A configuration may be adopted in which the density of a plurality of estimated values to be generated is changed based on.

  The estimated value generation means of the image processing apparatus includes a plurality of values based on at least one of a value related to the position or / and orientation of the target object in the captured image captured in the past by the imaging means, and a history of the position or / and orientation of the target object. Change the setting interval of the estimated value to be generated. In other words, instead of setting the position and orientation at a fixed interval when generating multiple estimates, the estimated value setting interval is narrowed (higher density) near the position and orientation that the target object is likely to take. In the vicinity of the position or posture where the target object is unlikely to be taken, the setting interval of the estimated value is widened (the density is lowered). As described above, in the image processing apparatus, the number of estimated values can be reduced and the processing load can be reduced by changing the density of a plurality of estimated values.

  In the image processing apparatus of the present invention, the fitness calculation means converts the three-dimensional position of each feature point of the template by each estimated value generated by the estimated value generating means, and images the converted three-dimensional position by the imaging means. Projecting on the captured image, evaluating the probability of the feature point at the projection position based on the evaluation information of the template and the information around the projection position of the captured image, and calculating the degree of fitness based on the evaluation value It is good.

  In the degree-of-fit calculation means of this image processing apparatus, for each estimated value, the three-dimensional position of each feature point of the template is converted according to the position or / and orientation of each estimated value, and the converted three-dimensional position is Each is projected onto a two-dimensional captured image. That is, the reference three-dimensional position of each feature point is moved according to the position and orientation of the estimated value, and the two-dimensional position on the captured image of the feature point at the three-dimensional position according to the position and orientation of the estimated value is obtained. Then, the suitability calculation means evaluates the probability of the feature point based on the template evaluation information and the information around the projection position of the captured image for each feature point, and the suitability based on the evaluation value for the plurality of feature points Is calculated. In other words, in the projected position on the captured image that retains the three-dimensional positional relationship of the feature point according to the position and orientation of the estimated value, the image information around the projected position is the evaluation information of the feature point corresponding to the position. The degree of similarity is evaluated to determine how much the estimated value matches the position and orientation of the target object in the captured image.

  In the image processing apparatus of the present invention, the fitness level calculation means converts information around the projection position of the captured image into the same physical quantity as the template evaluation information.

  In the degree-of-fit calculation means of this image processing apparatus, when evaluating, the image information around the projection position is converted into the same physical quantity as the physical quantity of the template evaluation information, and the evaluation value is obtained. For example, when the evaluation information is a luminance pattern, information around the projection position obtained from the captured image is converted into a luminance pattern.

  In the above-described image processing apparatus of the present invention, the degree-of-fit calculation means may be configured to be a constant value when the evaluation value of the probability of the feature point at the projection position is a predetermined value or less.

  The degree-of-fit calculation means of this image processing apparatus determines whether or not the evaluation value of the probability of the feature point at each projection position is a predetermined value, and sets the evaluation value to a constant value if it is less than or equal to the predetermined value. In this way, in the image processing apparatus, by removing bad evaluation values that are less than or equal to a predetermined value, it is possible to prevent the evaluation value that has been lowered due to the influence of noise or the like from affecting the fitness.

  In the image processing apparatus of the present invention, the fitness calculation means may be configured to generate a data structure composed of evaluation values at each projection position for each feature point each time the target object in the captured image changes.

  When the target object in the captured image changes (for example, when the captured image becomes the next frame), the fitness calculation unit of the image processing apparatus generates a data structure including evaluation values at each projection position on the image for each feature point. To do. By generating such a data structure, the evaluation value at the same projection position is already stored in the data structure of a certain feature point in the process of sequentially obtaining evaluation values for a plurality of estimated values (that is, the same When the projection position is projected again), it is not necessary to obtain an evaluation value for the projection position. Therefore, in the image processing apparatus, it is no longer necessary to calculate the evaluation value for the same projection position of the feature point, and the processing load can be reduced.

  In the image processing apparatus of the present invention, the fitness level calculating means may be configured to calculate the fitness level using a feature point having a high evaluation value among a plurality of feature points.

  The degree-of-fit calculation means of this image processing apparatus obtains evaluation values for a plurality of feature points for each estimated value, and uses only the evaluation values of the plurality of feature points that have a high evaluation value to determine the degree of fit. calculate. As described above, in the image processing apparatus, by using only a high evaluation value, an evaluation value that is lowered due to the influence of noise or the like, an evaluation value of a feature point that is not projected on the captured image due to the position or orientation of the estimated value, and the like. It is possible to prevent the degree of fitness from being affected and to obtain a highly accurate fitness level.

  In the image processing apparatus of the present invention, the fitness level calculating means may be configured to calculate a statistical quantity of evaluation values of a plurality of feature points and use the statistical quantity as the fitness level.

  The degree-of-fit calculation means of this image processing apparatus calculates the statistic (e.g., sum, average) of evaluation values for a plurality of feature points for each estimated value, and uses the statistic as the degree of fit. Since the statistic of the evaluation value of each feature point is used when evaluating the fitness level, the fitness level changes according to the overall similarity. As a result, the robustness with respect to changes in appearance due to the orientation, position, illumination conditions, etc. of the object is increased, and convergence to a local erroneous position or posture is suppressed.

  In the image processing apparatus according to the present invention, the determination means includes a fitness value of a value within a predetermined range from the maximum value of the fitness level, wherein the maximum value of the fitness level is a predetermined value or more, the number of fitness levels equal to or greater than the predetermined value is a predetermined number or more. It is also possible to determine that the target object exists in the captured image when at least one condition satisfying at least a predetermined number is satisfied.

  In the determination means of the image processing device, the maximum value of the fitness values in the fitness values of the plurality of estimated values is equal to or greater than a predetermined value, the number of fitness values greater than or equal to the predetermined value is equal to or greater than a predetermined value, It is determined that the target object exists in the captured image when the degree of the degree of matching of the values (for example, the degree of matching of 90% or more of the maximum value of the degree of matching) satisfies any one of the predetermined number or more. When the degree of matching is high, it can be estimated that corresponding feature points exist at each projection position of the captured image, so it can be determined that the target object exists in the captured image.

  The image processing apparatus according to the present invention may be configured to store an estimated value of the position and / or orientation of the target object and a fitness for the estimated value.

  In this image processing apparatus, the estimated value of the position and / or orientation of the target object and the degree of fitness for the estimated value are stored. By storing these pieces of information, the history of the position and orientation of the target object can be accumulated from the estimated value having a high degree of fitness. Therefore, from this stored information, the position and orientation that the target object is likely to take can be found. Therefore, in the image processing apparatus, the estimated value can be generated by the estimated value generation means using the stored information.

  In the image processing apparatus of the present invention, when the target object is black eyes or black eyes and white eyes, the information holding means holds a black eye model as a template when the target object is black eyes, and when the target object is black eyes and white eyes Holds a black eye model and a white eye model as templates.

  In the image processing apparatus of the present invention, when the target object is a black eye or a black eye and a white eye, the suitability calculation means, as the suitability of the estimated value, if the target object is a black eye, the black eye region at the projected position of the captured image When the target object is black eyes and white eyes, the average brightness value of the black eye area and the average brightness value of the white eye area at the projection position of the captured image are calculated.

  An image processing method according to the present invention includes an imaging step, an estimated value generating step for generating a plurality of estimated values of the position and / or orientation of the target object, a captured image captured in the imaging step, and a two-dimensional image of the target object. A fitness calculation step for calculating the fitness of a plurality of estimated values generated in the estimated value generation step based on a template comprising evaluation information and three-dimensional position information for a plurality of feature points, and a plurality of parameters calculated in the fitness calculation step And a determination step of determining the presence / absence, position, and / or orientation of the target object based on the degree of matching of the estimated values.

  In the estimated value generation step of the image processing method of the present invention, a value related to the position or / and orientation of the target object in the captured image captured in the past in the imaging step, a value related to the position or / and orientation that the target object can take structurally, The estimated value may be generated based on at least one of the position and / or orientation history of the target object.

  In the estimated value generation step of the image processing method of the present invention, at least one of a value related to the position or / and orientation of the target object and a history of the position or / and orientation of the target object in the captured image captured in the past in the imaging step. It is good also as a structure which changes the density of the estimated value produced | generated based on plural.

  In the adaptability calculation step of the image processing method of the present invention, the three-dimensional position of each feature point of the template is converted by each estimated value generated in the estimated value generating step, and the converted three-dimensional position is imaged by the imaging means. Projecting the captured image, evaluating the probability of the feature point at the projection position based on the evaluation information of the template and the information around the projection position of the captured image, and calculating the fitness based on the evaluation value Also good.

  In the fitness calculation step of the image processing method of the present invention, information around the projection position of the captured image is converted into the same physical quantity as the template evaluation information.

  The adaptability calculation step of the image processing method of the present invention may be configured to be a constant value when the evaluation value of the probability of the feature point at the projection position is a predetermined value or less.

  The fitness calculation step of the image processing method of the present invention may be configured to generate a data structure composed of evaluation values at each projection position for each feature point each time the target object in the captured image changes.

  The fitness calculation step of the image processing method of the present invention may be configured to calculate the fitness using a feature point having a high evaluation value among a plurality of feature points.

  In the fitness calculation step of the image processing method of the present invention, a statistic of evaluation values of a plurality of feature points may be calculated, and the statistic may be used as the fitness.

  In the determination step of the image processing method of the present invention, the maximum value of the fitness is a predetermined value or more, the number of fitnesses of the predetermined value or more is a predetermined number or more, and the fitness of a value within a predetermined range from the maximum value of the fitness is determined. A configuration may be adopted in which it is determined that the target object exists in the captured image when at least one condition satisfying a predetermined number is satisfied.

  The image processing method of the present invention may be configured to store an estimated value of the position and / or orientation of the target object and a fitness for the estimated value.

  In the image processing method of the present invention, when the target object is black eyes or black eyes and white eyes, the template is a black eye model when the target object is black eyes, and a black eye model when the target object is black eyes and white eyes. It is a white-eye model.

  In the image processing method of the present invention, when the target object is black eyes or black eyes and white eyes, in the fitness level calculating step, as the fitness level of the estimated value, the black eye region at the projection position of the captured image when the target object is black color When the target object is black eyes and white eyes, the average brightness value of the black eye area and the average brightness value of the white eye area at the projection position of the captured image are calculated.

  The image processing method has the same effects as the above-described image processing apparatuses.

  According to the present invention, the presence / absence, position, and / or orientation of a target object can be detected with high accuracy from a captured image with a small amount of data.

  Embodiments of an image processing apparatus and an image processing method according to the present invention will be described below with reference to the drawings.

  In the present embodiment, the present invention is applied to an image processing apparatus in which a target object is a human face or a human eyeball. In this embodiment, there are three forms, the first embodiment is a form for estimating the position and posture of the face, the second embodiment is a form for determining the presence or absence of the face, The third embodiment is a mode for estimating the posture of the eyeball. In each of the embodiments, an image processing apparatus for performing processing in advance to obtain information to be held and an image processing apparatus for determining the presence / absence, position, and orientation of a target object are configured. For example, the target person is a driver of a vehicle such as an automobile, and is used for estimating the position, posture, line of sight, etc. of the driver.

  The first embodiment will be described with reference to FIGS. FIG. 1 is a configuration diagram of an image processing apparatus for modeling processing according to the first embodiment and the second embodiment. FIG. 2 is a configuration diagram of an image processing apparatus for face position / posture estimation processing according to the first embodiment. FIG. 3 is an example of a captured image of the face captured by the first camera of FIG. FIG. 4 is an image showing feature points extracted from the captured image of FIG. FIG. 5 is an example of a three-dimensional model including points corresponding to feature points. FIG. 6 is a diagram illustrating a three-dimensional position of each feature point in FIG. FIG. 7 is an example of the captured images of the faces captured by the first camera and the second camera of FIG. FIG. 8 is an image showing feature points respectively extracted from the two captured images of FIG. FIG. 9 is an example of a captured image of a face captured by the camera of FIG. FIG. 10 is an image showing the face area detected from the captured image of FIG. FIG. 11 is an image showing a region similar to the reference luminance pattern extracted from the face region of the captured image of FIG. FIG. 12 is an explanatory diagram of the projection onto the two-dimensional image from the reference three-dimensional position of the feature point according to the estimated value of the face position / posture in the first stage. FIG. 13 is an example of a normalized correlation value of each feature point in the first stage. FIG. 14 is an explanatory diagram of the projection from the reference three-dimensional position of the feature point onto the two-dimensional image according to the estimated value of the face position / posture in the second stage. FIG. 15 is an example of a normalized correlation value of each feature point in the second stage. FIG. 16 is an example of a map including normalized correlation values at each projection position for each feature point.

  In the first embodiment, an image processing apparatus 1 and an image processing apparatus 11 are configured. The image processing apparatus 1 is an image processing apparatus for modeling processing. The image processing apparatus 11 is an image processing apparatus for face position / posture estimation processing.

  The configuration of the image processing apparatus 1 will be described. The image processing apparatus 1 obtains a luminance pattern of a plurality of feature points and a three-dimensional position of each feature point as information held by the image processing apparatus 11 before processing by the image processing apparatus 11. . For this purpose, the image processing apparatus 1 includes a first camera 2, a second camera 3, and an image processing unit 4. The image processing unit 4 includes a feature point extraction unit 4a, a feature point three-dimensional position estimation unit 4b, and an information holding unit 4c by executing an application program (software) for modeling processing on a computer such as a personal computer. The Although two first cameras 2 and second cameras 3 are provided, there are cases where only one first camera 2 can be used and two cameras 2 and 3 can be used. Two cases will be described.

  The first camera 2 and the second camera 3 are stereo cameras arranged on the imaging target in parallel with a predetermined interval, and simultaneously capture two captured images with different viewpoints. The cameras 2 and 3 are digital cameras including an image sensor such as a CCD [Charge coupled device], and transmit a captured image formed of digital image data to the image processing unit 4 as an image signal. The positional relationship between the two cameras 2 and 3 and the internal and external parameters of each camera 2 and 3 can be measured, and these pieces of information are stored in the image processing unit 4 in advance. Since the image processing unit 4 can perform processing if there is at least luminance information, the cameras 2 and 3 may be color cameras or monochrome cameras. In the case of a color camera, the image processing unit 4 converts the color image into a luminance image.

  When only one first camera 2 is used, the person to be recognized is directed toward the first camera 2. Therefore, the first camera 2 captures the face in front of the person and transmits the image signal to the image processing unit 4. FIG. 3 shows an example of a captured image BI captured by the first camera 2. When two cameras 2 and 3 are used, the person to be recognized is directed toward the first camera 2. Therefore, the first camera 2 captures the person's face from the front and transmits the image signal to the image processing unit 4. In the second camera 3, the person's face is imaged at the same time as the first camera 2 from a position slightly deviated from the front, and the image signal is transmitted to the image processing unit 4. FIG. 7 shows an example of the captured image BI1 captured by the first camera 2 and the captured image BI2 captured by the second camera 3.

  When only one first camera 2 is used, the feature point extraction unit 4a displays a characteristic point (feature point) whose luminance is characteristic from the inside of the face of the captured image BI captured by the first camera 2 compared to the surroundings. Extract 3 or more. The feature points are, for example, locations that are boundaries with the skin, such as the eyes of each eye, each eye's head, the left and right holes of the nose, and the left and right edges of the mouth, where the brightness is clearly different from the surrounding area. is there. Further, the feature point extraction unit 4a uses, as a reference brightness pattern, a luminance pattern of a rectangular area centered on the feature point as a feature amount for evaluating the likelihood of each feature point (probability as a feature point) from the captured image BI. Extract. The feature point extraction unit 4a stores the luminance pattern of each feature point in the information holding unit 4c. FIG. 4 shows reference luminances for six feature points extracted from the face of the captured image BI captured by the first camera 2 (each point on the left and right corners, each point on the eyes, and each point on the left and right ends of the mouth). Patterns IPa, IPb, IPc, IPd, IPe, and IPf are shown. Note that points extracted by a computer operator may be used as feature points. In addition, if there are points that are desirable for subsequent processing empirically, images around such points are extracted, and feature points are extracted as positions having a similar luminance pattern based on statistical features. It may be.

  When two cameras 2 and 3 are used, the feature point extraction unit 4a extracts feature points from the face of the captured image BI1 captured by the first camera 2 in the same manner as when only the first camera 2 is used. Three or more are extracted, and a reference luminance pattern of each feature point is extracted from the captured image BI1. The feature point extraction unit 4a stores the luminance pattern of each feature point in the information holding unit 4c. FIG. 8 shows reference luminance patterns IP1a, IP1b, IP1c, IP1d, IP1e, and IP1f for six feature points extracted from the face of the captured image BI1 captured by the first camera 2.

  When only one first camera 2 is used, the feature point 3D position estimation unit 4b extracts points corresponding to each feature point from the average 3D model of the face, and in the coordinate system of the 3D model. A three-dimensional position (three-dimensional coordinate) corresponding to each feature point is read out. The face three-dimensional model is a model composed of several hundred vertices indicating the three-dimensional shape of the face facing the front, and each vertex has three-dimensional coordinates. As a 3D model generation method, using a range scanner, stereo camera, etc., the 3D shape of a face (a collection of points indicating a 3D shape) when facing the front of an average faced person get. FIG. 5 shows an example of the three-dimensional model DM.

  Further, in the feature point three-dimensional position estimation unit 4b, for each feature point, from the positional relationship between the two-dimensional coordinates in the camera coordinate system of the image of the first camera 2 and the three-dimensional coordinates in the coordinate system of the three-dimensional model, A three-dimensional position in the camera coordinate system of the first camera 2 (that is, a three-dimensional position on the image of the first camera 2) is estimated. Then, the feature point three-dimensional position estimating unit 4b stores the three-dimensional position of each feature point in the camera coordinate system in the information holding unit 4c in association with the reference luminance pattern for each feature point. FIG. 6 shows a three-dimensional position of each feature point shown in FIG. 4 in the camera coordinate system.

  The three-dimensional position obtained here is a reference position for estimating the face position and posture. Therefore, in the first embodiment, the amount of change related to the position and posture from the three-dimensional position is used as the position and posture. In addition, assuming that the coordinate system of the three-dimensional model and the camera coordinate system coincide with each other, a change amount related to the position and orientation from that state can be used as the position and orientation.

  Here, a specific method for estimating the three-dimensional position of the first camera 2 in the camera coordinate system will be described. In the following description, the state of the face of the person to be recognized facing directly in front of the first camera 2 will be described as the reference three-dimensional position. The two-dimensional coordinates of each feature point on the image captured by the first camera 2, the internal parameters and external parameters of the camera, and the three-dimensional coordinates of each feature point in the three-dimensional model coordinate system are known. Therefore, in order to obtain the three-dimensional position in the unknown camera coordinate system, the third order is obtained when the positional relationship between the first camera 2 and the three-dimensional model (the positional relationship between the camera coordinates and the three-dimensional model coordinate system) is determined. It is estimated whether each feature point on the original model is reflected at the projected position of each feature point on the image.

  As a method for performing this estimation, for example, POSIT (D. DeMenthon and L.S. Davis, “Model-Based Object Pose in 25 Lines of Code”, International Journal of Computer Vision, 15, pp. 123-141, june. 1995.) is used. By using this method, the position and orientation of each feature point on the three-dimensional model in the camera coordinate system can be calculated. In the feature point three-dimensional position estimation unit 4b, the three-dimensional position in the camera coordinate system of the first camera 2 is calculated from the three-dimensional position (X, Y, Z) in the three-dimensional model coordinate system with respect to each feature point by Expression (1). (X ′, Y ′, Z ′) is calculated.

  In Expression (1), t is a position (translation vector) obtained by POSIT, and R is an attitude (rotation matrix) obtained by POSIT. The three-dimensional position (X ′, Y ′, Z ′) is a reference three-dimensional position in the camera coordinate system.

  When two cameras 2 and 3 are used, the feature point three-dimensional position estimation unit 4b is similar on the captured image BI2 of the second camera 3 corresponding to each feature point on the captured image BI1 of the first camera 2. Detect position. As this detection method, for example, the following method is used. The positional relationship, internal parameters, and external parameters of the cameras 2 and 3 are known. Therefore, the epipolar constraint shown in Expression (2) is the projection position m1, m2 (extended coordinates) on each captured image when the first camera 2 and the second camera 3 capture the same point of a certain object in the three-dimensional space. ).

  m1 is a projection position on a captured image of the first camera 2 at a certain point in the three-dimensional space, and is expressed by Expression (3) and known. m2 is a projection position on a captured image of the second camera 3 at a certain point in the three-dimensional space, and is expressed by Expression (4) and unknown. F is a basic matrix and is represented by Expression (5). A1 and A2 are internal matrices of the cameras 2 and 3, and are known. T is represented by the matrix of equation (6), and (t1, t2, t3) are each element of the translation vector. R is a rotation matrix from the camera coordinate system of the first camera 2 to the camera coordinate system of the second camera 3 in the camera coordinate system of the first camera 2. Since T (position) and R (attitude) in Expression (5) are calculated as external parameters, the basic matrix F can be calculated from Expression (5).

  Therefore, the feature point three-dimensional position estimation unit 4b uses the m1 which is the two-dimensional coordinate of each feature point extracted from the captured image of the first camera 2 and uses the captured image of the second camera 3 by Expression (2). Calculate the corresponding position above. In the feature point three-dimensional position estimation unit 4b, a position similar to each reference luminance pattern extracted from the captured image of the first camera 2 in an area where each corresponding position on the captured image of the second camera 3 exists. Search for (feature points). Then, the feature point three-dimensional position estimation unit 4b extracts, for each feature point, a luminance pattern of a rectangular area centered on the searched similar position from the captured image of the second camera 3 as a reference luminance pattern, and the reference luminance The pattern is stored in the information holding unit 4c extracted from the captured image of the first camera 2. FIG. 8 shows reference luminance patterns IP2a, IP2b, IP2c, IP2d, IP2e, and IP2f extracted from the captured image BI2 captured by the second camera 3. As this search method, for example, template matching is used. The template matching is expressed by, for example, Expression (7), and the similarity norm (x, y) at the coordinates (x, y) on the captured image of the second camera 3 is obtained.

f (x, y) is a luminance value at coordinates (x, y) on the captured image of the second camera 3. g (u, v) is a luminance value at the coordinates (u, v) of the reference luminance pattern extracted from the captured image of the first camera 2. f _ave is the average brightness of an area having the same size as the reference brightness pattern centered on the coordinates (x, y) on the captured image of the second camera 3. g _ave is the average luminance of the reference luminance pattern. V is a set of elements of coordinates that can be taken in the vertical direction of the reference luminance pattern. U is a set of coordinate elements that can be taken in the horizontal direction of the reference luminance pattern.

  Further, the feature point three-dimensional position estimation unit 4b uses the two-dimensional coordinates of the feature points extracted from the captured image of the first camera 2 and the two-dimensional coordinates of the feature points searched from the captured image of the second camera 3, Based on the principle of triangulation, the three-dimensional position (reference three-dimensional position) of each feature point in the camera coordinate system of the first camera 2 is calculated. Then, the feature point three-dimensional position estimating unit 4b stores the three-dimensional position of each feature point in the camera coordinate system in the information holding unit 4c in association with the reference luminance pattern for each feature point.

  The information holding unit 4c is configured in a predetermined memory area, and holds a reference luminance pattern for three or more feature points to be recognized and a three-dimensional position in the camera coordinate system of the first camera 2 as a template. When only the first camera 2 is used, only the reference luminance pattern obtained from the captured image of the first camera 2 is retained, and when using the cameras 2 and 3, it is obtained from each captured image of the cameras 2 and 3, respectively. The obtained reference luminance pattern is held.

  The configuration of the image processing apparatus 11 will be described. The image processing apparatus 11 holds a template for each feature point generated by the image processing apparatus 1 and estimates the position and orientation of the face of the person to be recognized on the captured image using the template. At that time, the image processing apparatus 11 sets a large number of estimated values of the position and orientation for the captured image of each frame, and calculates the degree to which each estimated value matches the position and orientation of the face in the captured image. Then, the position and orientation are estimated based on the fitness. In particular, in the image processing apparatus 11, in order to estimate the position and orientation with high accuracy, the first-stage estimated value generation and the fitness calculation are performed, and the estimated values (position and orientation) narrowed down in the first stage are further obtained. Based on this, the second-stage estimated value generation and fitness calculation are performed. For this purpose, the image processing apparatus 11 includes a camera 12 and an image processing unit 13. The image processing unit 13 executes an application program for face position / posture estimation processing on a computer to thereby perform an information holding unit 13a, a storage analysis unit 13b, an estimated value generation unit 13c, a fitness calculation unit 13d, a face position / posture An output unit 13e is configured. This computer may be the same computer as the image processing apparatus 1 or may be a different computer. In the case of the same computer, the information holding unit may be shared.

  In the first embodiment, the camera 12 corresponds to the imaging unit described in the claims, the information holding unit 13a corresponds to the information holding unit described in the claims, and the estimated value generating unit 13c. Corresponds to the estimated value generation means described in the claims, the fitness calculation unit 13d corresponds to the fitness calculation means described in the claims, and the face position / posture output unit 13e corresponds to the claims. This corresponds to the determination means described.

  The camera 12 is a digital camera including an image sensor such as a CCD, and transmits a captured image made up of digital image data to the image processing unit 13 as an image signal. At this time, the camera 12 continuously captures images in time and outputs continuous captured image (moving image) data at regular time intervals (for example, 1/30 seconds). The internal parameters and external parameters of the camera 12 can be measured, and these pieces of information are stored in the image processing unit 13 in advance. Since the image processing unit 13 can perform processing if there is at least luminance information, the camera 12 may be a color camera or a monochrome camera. In the case of a color camera, the image processing unit 13 converts a color image into a luminance image. Incidentally, when the image processing apparatus 11 is mounted on an automobile or the like, the camera 12 is disposed in a position where the face of the driver sitting in the driver's seat can be imaged from the front in the passenger compartment.

  The information holding unit 13a is configured in a predetermined memory area, and holds a reference luminance pattern for a plurality of feature points of a recognition target face obtained by the image processing apparatus 1 and a three-dimensional position in a camera coordinate system as a template.

  The storage analysis unit 13b is configured in a predetermined memory area, and stores the position and orientation estimated values generated by the estimated value generating unit 13c in association with the respective fitness values calculated by the fitness level calculating unit 13d. In addition, the estimated face position and posture in the captured image of each frame output from the face position / posture output unit 13e are stored. In the memory analysis unit 13b, the position and orientation of the face taken by the person to be recognized in the past are accumulated as a history. Further, the memory analysis unit 13b identifies combinations of positions and postures that can be taken in the six-dimensional space based on the six parameters of the positions and postures from the history of the positions and postures. In addition, the storage analysis unit 13b sets a normal distribution having a peak near a position and posture that can be frequently taken from the position and posture history, and generates a probability density function by superimposing a plurality of normal distributions. The storage analysis unit 13b may accumulate a history using the face position and posture output from the face position / posture output unit 13e, or may indicate a large number of estimated values and the fitness of the estimated values. The history may be accumulated using an estimated value whose fitness is greater than a threshold value.

  The image processing unit 13 stores the captured image in a predetermined memory area every time an image signal for the current frame is received. Then, the image processing unit 13 determines whether or not the face position and posture are estimated in the captured image of the previous frame. This is because if the current frame is the initial frame or if the face cannot be detected in the previous frame, the position and orientation of the face cannot be estimated in the previous frame, and therefore the subsequent processing cannot be performed. Therefore, a face is detected from the current frame, the approximate position and orientation are estimated, and the position and orientation of the previous frame are set. FIG. 9 shows an example of a captured image II of a certain frame input from the camera 12.

  When the face position and posture are not estimated in the previous frame, the image processing unit 13 performs face detection to determine whether or not a face exists in the captured image of the current frame. Find the area that exists. FIG. 10 shows a face area FA from which a face can be detected from the captured image II. As the face detection method, a conventional face detection process may be used, or the face presence / absence determination process in the second embodiment may be used. If the face cannot be detected, the image processing unit 13 waits for a captured image of the next frame.

  Next, the image processing unit 13 searches for points corresponding to each feature point from the face area FA. As this search method, for example, using template matching and using the reference luminance pattern of each feature point held by the information holding unit 13a, a position similar to each reference luminance pattern according to the above equation (7). Search for (feature points). FIG. 11 shows regions RAa, RAb, RAc, RAd, RAe, and RAf similar to each reference luminance pattern searched from the face region of the captured image II, and the center of each region is a corresponding point. Here, when three or more corresponding points cannot be searched, the subsequent process cannot be performed, and the image processing unit 13 waits for a captured image of the next frame.

  Next, in the image processing unit 13, for each feature point, the camera of the camera 12 with respect to the reference three-dimensional position is determined based on the correspondence relationship between the two-dimensional coordinates of the searched corresponding point and the reference three-dimensional position held in the information holding unit 13 a. The approximate position and orientation in the coordinate system are calculated and used as the estimated value of the position and orientation in the current frame. This estimated value is used as the estimated value of the position and orientation of the previous frame in the processing in the next frame. As this estimation method, for example, POSIT is used.

  When the position and orientation of the previous frame are estimated, the estimated value generation unit 13c provides many estimated values of the position and orientation of the face in the captured image of the current frame that can be obtained from the position and orientation (reference value) in the previous frame. Generate. The range of the position and orientation that can be taken is a range obtained by adding and subtracting each maximum value that can change within the time between frames to the six parameters of the position and orientation of the previous frame for each of the six parameters of position and orientation. . However, if this range includes a range in which the face to be recognized cannot be structurally taken, it is a range excluding a range that cannot be structurally taken. In other words, the range of positions and postures in which the face can operate is physically determined from the environment where the face of the person to be recognized is placed and the positional relationship between the face and the camera 12. An estimate is generated. The maximum value that can be changed within the time between frames is to measure the position and posture of the face in the environment where the face is placed in advance, and set it in advance from the maximum value of the change obtained from the measurement. Alternatively, it may be set based on a change in position and orientation between the previous frame and the previous frame.

  It is equally probable what values each parameter actually takes in the range of the six parameters of the position and orientation set as described above. Therefore, the estimated value generation unit 13c randomly extracts n times from the uniform distribution in the total six-dimensional space that takes the values in the range of the minimum value and the maximum value for each of the six parameters of position and orientation. The value is an estimated value est1 (i) (i = 1,..., N) of the position and orientation. Specifically, the estimated value generation unit 13c generates an estimated value using Expression (8).

Est_t _x is the estimate of x coordinate of the position, Est_t _y is the estimate of the y coordinate of the position, Est_t _z is an estimate of the z coordinate of the position, rotation angle around the X axis of Est_deg _x attitude Est_deg _y is an estimated value of the rotation angle around the Y axis of the posture, and est_deg _z is an estimated value of the rotation angle of the posture around the Z axis. Old_t _x is a previous frame of x coordinate of the position, Old_t _y is the previous frame values for the y-coordinate of the position, Old_t _z is the previous frame values for z coordinate of the position, Old_deg _x around the X axis of orientation a previous frame value of the rotation angle, old_deg _y is the previous frame value of the rotational angle around the Y axis orientation, old_deg _z is the previous frame value of the rotational angle around the Z axis of orientation. max_t _x is the maximum inter-frame change value of the x coordinate of the position, max_t _y is the maximum inter-frame change value of the y coordinate of the position, max_t _z is the maximum inter-frame change value of the z coordinate of the position, and max_deg _x Is the maximum inter-frame change of the rotation angle around the X axis of the posture, max_deg _y is the maximum inter-frame change of the rotation angle around the Y axis of the posture, and max_deg _z is the rotation angle around the Z axis of the posture. This is the maximum change between frames. u (-1,1) is a uniform distribution between -1 and 1.

  As a method for generating an estimated value, an estimated value may be generated based on a combination of a position and a posture that can be taken in the six-dimensional space derived from the history accumulated in the storage analysis unit 13b. The estimated value may be generated based on the probability density function derived from the history accumulated in the memory analysis unit 13b, or the estimated value is not a uniform distribution but near the position and posture of the previous frame. If there is a high possibility that the position and orientation of the current frame are present, the estimated value may be generated with a normal distribution having the peak position and orientation of the previous frame. By generating an estimated value using such a normal distribution, the setting interval of estimated values becomes denser as it gets closer to the peak, and the setting interval of estimated values becomes sparser as it gets away from the peak.

  Furthermore, in the estimated value generation unit 13c, when the fitness level for the first-stage estimated value est (i) is calculated by the fitness level calculation unit 13d, it is determined whether or not the maximum fitness level exceeds a threshold value. Whether or not this threshold is a degree of fitness that can be estimated that a face is present on the captured image (that is, each of the projected positions on the captured image of the current frame has a probable feature point). It is a threshold value for determining whether or not. If the maximum fitness is less than or equal to the threshold value, it is determined that a face cannot be detected on the captured image of the current frame, and the estimated value generation unit 13c waits for the captured image of the next frame. As this determination method, it may be determined that a face is present when the number of estimated fitness values exceeding a threshold is equal to or greater than a predetermined number, or a value equal to or greater than a certain percentage of the maximum fitness value. It may be determined that a face exists when the number of goodness-of-fit estimates that take (for example, 90% or more of the maximum value) is a predetermined number or more.

  On the other hand, if the maximum fitness exceeds the threshold value, it is determined that a face exists, and the estimated value generation unit 13c generates estimated values of further narrowed positions and postures. Since it can be estimated that there is a true position and posture near the position and posture of the estimated value having the maximum fitness in the first stage, a large number of estimated values of the position and posture are concentrated again in the vicinity. Set. Therefore, the estimated value generation unit 13c assumes a normal distribution in which the position and orientation of the estimated value having the maximum fitness is an average value, and randomly extracts n ′ times from the normal distribution. Is an estimated value est2 (i) (i = 1,..., N ′) of the position and orientation. Alternatively, in the estimated value generation unit 13c, the position and orientation of the estimated value of the fitness that takes a value equal to or greater than the maximum value of the fitness and the maximum constant ratio are extracted as samples, and the position is extracted from all the extracted samples. The average and variance are calculated for each of the posture parameters. Then, the estimated value generation unit 13c calculates the estimated value est of each parameter by Expression (9), and generates estimated values est2 (i) (i = 1,..., N ′).

N (a, b) is a normal distribution with mean a and variance b. mean is the average of the sample of parameters. σ ² is the variance of the parameter sample.

  In the fitness calculation unit 13d, for each estimation value generated by the estimation value generation unit 13c, the information holding unit 13a holds the six parameters of the position and orientation of each estimation value according to Expression (10). The reference three-dimensional position of each feature point is moved.

(X, Y, Z) is the reference three-dimensional position (three-dimensional coordinate) of the feature point held in the information holding unit 13a, and (X ′, Y ′, Z ′) is the tertiary of the feature point after movement. The original position (three-dimensional coordinates). R _z is a rotation matrix around the Z axis, and is represented by the matrix of Expression (11). R _y is a rotation matrix around the Y axis, and is represented by the matrix of Expression (12). R _x is a rotation matrix around the X axis, and is represented by the matrix of Expression (13). Est_t _x is the estimate of x coordinate of the position, Est_t _y is the estimate of the y coordinate of the position, Est_t _z is an estimate of the z coordinate of the position, rotation angle around the X axis of Est_deg _x attitude Est_deg _y is an estimated value of the rotation angle of the posture around the Y axis, and est_deg _z is an estimated value of the rotation angle of the posture around the Z axis.

  Furthermore, the fitness calculation unit 13d projects the three-dimensional position (X ′, Y ′, Z ′) after the movement onto the captured image of the camera 12 according to Expression (14). In FIG. 12, the reference three-dimensional positions 3Da, 3Db, 3Dc, 3Dd, 3De, and 3Df of the feature points are projected on the image according to the estimated value eat1 (i) generated in the first stage, 2D1b, 2D1c, 2D1d, 2D1e, and 2D1f are shown. FIG. 14 also shows the two-dimensional coordinates projected on the image according to the estimated values eat2 (i) generated in the second stage by the reference three-dimensional positions 3Da, 3Db, 3Dc, 3Dd, 3De, and 3Df of the feature points. 2D2a, 2D2b, 2D2c, 2D2d, 2D2e, and 2D2f are shown. Here, since the coordinates after the conversion in Expression (10) are expressed in the camera coordinate system, there is no need to convert the coordinate system. Further, in this projection, depending on the position and orientation of the estimated value, there is a case where it is not projected in the captured image.

  s is a scalar, A is an internal matrix of the camera 12, (u, v) is a two-dimensional coordinate on a captured image of feature points, and (X ′, Y ′, Z ′) are features after movement. The three-dimensional position (three-dimensional coordinates) of the point.

  Further, the fitness calculation unit 13d normalizes the reference luminance pattern held in the information holding unit 13a and the luminance pattern at the projection position on the captured image of the current frame for each feature point by Expression (15). A correlation value is calculated. This normalized correlation value is a value between −1 and 1, and the larger the value, the higher the degree of correlation. The closer the estimated position and orientation are to the face position and orientation of the captured image, the closer the reference luminance pattern is to the luminance pattern at the projection position extracted from the captured image of the current frame, and the larger the normalized correlation value is Become.

f (x, y) is a luminance value at coordinates (x, y) on the captured image of the current frame. g (u, v) is a luminance value at the coordinates (u, v) of the reference luminance pattern. f _ave is an average luminance of an area having the same size as the reference luminance pattern around the coordinate (x, y) on the captured image of the current frame. g _ave is the average luminance of the reference luminance pattern. V is a set of elements of coordinates that can be taken in the vertical direction of the reference luminance pattern. U is a set of coordinate elements that can be taken in the horizontal direction of the reference luminance pattern.

  When the reference three-dimensional position of each feature point is projected on the captured image in accordance with a large number of estimated values, the projected position is concentrated in a certain range and may be projected at the same position. In particular, when the position and orientation are narrowed down in the second stage, the number of cases projected at the same position increases. In this case, since the same luminance pattern is obtained at the same projection position on the captured image, the same value is obtained when the normalized correlation value is calculated. Therefore, a map is generated for each feature point so that the calculation is not repeated, and it is confirmed whether the normalized correlation value of the projection position calculated by the map has already been calculated.

  A map is set for each feature, and a table for storing normalized correlation values for all pixels of the captured image is prepared (see FIG. 16). Every time the frame changes or every time there is a change in the face position or posture of the captured image, the map is erased and newly generated. In the example shown in FIG. 16, a normalized correlation value of 0.5 is stored in (a1, b1) on the image, indicating that this projection position has already been calculated, (a1, b2 ), (A2, b1), (a2, b2), etc., indicate that normalized correlation values are not stored, and these projection positions are still being calculated.

  Whenever the projection position of a certain feature point is calculated, the goodness-of-fit calculation unit 13d refers to the map of the feature point and determines whether or not a normalized correlation value is already stored in the calculated projection position. When the normalized correlation value is already stored, the fitness calculation unit 13d extracts the normalized correlation value and uses it in post-processing. On the other hand, when the normalized correlation value is not yet stored, the fitness calculation unit 13d calculates a normalized correlation value at the projection position. Next, the goodness-of-fit calculation unit 13d determines whether the calculated normalized correlation value is smaller than a threshold value. If the calculated normalized correlation value is smaller than the threshold value, the normalized correlation value is replaced with a constant value. For example, when the normalized correlation value is smaller than 0, the normalized correlation value is replaced with 0. In this way, by eliminating a very small normalized correlation value, it is possible to prevent the normalized correlation value that has been lowered due to the influence of noise or the like from affecting the fitness. Then, the matching degree calculation unit 13d writes the normalized correlation value at the corresponding projection position of the feature point map. As described above, the fitness calculation unit 13d obtains normalized correlation values for all feature points for each estimated value. FIG. 13 shows the normalized correlation value of each feature point in the case of the first-stage estimated value est1 (i). FIG. 15 shows the normalized correlation value of each feature point in the case of the second-stage estimated value est2 (i). For some feature points, the normalized correlation value in the case of the first stage is shown. Is getting bigger.

  For each estimated value, the fitness level calculation unit 13d selects a feature point from among all the feature points, and calculates the fitness level using the normalized correlation value of the selected feature point. The goodness of fit evaluates the position and probability of a plurality of feature points at each projection position on the captured image of the current frame projected by a three-dimensional positional relationship according to the position and orientation of the estimated value, The degree of whether or not the position and orientation and the position and orientation of the face on the captured image are compatible is indicated, and the larger the value is, the more compatible it is. Specifically, the brightness pattern of each projection position on the captured image obtained by projecting each feature point of the standard three-dimensional position according to the position and orientation of the estimated value is similar to the reference brightness pattern of each feature point. , The goodness of fit increases.

  The feature point selection method used for calculating the degree of fitness may use all feature points, or may use only feature points with a normalized correlation value greater than the threshold value, or features projected in the captured image. Only the points may be used, or an average normal vector around the feature point in the coordinate system in which the three-dimensional position of the feature point is defined and an estimated value of the posture are taken into consideration on the captured image of the camera 12. Only feature points that are determined to be visible may be used. When only the feature points that are determined to be visible on the captured image of the camera 12 are used, the optical of the camera 12 is moved to the three-dimensional position after the reference three-dimensional position of the feature point is moved based on the estimated position and orientation. When the inner product of the vector formed by rotating the normal vector of the feature point based on the vector from the origin and the estimated value of the posture is larger than the threshold value of 0 or more, it is determined that the feature point can be used. In addition, although the selection method of the feature point used for the above fitness calculation is shown, you may select a feature point combining some of these selection methods.

  The calculation of the fitness may be performed for all the estimated values, but the calculation of the fitness may be stopped for an estimated value that can be predicted that the fitness is small. For example, if the average value of normalized correlation values of one or more feature points that are likely to be reliable among feature points is equal to or greater than a threshold value, the degree of fitness is calculated. The calculation of the degree may be stopped.

  As a method for calculating the fitness, a statistic of each normalized correlation value of the selected feature point is calculated. Examples of statistics include sums and average values.

  When the face position / posture output unit 13e finishes calculating the fitness for all the estimated values est2 (i) in the second stage, it uses the calculated fitness and the position and orientation of the estimated value corresponding to the fitness. Then, the position and orientation of the face in the captured image are estimated, and the estimated position and orientation are output. As the estimation method, for example, the position and orientation of the estimated value having the maximum fitness may be used, or the position and orientation may be calculated by a weighted average value based on the fitness of the estimated value having a fitness equal to or higher than the threshold. Alternatively, the position and orientation may be calculated by a weighted average value based on the fitness values of all the estimated values for which the fitness values have been calculated, or the fitness estimate value that takes a value above a certain percentage of the maximum fitness value. The position and orientation may be calculated by a weighted average value based on the degree of goodness of fit, or when the number of goodnesses of the estimated goodness of fit that takes a value of a certain percentage or more of the maximum value of goodness of fit is a predetermined number The position and orientation may be calculated by a weighted average value based on the fitness values of the estimated fitness values, or the fitness values of the estimated fitness values that take a value that exceeds a certain percentage of the maximum fitness value. If the number of is less than the predetermined number, And it may be used as the attitude. When using the weighted average value, the calculation is performed according to Equation (16).

est _i is a parameter of the estimated value i, w _i is the fitness of the estimated value i, and G is an index set of the fitness determined to be used for estimation.

  Next, the operation of the image processing apparatus 1 will be described with reference to FIG. In particular, the modeling process of the image processing unit 4 will be described with reference to the flowchart of FIG. FIG. 17 is a flowchart showing the flow of modeling processing in the image processing apparatus of FIG.

  The first camera 2 captures an image of the face of a person to be recognized facing directly in front of the first camera 2, and transmits an image signal of the captured image to the image processing unit 4. When the second camera 3 can also be used, the second camera 3 captures the face of the person to be recognized facing the front of the first camera 2 from the side at the same time as the first camera 2 and captures the image. The image signal of the image is transmitted to the image processing unit 4.

  The image processing unit 4 determines whether the first camera 2 and the second camera 3 can be used or only the first camera 2 can be used (S10).

  When it is determined in S10 that only the first camera 2 is usable, the image processing unit 4 receives an image signal from the first camera 2 and acquires a captured image of the face (S11). The image processing unit 4 extracts a plurality of feature points from the captured image of the first camera 2, and holds a luminance pattern around each feature point as a reference luminance pattern (S12). Further, the image processing unit 4 selects points corresponding to each feature point from the average three-dimensional model of the face, and extracts the three-dimensional position of each corresponding point in the coordinate system of the three-dimensional model (S13). . Then, the image processing unit 4 determines the camera of the first camera 2 from the relationship between the two-dimensional coordinates on the captured image of each feature point and the three-dimensional position of the corresponding point corresponding to each feature point in the three-dimensional model coordinate system. A three-dimensional position of each feature point in the coordinate system is estimated (S14). Further, the image processing unit 4 holds the estimated three-dimensional position (three-dimensional coordinates) in association with the reference luminance pattern as a standard three-dimensional position for each feature point.

  If it is determined in S10 that the first camera 2 and the second camera 2 can be used, the image processing unit 4 receives the image signals from the first camera 2 and the second camera 3, and converts each captured image of the face. Obtain (S15). The image processing unit 4 extracts a plurality of feature points from the captured image of the first camera 2, and holds the brightness pattern around each feature point as a reference brightness pattern (S16). For each feature point, the image processing unit 4 searches for a similar position from the captured image of the second camera 3 based on the luminance pattern of the feature point obtained from the captured image of the first camera 2 (S17). At this time, the image processing unit 4 holds the luminance pattern around each similar position on the captured image of the second camera 3 as a reference luminance pattern. Then, the image processing unit 4 determines the first camera 2 based on the relationship between the two-dimensional coordinates of the feature points on the captured image of the first camera 2 and the two-dimensional coordinates of the similar position on the captured image of the second camera 3. The three-dimensional position of each feature point in the camera coordinate system is estimated (S18). Further, the image processing unit 4 holds the estimated three-dimensional position (three-dimensional coordinates) in association with the reference luminance pattern as a standard three-dimensional position for each feature point.

  The operation of the image processing apparatus 11 will be described with reference to FIG. In particular, the face position / posture estimation process of the image processing unit 13 will be described with reference to the flowchart of FIG. 18, and the normalized correlation value calculation process of the image processing unit 13 will be described with reference to the flowchart of FIG. FIG. 18 is a flowchart showing the flow of face position / posture estimation processing in the image processing apparatus of FIG. FIG. 19 is a flowchart showing the flow of normalized correlation value calculation processing in the image processing apparatus of FIG.

  The camera 12 captures images continuously in time and transmits an image signal of the captured image to the image processing unit 13 at regular time intervals.

  The image processing unit 13 receives an image signal from the camera 12 and sequentially acquires captured images of the current frame (S20).

  The image processing unit 13 determines whether or not the face position and posture in the captured image have been estimated in the previous frame (S21). If it is determined in S21 that the position and orientation are not estimated in the previous frame, the image processing unit 13 determines whether or not a face has been detected in the captured image of the current frame (S22). If the face cannot be detected in S22, the image processing unit 13 returns to S20 and waits for the captured image of the next frame. When a face can be detected in S22, the image processing unit 13 determines whether or not three or more feature points have been detected from the face (S23). If it is determined in S23 that three or more feature points could not be detected, the image processing unit 13 returns to S20 and waits for a captured image of the next frame. If it is determined in step S23 that three or more feature points have been detected, the image processing unit 13 determines the reference from the correspondence between the detected two-dimensional coordinates of each feature point and the stored reference three-dimensional position of each feature point. The position and posture of the camera 12 in the camera coordinate system with respect to the three-dimensional position are calculated, and the position and posture are estimated values of the position and posture of the current frame (that is, this estimated value is the position and posture of the previous frame in the next frame). It becomes an estimated value) (S24). Then, the image processing unit 13 returns to S20 and waits for the captured image of the next frame.

  If it is determined in S21 that the position and orientation are estimated in the previous frame, the image processing unit 13 estimates the position and orientation estimates est1 (i in the current frame based on the position and orientation estimated in the previous frame. ) (I = 1,..., N) is generated (S25). Then, the image processing unit 13 moves the reference three-dimensional position of each feature point held by the position and orientation of each estimated value est1 (i), and images are taken by the camera 12 from each moved three-dimensional position. Projected positions (two-dimensional coordinates) projected on the captured image are calculated (S26). Further, the image processing unit 13 calculates a normalized correlation value between the luminance pattern at the projection position on the captured image and the held reference luminance pattern for each feature point in the case of est1 (i) (S27). . Then, the image processing unit 13 calculates the fitness as an evaluation value of the consistency of the estimated value est1 (i) with respect to the face position and posture of the captured image from the normalized correlation value of each feature point (S28). Through the processing so far, the first-stage estimated value est1 (i) is generated, and the fitness is calculated for each estimated value est1 (i).

  The image processing unit 13 extracts the maximum fitness from all the calculated fitness, and determines whether or not the maximum fitness exceeds a threshold value (S29). If it is determined in S29 that the face is equal to or smaller than the threshold value, it is determined that the face cannot be detected in the current frame, and the image processing unit 13 returns to S20 and waits for the captured image of the next frame.

  If it is determined in S29 that the threshold value is exceeded, the image processing unit 13 estimates the position and orientation estimated values est2 (i in the current frame near the position and orientation of the estimated value est1 (i) corresponding to the maximum fitness. ) (I = 1,..., N ′) is generated (S30). Then, the image processing unit 13 moves the reference three-dimensional position of each feature point held by the position and orientation of each estimated value est2 (i), and the camera 12 captures an image from each moved three-dimensional position. Projection positions (two-dimensional coordinates) projected on the captured image are calculated (S31). Further, the image processing unit 13 calculates a normalized correlation value between the luminance pattern at the projection position on the captured image and the held reference luminance pattern for each feature point in the case of est2 (i) (S32). . Then, the image processing unit 13 calculates the fitness of the estimated value est2 (i) from the normalized correlation value of each feature point (S33). Through the processing so far, the second-stage estimated value est2 (i) is generated, and the fitness is calculated for each estimated value est2 (i).

  The image processing unit 13 estimates the position and orientation of the face in the captured image of the current frame based on the position and orientation of the estimated value est2 (i) and the fitness calculated for each estimated value est2 (i). The estimated position and orientation are output (S34). Then, the image processing unit 13 returns to S20 and waits for the captured image of the next frame.

  In particular, when calculating the normalized correlation values in S27 and S32, when the image processing unit 13 calculates the projection positions for all the feature points, the calculated projection position is referred to the map for each feature point. Has already been projected (that is, whether or not a normalized correlation value has already been stored at the projection position of each feature point map) (S40).

  If it is determined in S40 that the projection position has not been projected yet, the image processing unit 13 determines from the captured image of the current frame the reference luminance pattern of the feature point held around the projection position. A luminance pattern is cut out with the same size (S41). Then, the image processing unit 13 calculates a normalized correlation value between the cut-out luminance pattern and the retained reference luminance pattern of the feature point (S42). Further, the image processing unit 13 determines whether or not the calculated normalized correlation value is smaller than the threshold value. When the calculated normalized correlation value is smaller than the threshold value, the normalized correlation value is replaced with a constant value (S43). Then, the image processing unit 13 writes the normalized correlation value in association with the projection position in the map of each feature point (S44).

  On the other hand, if it is determined in S40 that the calculated projection position has already been projected, the image processing unit 13 does not calculate a normalized correlation value for the feature point, and post-processes the projection from the map. A normalized correlation value of the position is extracted (S45).

  The image processing unit 13 determines whether or not the writing of normalized correlation values for all feature points has been completed (S46). If it is determined in S46 that the writing of normalized correlation values for all feature points has been completed, the image processing unit 13 proceeds to normalization correlation value calculation for the next estimated value. On the other hand, if it is determined in S46 that the writing of normalized correlation values for all feature points has not been completed, the image processing unit 13 returns to S40 and proceeds to the calculation of normalized correlation values for the next feature point. .

  According to the image processing apparatus 11, a large number of estimated values are generated, and the degree of suitability obtained by evaluating the probability as each feature point and the overall position of the feature point for each estimated value is obtained. The position and orientation of the face can be estimated with high accuracy. Further, according to the image processing apparatus 11, only the reference luminance pattern and the data with a small standard three-dimensional position are stored for each feature point, and only the processing for each feature point is performed instead of the entire image. The processing time is short.

  In the image processing apparatus 11, since the position and orientation are further narrowed down in the second stage using the position and orientation of the estimated value narrowed down in the first stage, it is possible to estimate the position and orientation with very high accuracy.

  In the image processing apparatus 11, when generating the estimated value, the estimated value is generated only in a range that the human face of the recognition target may take, so a useless estimated value is not generated and the processing load is increased. While mitigating, it suppresses estimating a local incorrect position and posture. In particular, in the image processing apparatus 11, when the position and orientation history is accumulated and the estimated value is generated in consideration of the history, the estimated value can be set in consideration of the wrinkles of individual human faces. Therefore, the estimated values can be distributed in a relatively narrow range, the robustness is improved, and the processing load is reduced.

  Since the image processing apparatus 11 uses the statistic of the normalized correlation value of each feature point when evaluating the fitness, the fitness also changes according to the overall similarity. Therefore, even if a part of the feature point is hidden or a difference in appearance occurs due to illumination fluctuation, etc., even if the similarity with the luminance pattern of that part decreases, the overall similarity is used. The degree of fitness does not decrease so much, and some of the influence can be suppressed. Further, even if a portion similar to some feature points exists on the captured image, if the overall similarity is not high, the degree of matching does not increase so much, and the influence of that portion can be suppressed. Thus, the robustness with respect to the change in appearance is high, and the convergence to an erroneous position or posture is suppressed.

  Since the image processing device 11 sets a map for each feature point, the normalized correlation value is not calculated redundantly for the same projection position, and the processing load can be reduced. In addition, when estimating the position and orientation, the image processing apparatus 11 has a filtering effect on the estimated position and orientation when an estimated value of the fitness that is equal to or greater than a certain ratio of the fitness of the maximum value is also used. is there. For this reason, the estimated position and orientation become continuous values and change smoothly.

  The second embodiment will be described with reference to FIG. FIG. 20 is a configuration diagram of an image processing apparatus for face presence / absence determination processing according to the second embodiment.

  In the second embodiment, an image processing apparatus 1 and an image processing apparatus 21 similar to those in the first embodiment are configured. The image processing device 21 is an image processing device for face presence / absence determination processing. Note that the description of the image processing apparatus 1 has been given in the first embodiment, and the description thereof will be omitted. Also in the second embodiment, a template of a plurality of feature points including a luminance pattern of a plurality of feature points and a three-dimensional position of each feature point in the face of a person to be recognized generated by the image processing apparatus 1 is subjected to image processing. It is held by the device 21.

  The configuration of the image processing device 21 will be described. The image processing device 21 holds a template for each feature point generated by the image processing device 1, and uses the template to determine the presence or absence of a human face to be recognized on the captured image. At that time, the image processing apparatus 21 sets a large number of estimated values of the position and orientation, calculates the degree to which each estimated value is adapted to the position and orientation of the face in the captured image, and based on the degree of adaptation To determine the presence or absence of a face. For this purpose, the image processing apparatus 21 includes a camera 22 and an image processing unit 23. The image processing unit 23 includes an information holding unit 23a, a storage analysis unit 23b, an estimated value generation unit 23c, a fitness calculation unit 23d, and a face presence / absence output unit 23e by executing an application program for face presence / absence determination processing on a computer. Composed. This computer may be the same computer as the image processing apparatus 1 or may be a different computer. In the case of the same computer, the information holding unit may be shared.

  In the second embodiment, the camera 22 corresponds to the imaging unit described in the claims, the information holding unit 23a corresponds to the information holding unit described in the claims, and the estimated value generating unit 23c. Corresponds to the estimated value generating means described in the claims, the fitness calculating unit 23d corresponds to the fitness calculating means described in the claims, and the face presence / absence output unit 23e is described in the claims. It corresponds to a determination means.

  The camera 22 is the same camera as the camera 12 according to the first embodiment, and a description thereof is omitted. The information holding unit 23a is an information holding unit similar to the information holding unit 13a according to the first embodiment, and a description thereof is omitted.

  The storage analysis unit 23b is configured in a predetermined memory area, and stores the position and orientation estimation values generated by the estimated value generation unit 23c in association with the respective fitness values calculated by the fitness level calculation unit 23d. In addition, the presence / absence of a face in the captured image of each frame output from the face presence / absence output unit 23e is stored. Then, the memory analysis unit 23b uses the estimated value having a fitness value larger than the threshold value from among a large number of estimated values and the fitness values of the estimated values to record the face position and posture taken by the person to be recognized in the past. Accumulate as. Further, in the memory analysis unit 23b, as in the memory analysis unit 13b according to the first embodiment, from the position and posture history, a combination of positions and postures that can be taken in the six-dimensional space is identified, or A probability density function is generated by overlapping a plurality of normal distributions having peaks at positions and postures that can be frequently taken.

  The estimated value generation unit 23c uses the same method as the estimated value generation unit 13c according to the first embodiment to determine the position of the face in the captured image of the current frame that can be obtained from the position and orientation (reference value) in the previous frame. A large number of estimated posture values est (i) (i,..., N) are generated. In the fitness level calculation unit 23c, each estimated value est (i) is calculated for each estimated value est (i) generated by the estimated value generation unit 23c by the same method as the fitness level calculation unit 13c according to the first embodiment. The three-dimensional position obtained by moving the reference three-dimensional position of each feature point held in the information holding unit 23a using the position and orientation of the image is projected onto the captured image of the camera 22, and the normalization of each feature point is further performed. The correlation value is calculated, and the fitness is calculated using the normalized correlation value of each feature point. In the calculation of the normalized correlation value, a map for each feature point is used as in the first embodiment.

  When the face presence / absence output unit 23e finishes calculating the fitness for all the estimated values est (i), the maximum fitness is extracted from the calculated fitness and whether or not the maximum fitness exceeds a threshold value. Determine. This threshold value is a threshold value for determining whether or not the degree of fitness is such that it can be estimated that a face is present on the captured image. When the maximum fitness is less than or equal to the threshold value, the face presence / absence output unit 23e determines that no face exists on the captured image of the current frame, and outputs the determination result. On the other hand, when the maximum matching level exceeds the threshold value, the face presence / absence output unit 23e determines that a face exists on the captured image of the current frame, and outputs the determination result. As this determination method, it may be determined that a face is present when the number of estimated fitness values exceeding a threshold is equal to or greater than a predetermined number, or a value equal to or greater than a certain percentage of the maximum fitness value. It may be determined that a face is present when the number of estimated fitness values for taking is greater than or equal to a predetermined number.

  Note that it is also possible to determine all the fitness levels and threshold values, and when there are a plurality of fitness levels exceeding the threshold value, it can be determined that there are a plurality of faces on the captured image. In this case, the person's face for generating the reference luminance pattern held as the template is set as the average person's face, or similar to the person's face for generating the reference luminance pattern held as the template. The presence / absence of a person's face can be determined.

  The operation of the image processing device 21 will be described with reference to FIG. In particular, the face presence / absence determination process of the image processing unit 23 will be described with reference to the flowchart of FIG. FIG. 21 is a flowchart showing a flow of face presence / absence determination processing in the image processing apparatus of FIG.

  The camera 22 captures images continuously in time and transmits an image signal of the captured image to the image processing unit 23 at regular time intervals.

  The image processing unit 23 receives an image signal from the camera 22 and sequentially acquires captured images of the current frame (S50). The image processing unit 23 generates position and orientation estimated values est (i) (i = 1,..., N) that can be taken in the current frame based on the position and orientation estimated in the previous frame (S51). Then, the image processing unit 23 moves the reference three-dimensional position of each feature point held by the position and orientation of each estimated value est (i), and the camera 22 captures an image from each moved three-dimensional position. Projection positions (two-dimensional coordinates) projected on the captured image are calculated (S52). Further, the image processing unit 23 calculates a normalized correlation value between the luminance pattern at the projection position on the captured image and the held reference luminance pattern for each feature point in the case of the estimated value est (i) ( S53). Then, the image processing unit 23 calculates the fitness of the estimated value est (i) from the normalized correlation value of each feature point (S54).

  After calculating the fitness for all the estimated values est (i), the image processing unit 23 extracts the maximum fitness from the fitness and determines whether or not the maximum fitness exceeds a threshold (that is, Whether or not a face exists in the captured image is determined (S55). If it is determined in S55 that the maximum fitness is equal to or less than the threshold value, the image processing unit 23 determines that no face exists on the captured image of the current frame, and outputs the determination result. On the other hand, if it is determined in S55 that the maximum matching level exceeds the threshold value, the image processing unit 23 determines that a face exists on the captured image of the current frame, and outputs the determination result.

  According to this image processing device 21, a large number of estimated values are generated, and the degree of suitability obtained by evaluating the certainty as each feature point and the overall position of the feature point for each estimated value is obtained, thereby obtaining from the captured image. The presence or absence of a face can be determined with high accuracy. Further, according to the image processing device 21, only the reference luminance pattern of each feature point and data with a small standard three-dimensional position are held, and only the processing for each feature point is performed instead of the entire image. Can be reduced and processing time is short. Furthermore, the image processing device 21 has the same effects as the image processing device 11 of the first embodiment with respect to generation of estimated values, evaluation based on fitness, and utilization of maps.

  A third embodiment will be described with reference to FIGS. FIG. 22 is a configuration diagram of an image processing apparatus for eyeball center position estimation processing according to the third embodiment. FIG. 23 is a configuration diagram of an image processing apparatus for eyeball posture estimation processing according to the third embodiment. FIG. 24 is a diagram showing an eyeball structure. FIG. 25 is a diagram illustrating an eyeball model. FIG. 26 is a diagram illustrating the relationship between the eyeball and the camera. FIG. 27 is an example of a captured image of the eye captured by the camera. FIG. 28 is an explanatory diagram of the projection of the points in the black eye on the two-dimensional image according to the estimated value of the rotation of the eyeball. FIG. 29 is a diagram illustrating the relationship between the camera coordinate system and the eyeball coordinate system.

  In the third embodiment, an image processing device 31 and an image processing device 41 are configured. The image processing device 31 is an image processing device for eyeball center position estimation processing. The image processing device 41 is an image processing device for eyeball posture estimation processing.

  A configuration of the image processing apparatus 31 will be described. In the image processing device 31, before the processing by the image processing device 41, as the information held by the image processing device 41, the center position of the eyeball and the three-dimensional position of each point in the black eye in the case of the eyeball center position are obtained. At that time, the image processing device 31 sets a large number of estimated values of the eyeball center position, calculates the degree of matching of each estimated value with the eyeball center position in the captured image, and based on the degree of fitness, the eyeball Estimate the center position. For this purpose, the image processing apparatus 31 includes a camera 32 and an image processing unit 33. The image processing unit 33 includes an information holding unit 33a, an estimated value generation unit 33b, a fitness calculation unit 33c, and an eyeball center position output unit 33d by executing an application program for eyeball center position estimation processing on a computer. .

  In the image processing apparatus 31 according to the third embodiment, the camera 32 corresponds to the imaging unit described in the claims, the information holding unit 33a corresponds to the information holding unit described in the claims, The estimated value generator 33b corresponds to the estimated value generator described in the claims, the fitness calculator 33c corresponds to the fitness calculator described in the claims, and the eyeball center position output unit 33d is patented. This corresponds to the determination means described in the claims.

  The camera 32 is the same camera as the first camera 2 according to the first embodiment, and a description thereof is omitted. In the imaging here, the person to be recognized is directed to look into the lens center of the camera 32. Therefore, the camera 32 captures an image of the eye looking into the person's lens and transmits the image signal to the image processing unit 33.

  The information holding unit 33a is configured in a predetermined memory area and holds an eyeball model. A method for constructing the eyeball model will be described. First, an eyeball structure as shown in FIG. 24 is assumed. In this eyeball structure, the left-right axis of the eyeball is taken as the X axis, the up-down direction axis of the eyeball is taken as the Y-axis, and the depth direction axis of the eyeball is taken as the Z-axis. P is the center of the eyeball, R is the eyeball radius, and r is the iris radius. The eyeball radius R and iris radius r are fixed using standard values. A solid angle formed by a line segment connecting the eyeball center P and the outline of the iris (black-eye region) and a line segment connecting the eyeball center P and the iris center is denoted by θ. A rotation angle around the X axis that approximates the solid angle θ is deg_x, and a rotation angle around the Y axis is deg_y. The relationship between the rotation angle deg_x, the rotation angle deg_y, and the solid angle θ is expressed by Expression (17). Further, the solid angle θ is expressed by Expression (18), for example, when a standard eyeball radius R and iris radius r are used.

  Here, the eyeball structure shown in FIG. 24 is approximated to a simple eyeball model including the eyeball center P, the eyeball radius R, and the iris radius r as shown in FIG. 25 to construct an eyeball model. The eyeball model is held in the information holding unit 33a.

  In the estimated value generation unit 33b, the eyeball model moves the eyeball center position in front of the camera 32 while rotating the eyeball so that the line connecting the optical center O of the camera 32 and the eyeball center P matches the line of sight of the eyeball model. A large number of estimated values of the center position of the eyeball that can be taken when changed are generated (see FIG. 26). Here, for example, using a conventional technique for detecting the position of a face, if the face can be detected from the captured image and the face can be detected with that size, the position of the face is It can be predicted that there is. When such face position prediction values are available, use the relationship between the face position on the captured image and the average face position, and the relative position relationship of the eyeball center position with respect to the face position. Thus, since the approximate center position of the eyeball can be estimated, an estimated value is generated near the position.

  First, in the case of the eyeball center position of each estimated value, the fitness level calculating unit 33c calculates a two-dimensional position taken by each point in the black eye on the captured image. When the eyeball center position P and the line-of-sight direction (vector Q from the eyeball center to the center of the black eye) are determined, the fitness level calculation unit 33c determines the number of black eyes in the eyeball model from the above equations (17) and (18). The three-dimensional position of the point is calculated by equation (19).

In Equation (19), P is an estimated value of the center position of the eyeball, Q is a vector (three-dimensional coordinate) from the center of the eyeball to the center of the black eye, and Q ′ is the three-dimensional coordinate of the point in the black eye. Here, in order to simplify the subsequent processing, it is assumed that P and Q in the camera coordinate system are obtained, and Q is obtained from the eyeball model. R _y is a rotation matrix around the Y axis, and is represented by Expression (20). R _x is a rotation matrix around the X axis, and is represented by Expression (21). In this case, for each point in the black eye, the rotation angle deg_x around the X axis, the rotation angle deg_y around the Y axis, and the solid angle θ satisfy the relationship of Expression (22).

  Furthermore, the fitness level calculation unit 33c calculates the projection position on the captured image of the three-dimensional coordinates Q 'of each point in the black eye according to Expression (23). A region formed by the multiple projection positions is a region on a captured image of the iris (black eye) on the eyeball model when the eyeball model is moved at the estimated eyeball center position.

  s is a scalar, A is an internal matrix of the camera 32, (u, v) is a projection position (two-dimensional coordinates) on a captured image of a point in the black eye, and (X ′, Y ′, Z ′) ) Is the three-dimensional position (three-dimensional coordinates) Q ′ of the point in the black eye. Here, a large number of points in the black eye are calculated by setting the values of deg_x and deg_y in various combinations. The step size of deg_x and deg_y is set to, for example, about several deg so that the point group in the black eye does not become full of gaps when projected onto the captured image.

  The suitability calculation unit 33c acquires the brightness value of the projection position of each point in the black eye from the captured image of the camera 32 for each estimated value. Then, the fitness level calculation unit 33c calculates an average value of luminance values at all projection positions for each estimated value, and uses this average value as the fitness level. Note that, also in the fitness calculation unit 33c, a map is set and the processing load is reduced as in the first embodiment.

  Thus, by giving the eyeball center position of each estimated value, the eyeball model is rotated so that the line of sight coincides with the line segment connecting the eyeball center position and the optical center, and the disc corresponding to the iris on the eyeball model (see FIG. 25) is determined on the captured image, and the point inside the disk corresponds to the point in the black eye. That is, the average value of the calculated luminance values is an index of blackness. By capturing this index as the degree of fitness, the black eye is particularly black in the vicinity of the eye, so the average value is predicted to be small. Therefore, it can be considered that the index having the minimum value is most likely.

  Therefore, the eyeball center position output unit 33d extracts the fitness value of the minimum value from the calculated fitness values when calculation of the fitness values for all estimated values is completed. Then, the eyeball center position output unit 33d estimates the estimated value having the minimum fitness as the eyeball center position, and outputs the estimated eyeball center position. The eyeball center position output unit 33d also outputs the three-dimensional coordinates of each point in the black eye in the case of the estimated eyeball center position.

  A configuration of the image processing apparatus 41 will be described. The image processing apparatus 41 holds a template for the eyeball model, and estimates the posture (eye-gaze direction) of the eyeball on the captured image using the template. Here, since it is assumed that the face does not move and the eyeball center position P obtained by the image processing device 31 is fixed, the change that occurs in the eyeball model is a rotational motion with respect to the camera coordinate system. Estimated as posture. At that time, the image processing apparatus 41 sets a large number of estimated values of the eyeball posture, calculates the degree to which each estimated value matches the eyeball posture in the captured image, and determines the eyeball posture based on the degree of fitness. presume. For this purpose, the image processing apparatus 41 includes a camera 42 and an image processing unit 43. The image processing unit 43 executes an application program for eyeball posture estimation processing on a computer, whereby an information holding unit 43a, a storage analysis unit 43b, an estimated value generation unit 43c, a fitness calculation unit 43d, and an eyeball posture output unit 43e Composed.

  In the image processing apparatus 41 according to the third embodiment, the camera 42 corresponds to the imaging unit described in the claims, the information holding unit 43a corresponds to the information holding unit described in the claims, The estimated value generator 43c corresponds to the estimated value generator described in the claims, the fitness calculator 43d corresponds to the fitness calculator described in the claims, and the eyeball posture output unit 43e claims. It corresponds to the determination means described in the range.

  The camera 42 is the same camera as the camera 12 according to the first embodiment, and a description thereof is omitted. Note that when the image processing device 41 is mounted in an automobile or the like, the camera 42 is arranged in a position in the passenger compartment where the vicinity of the eyes of the driver sitting in the driver's seat can be imaged from the front. FIG. 27 shows an example of a captured image of a certain frame captured by the camera 42.

  The information holding unit 43a is configured in a predetermined memory area, the same eyeball model held by the image processing device 31, and the eyeball center position P output from the image processing device 31 and the eyeball model in the case of the eyeball center position P. Holds the 3D position of each point in the black eye in the camera coordinate system.

  The storage analysis unit 43b is configured in a predetermined memory area and stores the estimated value of the posture of the eyeball generated by the estimated value generation unit 43c and the fitness for each estimated value calculated by the fitness calculation unit 43d in association with each other. At the same time, the posture of the eyeball in the captured image output from the eyeball posture output unit 43e is stored. In the memory analysis unit 43b, the posture of the eyeball taken by the person to be recognized in the past is accumulated as a history. Further, the memory analysis unit 43b identifies the eyeball postures that can be taken from the past history. In addition, the memory analysis unit 43b sets a normal distribution having a peak in the vicinity of the posture of the eyeball that can be frequently obtained from the past history, and generates a probability density function by superimposing a plurality of normal distributions. Note that the memory analysis unit 43b may accumulate the history using the eyeball posture output from the eyeball posture output unit 43e, or the fitness level may be selected from a large number of estimated values and the fitness levels of the estimated values. The history may be accumulated using an estimated value that is greater than the threshold.

  Based on the eyeball model held in the information holding unit 43a, the estimated value generation unit 43c estimates the eyeball posture (rotation angles deg_x, deg_y) in the captured image of the current frame that can be obtained from the eyeball posture in the previous frame. Generate many values. The range of postures that can be taken is a range obtained by adding and subtracting each maximum value that can be changed within the time between frames to the two parameters of the posture of the previous frame for each of the two parameters of the posture. However, when this range includes a range that the eyeball to be recognized cannot be structurally taken, it is a range excluding the range that cannot be structurally taken. In other words, since the range of the posture of the rotatable eyeball is physically determined, the estimated value is not generated beyond the range. As the maximum value that can be changed within the time between frames, the posture of the eyeball in the environment where the eyeball is placed may be measured in advance, and may be set in advance from the maximum value of the change obtained from the measurement, or You may set based on the change of the attitude | position between a front frame and a frame before the last.

  In the range of the two parameters of the posture that can be set as described above, it is equally likely that each parameter actually takes a value. Therefore, the estimated value generation unit 43c randomly extracts n times from the uniform distribution in the total two-dimensional plane that takes a value in the range of the minimum value and the maximum value for each of the two parameters of the posture, and calculates the extracted value. The estimated value of the posture.

  As a method for generating an estimated value, an estimated value may be generated based on a combination of a position and a posture that can be obtained from the history accumulated in the storage analysis unit 43b, or the storage analysis unit 43b. The estimated value may be generated based on the probability density function derived from the history stored in the history, or the estimated value may not have a uniform distribution and the current frame posture may be near the previous frame posture If the characteristics are high, the estimated value may be generated with a normal distribution having the peak of the posture of the previous frame.

  For each estimated value generated by the estimated value generating unit 43c, the fitness calculating unit 43d calculates a two-dimensional position taken by each point in the black eye on the captured image using two parameters of the posture of each estimated value. . Here, the two-dimensional position is calculated using the fixed eyeball center position P held in the information holding unit 43a and the estimated rotation angles deg_x and deg_y. As shown in FIG. 29, a virtual eyeball coordinate system EC set for the eyeball is considered, and the eyeball orientation direction ED and the optical axis direction CD of the camera 42 are opposite to each other and the eyeball coordinate system EC is The X axis and the X axis of the camera coordinate system CC are in opposite directions, and this state is defined as a reference posture. The rotation angles deg_x and deg_y are the rotation angles of the eyeball from this reference posture.

  Here, the eyeball model held by the information holding unit 43a and the three-dimensional position in the camera coordinate system of each point in the black eye of the eyeball model are used. At this time, the center of the eyeball model is set to the eyeball center position P (fixed) obtained by the image processing device 31, and the eyeball center position P is the origin of the eyeball coordinate system EC (see FIG. 29). In this state, the eyeball model is rotated using the estimated rotation angles deg_x and deg_y. In the fitness calculation unit 43d, the three-dimensional position in the camera coordinate system of each point in the black eye before rotation is defined as Q, and the three-dimensional position Q ′ of each point in the black eye after rotation is calculated by Expression (24). To do.

R _y is a rotation matrix around the Y axis and is expressed by Expression (25), and the rotation angle deg_y of the estimated value is used. R _x is a rotation matrix around the X axis, and is represented by Expression (26), and an estimated rotation angle deg_x is used. In this case, for each point in the black eye, the rotation angle deg_x, the rotation angle deg_y, and the solid angle θ satisfy the relationship of Expression (27).

  Then, similarly to the image processing device 31, the suitability calculation unit 43 d calculates the projection position on the captured image of the three-dimensional coordinates Q ′ of each point in the black eye by the above equation (23). The region formed by the large number of projection positions is a region on a captured image of the iris (black eye) on the eyeball model when the eyeball model is rotated at the estimated rotation angles deg_x and deg_y. FIG. 28 shows two-dimensional coordinate IDs projected on the image according to the estimated rotation angles deg_x, deg_y.

  Further, in the fitness level calculation unit 43d, as in the image processing device 31, the brightness value of the projection position of each point in the black eye is acquired from the captured image of the camera 42 for each estimated value, and at all projection positions. The average value of luminance values is taken as the fitness. In the fitness level calculation unit 43d, as in the first embodiment, a map is set and the processing load is reduced.

  When the eyeball posture output unit 43e finishes calculating the fitness level for all the estimated values, the fitness level of the minimum value is extracted from the calculated fitness levels. Then, the eyeball posture output unit 43e estimates the estimated value having the minimum fitness as the posture (rotation angle deg_x, deg_y), and outputs the estimated eyeball posture.

  As a method of estimating the fitness, other methods may be used, for example, the posture may be estimated by a weighted average value based on the fitness of an estimated value having a fitness equal to or less than a threshold, or all the fitness values calculated may be calculated. The posture may be estimated by a weighted average value based on the degree of fitness of the estimated value, or the degree of fitness that takes a value not more than a fixed multiple of the minimum value of the fitness (for example, a value not more than 1.1 times the maximum value). The posture may be estimated by a weighted average value based on the degree of fitness of the values, or when the number of fitness levels of the estimated fitness value that takes a value that is a certain multiple of the minimum value of the fitness level is greater than or equal to a predetermined number The posture may be estimated by a weighted average value based on the fitness value of the fitness value, or the number of fitness values of the fitness value that takes a value not more than a certain multiple of the minimum value of the fitness value is a predetermined number. If it is less than this, the posture of the estimated value with the minimum fitness may be used.

  Next, the operation of the image processing apparatus 31 will be described with reference to FIG. In particular, the eyeball center position estimation process of the image processing unit 33 will be described with reference to the flowchart of FIG. FIG. 31 is a flowchart showing a flow of eyeball center position estimation processing in the image processing apparatus of FIG.

  The camera 32 captures an image of an eye looking into the camera and transmits an image signal of the captured image to the image processing unit 33.

  The image processing unit 33 sets an eyeball model including an eyeball center position P, an eyeball radius R, and an iris radius r (S60). The image processing unit 33 receives an image signal from the camera 32 and acquires a captured image of the eye (S61).

  In the image processing unit 33, when the line segment connecting the eyeball center of the eyeball model and the optical center of the camera 32 rotates so as to coincide with the line of sight of the eyeball model, an estimated value of the position that the eyeball center can take with respect to the camera 32 Are generated (S62). Then, in the case of the eyeball center position of each estimated value, the image processing unit 33 moves each point in the black eye according to each rotation angle deg_x, deg_y, and the camera 32 from each moved three-dimensional position. Projection positions (two-dimensional coordinates) projected on the captured image are calculated (S63). Further, in the case of the eyeball center position of each estimated value, the image processing unit 33 calculates an average value of luminance at the projection position of each point in the black eye on the captured image, and uses the average value as the fitness ( S64).

  When the degree of fitness for the eyeball center position of all the estimated values is calculated, the image processing unit 33 calculates the eyeball center position in the captured image using the eyeball center position of the estimated value and the degree of fitness calculated for each estimated value. The estimated eyeball center position is output (S65). Further, the image processing unit 33 outputs the three-dimensional position in the camera coordinate system of each point in the black eye of the eyeball model in the case of the estimated eyeball center position.

  Next, the operation of the image processing apparatus 41 will be described with reference to FIG. In particular, the eyeball posture estimation process of the image processing unit 43 will be described with reference to the flowchart of FIG. FIG. 32 is a flowchart showing the flow of eyeball posture estimation processing in the image processing apparatus of FIG.

  The camera 42 captures images continuously in time and transmits an image signal of the captured image to the image processing unit 43 at regular time intervals.

  The image processing unit 43 sets an eyeball model including the eyeball center position P, the eyeball radius R, and the iris radius r (S70). In this eyeball model, the eyeball center position P is fixed at the eyeball center position estimated by the image processing device 31, and the three-dimensional position serving as a reference for each point in the black eye in the case of the eyeball center position P is the image processing device 31. This is a calculated value. The image processing unit 43 receives an image signal from the camera 42 and acquires a captured image of the current frame (S71).

  The image processing unit 43 generates a large number of estimated rotation values that the eyeball can take with respect to the camera coordinate system based on the eyeball posture estimated in the previous frame (S72). Then, the image processing unit 43 moves the three-dimensional position of each point in the black eye held according to the rotation angles deg_x and deg_y of the respective estimated values, and the camera 42 from each of the moved three-dimensional positions. Projection positions (two-dimensional coordinates) projected on the captured image captured in (1) are calculated (S73). Further, in the case of the rotation angles deg_x and deg_y of each estimated value, the image processing unit 43 calculates the average value of the luminance at the projection position of each point in the black eye on the captured image, and uses the average value as the fitness. (S74).

  When the degrees of fitness for the rotation angles deg_x and deg_y of all the estimated values are calculated, the image processing unit 43 uses the rotation angles of the estimated values and the fitness calculated for each estimated value to calculate the eyeball posture in the captured image. The estimated eyeball posture is output (S75).

  According to the image processing device 31 and the image processing device 41, a large number of estimated values are generated, and an eyeball is obtained from the captured image by obtaining a degree of fitness that evaluates the probability of the black eye and the position of the black eye for each estimated value. The center position and eyeball posture can be estimated with high accuracy. Further, according to the image processing device 31 and the image processing device 41, only the data with a small eyeball model is held, and only the processing for the eyeball model is performed instead of the entire image, so that the processing load can be reduced and the processing time can be reduced. short.

  In particular, the black eye on the image is substantially a circle when the line of sight is facing the camera, but becomes an ellipse as the deviation of the line of sight from the optical center increases. Therefore, the image processing device 31 and the image processing device 41 use the eyeball model, move the position and posture of the eyeball model in a three-dimensional manner, and use the projection position on the image. The change of the black eye can be expressed with high accuracy. Therefore, it is possible to search for the position and orientation of the eyeball model that looks the same as the black eye on the image, and to estimate the position and orientation. As a result, the posture (line of sight) of a black eye that looks like an ellipse on the captured image can be estimated with high accuracy.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in the present embodiment, the target object is applied to a human face or eyeball, but it can also be applied to other three-dimensional objects, for example, a similar three-dimensional shape such as a human body or a vehicle. It can be applied to an object whose three-dimensional shape can be specified.

  In the present embodiment, each unit is configured by executing an application program (software) on a computer such as a personal computer. However, each unit may be configured by hardware.

  In this embodiment, the image processing apparatus for determining the presence / absence, position, and orientation of the target object is provided with one camera, and the presence / absence, position, and orientation are determined from a single captured image. The image processing apparatus may include a plurality of cameras (for example, stereo cameras), and may determine presence / absence, position, and posture from a plurality of captured images. In this case, the processing load increases, but the presence / absence determination and position / posture estimation accuracy are improved. For example, when a plurality of cameras are provided in the first embodiment, for each of a plurality of captured images, calculation of a projection position, generation and retention of a map for each feature point, normalized correlation calculation, fitness calculation, etc. There is a need to do.

  In this embodiment, luminance information is used as information for evaluating feature points. However, other image information such as saturation and RGB color information may be used. In this embodiment, the luminance pattern is used as the information for evaluating the feature points. However, the luminance histogram, the luminance distribution shape, the edge pattern (edge image), the frequency characteristics by Fourier transform, and the like are used. Information may be used.

  In the present embodiment, the map is generated for each feature point, but the map may not be generated.

  In this embodiment, a storage analysis unit is provided to store the estimated value and the fitness of the estimated value, create a history of the position and orientation of the target object taken in the past, and also use the history to calculate the estimated value. However, the estimated value may be generated without using the history without using the storage analysis unit.

  In the first embodiment and the second embodiment, the estimated values for both the position and orientation of the face are generated. However, either one of the position and orientation is fixed, and only the other is estimated. It is good also as a structure which produces | generates a value. Some target objects change only one of the position and orientation, and in that case, an estimated value is generated only for the changed object.

  In the first embodiment, the configuration is such that the position and orientation are estimated by generating estimated values and calculating the fitness for each estimated value in two stages. However, the position and orientation are estimated in one stage. You may do it.

  In the third embodiment, the eyeball center position is estimated based on the eyeball model. However, when the relative positional relationship of the average eyeball center position with respect to the face position is known, the first embodiment is performed. It is also possible to estimate the center position of the eyeball from the relative positional relationship based on the position and posture of the face estimated in this form.

  In the third embodiment, the posture of the eyeball is estimated when it is assumed that the face does not move. However, the posture of the eyeball when the face (head) moves can also be estimated. For example, first, the face position and posture are estimated in the first embodiment, and the eyeball posture is estimated in consideration of the estimated face position and posture.

  In the third embodiment, each estimation is performed using an eyeball model that considers only black eyes, but each estimation may be performed using an eyeball model that considers white eyes in addition to black eyes. In addition to black eyes, the eyes above the captured image may be black in the vicinity of the corners of the eyes and the upper eyelids as well. Therefore, an eyeball model that can take into account the positional relationship between black eyes and surrounding white eyes as shown in FIG. 30 can also be used. When this eyeball model is used, in addition to the average value of the luminance at the projection position on the image of each point in the black eye, the average value of the luminance at the projection position on the image of each point in the white eye can be obtained. The value obtained by subtracting the average luminance value of the black eye region from the average luminance value of the white eye region is used as the fitness level, and it can be determined that the position where the fitness level is the highest is most likely black.

  Further, in the third embodiment, when evaluating the probability as a black eye, the black eye region has the lowest luminance as compared with other regions, and therefore, the configuration is such that the smallest fitness is extracted from the fitness. However, the threshold value for determining the average value of the luminance indicating the black eye area may be held as the evaluation information, and the degree of fit below the threshold value may be extracted, or the iris may be in other colors such as blue or green In this case, a threshold value for determining the average value of the brightness of the color area may be held as evaluation information, and the degree of fit within the threshold value may be extracted. A configuration may be adopted in which a reference luminance pattern indicating an iris pattern is held, a normalized correlation value is obtained from the reference luminance pattern and the luminance pattern at the projection position, and the fitness is calculated from the normalized correlation value.

It is a block diagram of the image processing apparatus for modeling processing which concerns on 1st Embodiment and 2nd Embodiment. 1 is a configuration diagram of an image processing apparatus for face position / posture estimation processing according to a first embodiment. FIG. It is an example of the captured image of the face imaged with the 1st camera of FIG. It is an image which shows the feature point extracted from the captured image of FIG. It is an example of a three-dimensional model including points corresponding to feature points. It is a figure which shows the three-dimensional position of each feature point of FIG. It is an example of the captured image of the face each imaged with the 1st camera and 2nd camera of FIG. It is an image which shows the feature point each extracted from the two captured images of FIG. It is an example of the captured image of the face imaged with the camera of FIG. 10 is an image showing a face area detected from the captured image of FIG. 9. It is an image which shows the area | region similar to the reference luminance pattern extracted from the face area | region of the captured image of FIG. It is explanatory drawing of the projection on the two-dimensional image from the reference | standard three-dimensional position of the feature point according to the estimated value of the face position and attitude | position in the 1st step. It is an example of the normalized correlation value of each feature point in the first stage. It is explanatory drawing of the projection on the two-dimensional image from the reference | standard three-dimensional position of a feature point according to the estimated value of the face position and attitude | position in a 2nd step. It is an example of the normalized correlation value of each feature point in the second stage. It is an example of the map which consists of the normalized correlation value in each projection position for every feature point. It is a flowchart which shows the flow of the modeling process in the image processing apparatus of FIG. 3 is a flowchart showing a flow of face position / posture estimation processing in the image processing apparatus of FIG. 2. 3 is a flowchart showing a flow of normalized correlation value calculation processing in the image processing apparatus of FIG. 2. It is a block diagram of the image processing apparatus for the face presence determination process which concerns on 2nd Embodiment. FIG. 21 is a flowchart showing a flow of face presence / absence determination processing in the image processing apparatus of FIG. 20. FIG. It is a block diagram of the image processing apparatus for the eyeball center position estimation process which concerns on 3rd Embodiment. It is a block diagram of the image processing apparatus for the eyeball attitude | position estimation process which concerns on 3rd Embodiment. It is a figure which shows an eyeball structure. It is a figure which shows an eyeball model. It is a figure which shows the relationship between an eyeball and a camera. It is an example of the captured image of the eye imaged with the camera of FIG. It is explanatory drawing of the projection on the two-dimensional image of the point in a black eye according to the estimated value of rotation of an eyeball. It is a figure which shows the relationship between a camera coordinate system and an eyeball coordinate system. It is a figure which shows the eyeball model also including a white eye. It is a flowchart which shows the flow of the eyeball center position estimation process in the image processing apparatus of FIG. It is a flowchart which shows the flow of the eyeball attitude | position estimation process in the image processing apparatus of FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1, 11, 21, 31, 41 ... Image processing apparatus, 2 ... 1st camera, 3 ... 2nd camera, 12, 22, 32, 42 ... Camera, 4, 13, 23, 33, 43 ... Image processing part, 4a: Feature point extraction unit, 4b: Feature point three-dimensional position estimation unit, 4c, 13a, 23a, 33a, 43a ... Information holding unit, 13b, 23b, 43b ... Memory analysis unit, 13c, 23c, 33b, 43c ... Estimation Value generation unit, 13d, 23d, 33c, 43d ... fitness calculation unit, 13e ... face position / posture output unit, 23e ... face presence / absence output unit, 33d ... eyeball center position output unit, 43e ... eyeball posture output unit

Claims (26)

Imaging means;
Information holding means for holding a template consisting of evaluation information about a plurality of feature points on a two-dimensional image of a target object and three-dimensional position information;
Estimated value generation means for generating a plurality of estimated values of the position or / and orientation of the target object;
A fitness calculation for calculating a fitness of a plurality of estimated values generated by the estimated value generating device based on each template of a captured image captured by the imaging device and a plurality of feature points held by the information holding device. Means,
And determining means for determining the presence / absence, position or / and orientation of the target object in the captured image captured by the imaging means based on the fitness of the plurality of estimated values calculated by the fitness calculating means. Image processing device.   The estimated value generation means includes a value related to a position or / and orientation of the target object in a captured image captured in the past by the imaging means, a value related to a position or / and orientation that the target object can take structurally, a position of the target object or / The image processing apparatus according to claim 1, wherein the estimated value is generated based on at least one of the history of the posture and the posture.   The estimation value generation means generates a plurality of estimations based on at least one of a value related to a position or / and orientation of the target object in a captured image captured in the past by the imaging means, and a history of the position or / and orientation of the target object. The image processing apparatus according to claim 1, wherein the density of values is changed.   The fitness calculation means converts a three-dimensional position of each feature point of the template by each estimated value generated by the estimated value generation means, and projects the converted three-dimensional position on a captured image captured by the imaging means. The probability of the feature point at the projection position is evaluated based on the evaluation information of the template and the information around the projection position of the captured image, and the fitness is calculated based on the evaluation value. The image processing apparatus according to any one of claims 1 to 3.   The image processing apparatus according to claim 4, wherein the fitness calculation unit converts information around a projection position of the captured image into the same physical quantity as the evaluation information of the template.   The image processing apparatus according to claim 4, wherein the fitness calculation unit sets a constant value when an evaluation value of the probability of the feature point at the projection position is equal to or less than a predetermined value.   7. The method according to claim 4, wherein the matching level calculation unit generates a data structure including an evaluation value at each projection position for each feature point each time the target object in the captured image changes. The image processing apparatus described in the item.   The image processing apparatus according to claim 4, wherein the fitness level calculation unit calculates the fitness level using a feature point having a high evaluation value among a plurality of feature points. .   The image according to any one of claims 4 to 8, wherein the fitness calculation means calculates a statistic of evaluation values of a plurality of feature points, and uses the statistic as a fitness. Processing equipment.   The determination means has at least a predetermined value greater than or equal to a predetermined value, the number of fitness degrees greater than or equal to a predetermined value is greater than or equal to a predetermined number, and the number of fitness values within a predetermined range from the maximum fitness value The image processing apparatus according to claim 1, wherein the image processing apparatus determines that the target object exists in the captured image when one condition is satisfied.   The image processing apparatus according to claim 1, wherein an estimated value of the position and / or orientation of the target object and a degree of fitness for the estimated value are stored. The target object is black eyes or black eyes and white eyes,
The information holding means holds a black eye model as a template when the target object has black eyes, and holds a black eye model and a white eye model as templates when the target object has black eyes and white eyes. The image processing device according to claim 11. The target object is black eyes or black eyes and white eyes,
The degree-of-fit calculation means calculates the average value of the luminance of the black-eye area at the projected position of the captured image when the target object is black eyes as the degree of match of the estimated value, and images when the target object is black-eye and white-eye The image processing apparatus according to claim 1, wherein an average value of luminance of a black eye region and an average value of luminance of a white eye region at an image projection position are calculated. Imaging step;
An estimated value generating step for generating a plurality of estimated values of the position and / or orientation of the target object;
Adaptation of a plurality of estimated values generated in the estimated value generating step based on a template consisting of evaluation information and three-dimensional position information about a plurality of feature points on a two-dimensional image of the captured image captured in the imaging step and the target object A fitness calculation step for calculating a degree,
An image processing method comprising: a determination step of determining presence / absence, position, and / or orientation of the target object based on the fitness of the plurality of estimated values calculated in the fitness calculation step.   In the estimated value generation step, a value related to the position or / and orientation of the target object in the captured image captured in the past in the imaging step, a value related to a position or / and orientation that the target object can take structurally, a position of the target object or / The image processing method according to claim 14, wherein an estimated value is generated based on at least one of the history of the posture and the posture.   In the estimation value generation step, a plurality of estimations are generated based on at least one of a value related to the position or / and orientation of the target object in the captured image captured in the past in the imaging step, and a history of the position or / and orientation of the target object. 16. The image processing method according to claim 14, wherein the density of values is changed.   In the fitness calculation step, the three-dimensional position of each feature point of the template is converted by each estimated value generated in the estimated value generating step, and the converted three-dimensional position is projected on the captured image captured by the imaging unit. The probability of the feature point at the projection position is evaluated based on the evaluation information of the template and the information around the projection position of the captured image, and the fitness is calculated based on the evaluation value. The image processing method according to any one of claims 14 to 16.   The image processing method according to claim 17, wherein in the matching level calculation step, information around a projection position of the captured image is converted into the same physical quantity as the evaluation information of the template.   The image processing method according to claim 17 or 18, wherein, in the fitness calculation step, the evaluation value of the probability of the feature point at the projection position is set to a constant value when the evaluation value is equal to or less than a predetermined value.   20. The data structure including evaluation values at each projection position for each feature point is generated each time the target object in the captured image is changed in the matching degree calculation step. The image processing method described in the item.   The image processing method according to any one of claims 17 to 20, wherein in the fitness level calculating step, the fitness level is calculated using a feature point having a high evaluation value among a plurality of feature points. .   The image according to any one of claims 17 to 21, wherein, in the fitness calculation step, a statistic of evaluation values of a plurality of feature points is calculated, and the statistic is used as the fitness. Processing method.   In the determining step, the maximum value of the fitness level is a predetermined value or more, the number of fitness levels greater than or equal to the predetermined value is a predetermined number or more, and the number of fitness levels within a predetermined range from the maximum value of the fitness level is at least a predetermined number or more. The image processing method according to any one of claims 14 to 22, wherein a target object is determined to exist in the captured image when one condition is satisfied.   The image processing method according to any one of claims 14 to 23, wherein an estimated value of the position and / or orientation of the target object and a degree of fitness for the estimated value are stored. The target object is black eyes or black eyes and white eyes,
25. The template according to any one of claims 14 to 24, wherein the template is a black eye model when the target object is black eyes, and a black eye model and a white eye model when the target object is black eyes and white eyes. An image processing method described in 1. The target object is black eyes or black eyes and white eyes,
In the fitness calculation step, as the fitness of the estimated value, the average value of the luminance of the black eye region at the projected position of the captured image is calculated when the target object is black eye, and the image is captured when the target object is black eye and white eye The image processing method according to any one of claims 14 to 25, wherein an average value of luminance of a black eye region and an average value of luminance of a white eye region at an image projection position are calculated.

Images