JP2007272578A - Image processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and method to increase the accuracy of tracking a feature point and the accuracy of estimating a three-dimensional position. <P>SOLUTION: Based on a feature point stored in a feature point image storing means 2 and an image of periphery and on a depth-candidate pair showing the positional relationship of feature points, created by a depth candidate pair creating means 3, a tracking means 4 tracks the position of the feature point in an image acquired by an image acquisition means 1. The tracking may be performed by e.g. searching for an image similar to an image of an area around the feature point (preferably an image converted according to the position and the direction of the feature point). The result of the tracking is evaluated by a tracking condition evaluating means 5. The feature point can be tracked more accurately as the accuracy of the depths of the depth-candidate pair created by the depth candidate pair creating means 3 is higher, and high evaluations can be obtained from the tracking condition evaluating means 5. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for estimating a three-dimensional depth of a point on an object based on an image sequence of an object acquired by a monocular camera or the like.

In addition to the technique using binocular vision by a stereo camera as a technique for acquiring the three-dimensional position of the object, a technique for obtaining the three-dimensional coordinates of the object from an image acquired by a monocular camera has been proposed (non-patent document). 1). The technique described in this document estimates the two-dimensional coordinates of a plurality of feature points on an object shown in an image sequence acquired by a monocular camera, and estimates the three-dimensional coordinates of the feature points based on the change. It is something to try.
J. Costeira and T. Kanade, "A Multi-bodyFactorization Method for Motion Analysis" technical report CMU-CS-TR-94-220, Carnegie Mellon University, USA, September 1994

  In this technique, in order to accurately estimate the three-dimensional coordinates of the feature points, it is necessary to accurately track the feature points on the object in the image sequence. However, in an actual environment, it is difficult to track feature points with high accuracy, so that a decrease in tracking accuracy causes a decrease in estimation accuracy of three-dimensional coordinates.

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method capable of improving the tracking accuracy of feature points and increasing the estimation accuracy of a three-dimensional position.

  In order to solve the above problems, an image processing apparatus according to the present invention includes (1) means for acquiring a plurality of images including three or more feature points on an object whose depth is to be estimated, and (2) a predetermined viewpoint. (3) a depth candidate pair generating unit that generates a depth candidate pair that is a set of depth candidate values for the feature points; and (4) a feature point. A tracking unit that tracks the position of the feature point in the acquired image using the image held by the image holding unit and the generated depth candidate pair; and (5) a tracking state evaluation unit that evaluates a tracking state by the tracking unit; It is characterized by having.

  On the other hand, the image processing method according to the present invention (1) obtains a plurality of images including three or more feature points on an object whose depth is to be estimated, and (2) each feature point from a predetermined viewpoint and its feature point A feature point image that is a peripheral image is stored, (3) a depth candidate pair that is a set of depth candidate values for the feature point is generated, and (4) a stored feature point image and the generated depth candidate pair are used. The method further comprises a step of tracking the position of the feature point in the acquired image and evaluating the tracking state.

  According to the present invention, a depth representing a positional relationship between feature points is estimated from an image sequence including two or more images in which three or more feature points on an object whose depth is to be estimated are captured. At this time, the feature point and a predetermined viewpoint around the feature point, for example, an image viewed from the front, are stored, and the position of the feature point is tracked based on the depth candidate pair that is a set of the depth candidate value and the feature point. Evaluate tracking status. Although the evaluation result of the tracking situation in each image represents the tracking accuracy, the tracking accuracy for the image sequence is improved as a whole as the accuracy of the depth candidate pair is higher. Based on this relationship, a probable depth candidate pair is obtained.

  The tracking unit generates a feature point peripheral image candidate at the time of position / posture change of the object from the image held by the feature point image holding unit and the generated depth candidate pair, and the feature point peripheral image candidate generated from the acquired image It is good to track by searching. In the tracking step, a feature point peripheral image candidate at the time of changing the position / posture of the target object is generated from the retained feature point image and the generated depth candidate pair, and the generated feature point peripheral image candidate is searched from the acquired image. It is good to track.

The image processing apparatus according to the present invention further includes depth candidate pair selection means for selecting an appropriate depth candidate pair based on the evaluation result by the tracking situation evaluation means, and the tracking means is held by the feature point image holding means. Tracking may be performed by generating feature point peripheral image candidates at the time of position / posture change of the object from the image and a plurality of generated depth candidate pairs, and searching for the generated feature point peripheral image candidates from the acquired image. In other words, feature point peripheral image candidates corresponding to the position / posture change of the object are generated, and feature points are tracked by searching for feature points from images obtained by pattern matching with the image candidates. Execute.

  The depth candidate pair generation unit may further include a depth candidate pair selection unit that generates a plurality of sets of depth candidate pairs and selects an appropriate depth candidate pair based on the evaluation result by the tracking state evaluation unit. It is preferable to provide a step of generating a plurality of pairs of depth candidates and performing tracking and selecting an appropriate depth candidate pair based on the evaluation result of the tracking situation. That is, a plurality of depth candidate pairs are generated, and an appropriate depth candidate pair is selected from the plurality of depth candidate pairs based on the evaluation result of the tracking situation for each depth candidate pair.

  In the depth candidate pair generation means or the depth candidate pair generation process, the value of each element of the candidate pair is changed by the tracking state evaluation means for the generated depth candidate pair or based on the evaluation result in the evaluation process. A new depth candidate pair may be generated.

  Based on the evaluation result of the depth candidate pair, the original depth candidate pair is corrected to generate a new depth candidate pair, and re-evaluation is performed. This optimizes the depth candidate pair.

  In the depth candidate pair generation means or the depth candidate pair generation step, a plurality of depth candidate pairs may be generated by extracting the depth of each feature point from a set range under a predetermined condition.

  When multiple depth candidate pairs are set, the depth of each feature point is set within a predetermined range, and at that time, the depth position of each candidate pair based on an equal interval or a preset distribution (for example, a normal distribution) Set.

  Based on the depth candidate pair selected from the depth candidate pair generation means or the depth candidate pair generated in the depth candidate pair generation process, the depth candidate pair is generated with further strict depth range and extraction conditions, and tracked The tracking, evaluation, and selection by the means, the tracking status evaluation means, and the depth candidate pair selection means may be re-executed.

  A depth candidate pair is generated by further narrowing down the setting range based on the depth candidate pair selected from the initial set of depth candidate pairs, and the depth interval when the depth candidate pair is generated is subdivided. At this time, the number of depth candidate pairs to be evaluated is optimized by sequentially narrowing down.

  The image processing apparatus according to the present invention further includes means for acquiring estimated information of the position where the feature point exists, and the depth candidate pair generating means generates a depth candidate pair based on the acquired position estimation information of the feature point. May be. On the other hand, the image processing method according to the present invention may acquire estimation information of the presence position of a feature point and generate a depth candidate pair based on the acquired position estimation information of the feature point.

  As the estimation information of the existence position of the feature point, for example, the positional information of the feature point obtained from the standard positional relation information of the feature point according to the type of the object or the information (focus position) obtained from the image acquisition unit Furthermore, the distance between the image acquisition means and the object may be estimated based on the size of the object in the image and the distance between the feature points in the image. Based on these pieces of information, the positional relationship between feature points is estimated, and an initial depth candidate pair is generated.

  When the evaluation result by the tracking state evaluation means or in the evaluation process is equal to or less than a predetermined value, the tracking for the depth candidate pair may be terminated. If the evaluation result is equal to or less than a predetermined value, that is, a favorable evaluation is not obtained, it is assumed that the accuracy of the depth candidate pair is low, so the processing is stopped without performing the tracking processing for all the image sequences. In this way, the processing speed is increased.

  The tracking status evaluation unit may perform the determination based on the similarity between the feature point image held by the feature point image holding unit and the image near the tracking position. In the evaluation step, the held feature point image and the image near the tracking position The tracking status should be evaluated based on the degree of similarity. That is, the feature point and the tracking position are comprehensively evaluated in a wider range including the peripheral image instead of the narrow point portion.

  The tracking state evaluation means or process may perform evaluation by integrating the tracking states of a plurality of feature points. For example, the evaluation may be performed by adding the similarities calculated for each feature point, or the evaluation may be performed based on the similarities that are equal to or greater than a predetermined value among the similarities calculated for each feature point. Further, calculation may be performed by using a sum of squares or weighting instead of simple addition.

  The tracking status may be evaluated based on the similarity between the acquired image and the image candidate. When evaluating the tracking status, it is preferable to integrate the tracking status of a plurality of images. For example, calculation is performed by adding evaluation results of individual tracking situations, or evaluation is performed by adding evaluation results for a plurality of images of individual feature points.

  The tracking unit creates an image sequence corresponding to the position and orientation based on the image of the feature point image holding unit and the depth candidate pair, and performs tracking by comparing the acquired image sequence with the acquired image. The evaluation unit may perform the evaluation by integrating the evaluation results of the image sequences from which a predetermined or higher evaluation result is obtained for the depth candidate pair. In the tracking process, an image sequence corresponding to the position and orientation is created based on the retained feature point image and the depth candidate pair, and tracking is performed by comparing the acquired image sequence with the acquired image. The evaluation may be performed by integrating the evaluation results of the image sequences in which the evaluation results of a predetermined value or more are obtained for the candidate pair. For example, evaluation may be performed by integrating evaluation results of a plurality of images with high similarity.

  The image processing apparatus further includes a feature point selection unit that selects a feature point for depth estimation from among the feature points, and the depth candidate pair selection unit generates a depth candidate pair for the selected feature point and performs depth estimation. Good. On the other hand, in the image processing method, it is preferable to select a feature point for depth estimation from among the feature points, generate a depth candidate pair for the selected feature point, and perform depth estimation. When depth estimation is completed for the selected feature point, depth estimation may be performed by adding another feature point for which depth estimation has not been completed.

  If there are many feature points and depth candidate pairs are generated for all feature points simultaneously, the number of necessary candidate pairs becomes enormous, making it difficult to select an appropriate depth candidate pair. Depths that need to be generated before selection by generating depth candidate pairs for a small number of feature points selected from the feature points, selecting depth candidate pairs for the feature points, and then adding feature points sequentially The number of candidate pairs can be reduced.

  According to the image processing apparatus and the image processing method of the present invention, the depth representing the positional relationship between the feature points is estimated, and based on the depth candidate pair that is a combination of the depth candidate values and the image viewed from a specific viewpoint. Since the position of the feature point is tracked, the feature point can be accurately tracked even with respect to the change in the position / posture of the object. Since the tracking accuracy depends on the accuracy of the depth candidate pair, the accuracy of the depth candidate pair is also improved by evaluating the tracking accuracy.

  It is easy to track feature points by pre-determining peripheral images of feature points corresponding to changes in the position and orientation of the target object, and searching for the peripheral images found from the acquired images. become.

  By generating a plurality of depth candidate pairs in advance and selecting the most probable candidate pair, an appropriate candidate pair can be selected. Alternatively, the depth candidate pair can be optimized by changing and correcting the candidate pair according to the evaluation situation. Furthermore, when setting the depth candidate pair, extraction is performed from the set range, and furthermore, by correcting this extraction condition, it is possible to reduce the amount of calculation until the optimum solution is obtained and shorten the calculation time. In particular, the accuracy of the initial estimated value can be increased by using the estimation information of the existing position. At this time, for the candidate pair with a low evaluation result, the processing time is shortened without performing unnecessary tracking by stopping the tracking processing. In the evaluation of the tracking status, tracking and evaluation can be performed in a unified manner by making a determination based on the similarity of images.

  A depth candidate pair is set for a part of feature points, and is sequentially added to perform narrowing down, so that depth estimation can be performed quickly even when there are many feature points.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the description, the same reference numerals are given to the same components in the drawings as much as possible, and duplicate descriptions are omitted.

  FIG. 1 is a block configuration diagram showing a first embodiment of an image processing apparatus according to the present invention. The image processing apparatus generates an image acquisition unit 1 that acquires an image sequence (moving image or the like) obtained by capturing a recognition target, a feature point image holding unit 2 that holds a feature point and its peripheral image, and a depth candidate pair. Depth candidate pair generating means 3, tracking means 4 for tracking feature points in each image in the image sequence, and tracking situation evaluation means 5 for evaluating the tracking situation by the tracking means.

  The image acquisition unit 1 is a device that captures a moving image such as a television camera and captures it as an image sequence. In addition, a video capture device or the like that captures an image sequence from moving image data held in a video recorder or the like may be used.

  The feature point image holding unit 2 includes a storage unit such as a memory or a magnetic disk. Here, an example will be described in which a detection unit that detects a feature point from a reference image and an extraction unit that extracts a peripheral image thereof are provided. These detection means and extraction means are constituted by a CPU, a ROM, a RAM, etc., and may be constituted separately from the storage means in terms of hardware, or share a part or all of the hardware. It may be realized as software.

  The depth candidate pair generation unit 3, the tracking unit 4, and the tracking state evaluation unit 5 are all configured by a CPU, a ROM, a RAM, and the like. Each of these means and each means in the feature point image holding means 2 may be configured as independent and independent hardware, but share some or all of the hardware, and each means can be shared by software. It may be realized. In this case, the software constituting each means may be independent, but a part of the software may be shared, or a plurality of means may be incorporated in one software. Good.

  Next, the first embodiment of the image processing method according to the present invention, which is the operation of the present embodiment, will be specifically described with reference to the flowchart of FIG. Here, a facial feature portion is obtained as a pre-process for acquiring a driver's face image by a camera disposed in a vehicle interior of the vehicle, for example, on a steering column, and detecting the driver's face orientation and face position. An example of acquiring the three-dimensional positional relationship, that is, the depth will be described.

First, an image sequence (moving image) is input to the image acquisition means 1 (step S1). The captured image sequence (see FIG. 3) may be stored, for example, in temporary storage means (not shown). Next, the feature point image holding unit 2, detects a face feature point in the initial image I ₀ (step S3). Here, as the initial image, a face image when the driver is facing directly in front may be used. In the feature point detection, the acquired image is displayed on a display device (not shown), and the feature point position in the display image is designated by the driver so that the position is detected. Alternatively, a reference image consisting of a typical luminance pattern of the feature points to be extracted in advance is stored, and the feature points are detected by comparing this with the portion of the input image and extracting portions with high similarity. May be. For example, the following formula may be used as an index of similarity.

Here, f (x, y) is the luminance value at (x, y) on the input image, and g (u, v) is the luminance value at the pixel position (u, v) of the reference luminance pattern. F with a horizontal line drawn is the average luminance of an area having the same size as the reference luminance pattern centered on (x, y) on the input image, and g with a horizontal line drawn on it is the average luminance of the reference luminance pattern. , U and V respectively represent a set whose elements are horizontal and vertical coordinates of the reference luminance pattern.

When the feature point position is detected in this way, it is stored in the storage means in the feature point image holding means 2 together with the surrounding image of a predetermined size. 4 the feature point position in the input image I ₀ in (a) are shown by crisscross. FIG. 4B shows an image group Is _{0 to} Is ₅ cut out around these feature points. Here, an example in which the ends of both eyes and the ends of the mouth are cut out is shown.

  Next, a depth candidate pair is generated (step S5). Here, by generating candidate values of the depths of the feature points, depth candidate pairs that are combinations thereof are generated. Here, a case where one depth candidate pair is created will be described as an example. When the target is a face, the positional relationship between the camera and the face can be ascertained from the size of the face projected on the camera, the part within the face (for example, the size of the eyes), and the camera specification data. Can grasp the actual position.

A face model is constructed from the generated depth candidate pair and the peripheral image of the feature point (step S7). FIG. 5 is a diagram for explaining the face model. The coordinates on the image of the i-th feature point in the initial image are (u _i , v _i ). If the representation of this point in the three-dimensional space by the camera coordinate system is M _i ⁰ = (Xi, Yi, Zi), the following relational expression is established.

Here, s _i represents the set depth, and P, A, R, and t represent a projection matrix, a camera internal matrix, a camera external rotation matrix, and a camera external parallel progression, respectively. The camera internal matrix is known. If the camera coordinate system is the world coordinate system, equation (2) can be rewritten as follows.

Therefore,

Is established, and the spatial position of each feature point can be calculated from the depth candidate pair and the initial image. The combination of the correspondence between the spatial position and the feature point peripheral image is the face model Mf, and the feature point peripheral image is arranged at the spatial position of each feature point as shown in FIG. The model can be simplified as compared with the model Mf ′ imitating the actual three-dimensional shape of the face as shown in FIG.

In subsequent step S9, feature point peripheral images that can be acquired by the camera when the face is in various positions and postures are generated based on the face model, and stored as a database associated with the positions and postures. Here, the change matrix H _i representing the change in the appearance of the i-th feature point representing the amount of change when the face is rotated by w _x , w _y , and w _{z with} respect to the XYZ axes at the position t is expressed by the following equation: It can ask for.

Here, A ₁ is a camera internal parameter, R is a rotation matrix of face motion, t is a translation vector of face motion, M _i is the three-dimensional coordinate of the i-th feature point in the reference coordinate system at the time of reference image shooting. . N _i is a matrix (0 0 1) ^T. As described above, when the position of the i-th feature point on the reference image is (u _i , v _i ) and the pixel position after the motion is (u ′ _i , v ′ _i ), the following relationship is established. .

Here, α is a scalar value.

  Here, the peripheral image of the feature point after exercise is represented by a rectangular area. FIG. 6 shows a relationship between an initial image (see FIG. 6A) and an image after exercise corresponding to the initial image (see FIG. 6B). As shown in FIG. 6B, the size of the peripheral image of the feature point after exercise is assumed to be width 2w + 1 and height 2h + 1. The positions of the end points a ′, b ′, c ′, d ′ of the rectangular area are represented by the following equations.

Since the pixel position (u, v) in the initial image corresponding to the pixel (u ′, v ′) in the rectangular area surrounded by these four points satisfies the relationship of Equation (6), Can be sought.

The luminance value at the pixel position (u, v) obtained by the equation (8) is set as the luminance value at the (u ′, v ′) position after exercise. Here, the pixel position (u, v) obtained by the equation (8) is not an integer value, that is, it may represent an intermediate position of the pixel. In this case, however, the pixel position is based on the luminance value of the surrounding pixels. What is necessary is just to obtain | require the luminance value of the said intermediate position by interpolation. The feature point peripheral image generated in this way is stored in the storage means in the feature point image holding means 2 as a database (see FIG. 7) associated with the face position / orientation.

Next, the verification phase is entered. Here, the verification image sequence taken out the i-th image from (see FIG. ₈₎ I i ~I _n, performs verification. First, the tracking unit 4 takes out the first image (step S11), compares the image with the face image database, and calculates the face image data having the maximum similarity and the similarity with the image. (Step S13).

  When the input image is I and the image in the database is T, the similarity can be obtained by the following equation (9).

Since the face image database has luminance values only in the vicinity of the feature points, the expression (9) is set not to be subject to calculation for areas other than this area. Here, the periphery of each feature point is calculated with the same weight, but the weight may be changed depending on the feature point. This weighting information may be stored in the face image database in association with individual position / posture information. For example, in a right-facing face, the right feature point is likely to be hidden. Therefore, the weight for the image around the right feature point may be set low, and the weight for the image around the left feature point may be set large.

  The tracking state evaluation means 5 adds the similarity calculated in this way (step S15), and determines whether the verification image sequence is completed (step S17). If collation with the entire verification image sequence has not been completed, the process proceeds to step S21, the next verification image is selected, and the flow returns to step S13 to perform verification for all the image sequences. If it is determined in step S17 that the collation has been completed, the process proceeds to step S19, and the tracking state evaluation unit 5 outputs the accumulated value as the evaluation result and ends the process.

  The higher the accuracy of the generated depth candidate pair, the higher the accuracy of the face model generated using this, and the probability that the feature point peripheral image in the verification image sequence matches the image in the face image database. Also gets higher. As a result, the cumulative value itself as the evaluation result also takes a higher numerical value as the accuracy of the depth candidate pair is higher. In other words, the accuracy of the depth candidate pair can be evaluated by the accumulated value.

  Here, the configuration is such that one depth candidate pair is generated and an evaluation value corresponding to the depth candidate pair is generated. However, it is more preferable that the depth candidate pair is corrected and selected based on the evaluation value. This correction / selection may be performed by the operator (in this embodiment, the driver) manually setting another value based on the evaluation result. However, the correction / selection is automatically performed. Is also possible. In the second embodiment having a block configuration shown in FIG. 9, a plurality of depth candidate pairs are generated, and an appropriate depth candidate pair is selected from the generated depth candidate pair group.

  In the second embodiment, depth candidate pair selection means 6 is added to the first embodiment shown in FIG. This depth candidate pair selection means 6 may be independent hardware composed of a CPU, ROM, RAM, etc., but shares some or all of the hardware with other means 1 to 6 and is implemented by software. It may be realized.

The operation of this embodiment, that is, the second embodiment of the image processing method according to the present invention will be described with reference to the flowchart of FIG. Image sequence input (step S1) and face feature point detection (step S3) are substantially the same as in the first embodiment shown in FIG. Subsequently, in place of the generation of depth candidate pairs in step S5 in the first embodiment, a depth candidate group that is a plurality of depth candidate pairs is generated (step S5a). In generating the depth candidate group, for example, if the upper limit value and the lower limit value of the depth of the i-th feature point are high _i and low _i , respectively, and the number of types of depth is n, the k-th depth step _{i, k} Can be expressed by the following equation.

Here, the interval is equal, but the depth range of each feature point may be set at unequal intervals based on the normal distribution or other distributions. Also, a specific feature point may be set as a reference position, and the depth of another feature point may be set in relation to this. In particular, when the relationship between irregularities is clear like a face, there is an effect of eliminating an irrational depth setting far from the reality. At this time, when setting the depth range, if the possible depth range is estimated based on the focus position information in the imaging means, the position information of the driver's seat, etc., the initial depth estimation accuracy can be improved.

  When a plurality of depth candidate pairs (depth candidate group) are set, the first candidate pair is selected from them (step S6), and a face model is constructed (step S7), as in the first embodiment, and a face image is selected. A database is constructed (step S9), and the processing up to accumulation of similarity evaluation results of the collation process (steps S11 to S15) is executed.

  The subsequent step S16 is actually executed for the second and subsequent candidate pairs. As described above, the cumulative value of the similarity evaluation result takes a value corresponding to the probability of the depth candidate pair. Therefore, the currently accumulated evaluation result is compared with the accumulated value up to the image (d-th depth candidate pair in FIG. 11) in the result of the most favorable accumulated value of the evaluation result calculated before that time. If this is an inferior value, there is a possibility that the depth candidate pair has a higher accuracy, but it is not good and the difference is large (for example, the (m + 1) th depth candidate pair shown in FIG. 11). It is clear that the depth candidate pair has low accuracy. In such a case, it is not necessary to track and verify all image sequences. Therefore, when it is determined that the accuracy is low as described above, the process using the depth candidate pair in the subsequent image sequence is stopped by moving to step S18 described later.

  If the accuracy is sufficient, the process proceeds to step S17, where it is determined whether or not all the verification image sequences have been tracked (step S17). If not, the next image sequence is selected. Then, the tracking is continued by returning to step S13 (step S21).

  On the other hand, if the tracking of all the verification image sequences has been completed or has been stopped because the accuracy is low, the process proceeds to step S18, and it is determined whether or not all candidate pair groups have been completed (step S18). ). If the candidate pair group has not ended, the process proceeds to step S22, the next depth candidate pair is selected, and the process returns to step S7 to track and evaluate all the set depth candidate pair groups. Do.

  If it is determined in step S18 that all candidate pairs have been evaluated, the process proceeds to step S20, and an appropriate depth candidate pair can be selected by selecting a candidate pair having the highest similarity. .

  Here, the case where a plurality of depth candidate group setting → selection processes are performed once has been described, but in this case, it is necessary to set depth candidates at fine intervals in order to increase the accuracy of the candidate group. For this reason, if the range of depth candidates is wide, it is necessary to perform enormous calculation. In order to avoid this, for example, evaluation is first performed on a depth candidate group set at a rough interval in a wide depth range, and a narrow range including a candidate pair having an excellent evaluation result is defined as a depth range. It is better to subdivide and perform the evaluation again. If necessary, the depth candidate pair can be obtained with high accuracy with a small amount of calculation by repeating this several times. In addition, in the second and subsequent evaluation processes, the range of the face image database to be constructed can be limited, thereby further reducing the amount of calculation.

  Thus, not only a method of setting a plurality of depth candidate pairs, but also a method of appropriately correcting according to the evaluation result of the depth candidate pair may be used. FIG. 12 is a block diagram showing the configuration of the image processing apparatus according to the third embodiment for performing such correction. This embodiment is basically composed of the same components as those of the first embodiment shown in FIG. 1, except that the evaluation result of the tracking situation evaluation unit 5 is input to the depth candidate pair generation unit 3. Is different.

  The operation of this embodiment, that is, the third embodiment of the image processing method according to the present invention will be described with reference to the flowchart shown in FIG. This process is substantially the same as the process of the second embodiment shown in FIG. 9 except that a plurality of candidates are not generated in advance.

  Instead of generating a plurality of candidate pairs in advance, an initial depth candidate pair is generated (step S5b), a model database is constructed (steps S7 and S9), and a matching process (steps S11 to S17) is executed. After the collation, it is evaluated whether or not the result of the candidate pair group has converged. If it has not converged, the process returns to step S28, the next depth candidate pair is set, and the processes of steps S7 to S17 are repeated. Various variable search algorithms can be used as the depth candidate pair setting method in step S28.

  In the above calculation, evaluation was performed by integrating all evaluation results in the image sequence. However, representative images are extracted according to the position and orientation of the face, and the evaluation results for them are accumulated. May be. Further, instead of adding together and evaluating all the similarities for individual feature points, each feature point may be evaluated individually. When each feature point is handled individually in this way, when only the depth evaluation for some of the feature points is incorrect, the evaluation of other feature points is not affected. Note that feature points estimated to have the same depth may be grouped and evaluated for each group.

If there are many feature points on the object, the number of depth candidate pairs becomes enormous when the processing according to the second and third embodiments is performed on all feature points at the same time. Become. About 10 pieces of feature points, even if the depth candidates of the feature points is only three, the number of depth candidates pairs which may reach in triplicate ¹⁰ types = 59049. Therefore, by narrowing down the number of feature points to be tracked first, obtaining depth candidate pairs for those feature points, and then repeatedly obtaining depth candidate pairs for the remaining feature points, the processing time can be reduced. It is preferable to shorten.

  FIG. 14 is a flowchart showing this processing (fourth embodiment). Image sequence input (step S1) and face feature point detection (step S3) are the same as in the first to third embodiments. Next, among the feature points, a depth candidate pair is set first, and the feature points for which the depth is to be determined are determined (step S4). Then, a depth candidate pair calculation process (step S30) including a model database construction and a matching process is executed for the feature point. This corresponds to the processing in steps S5 to S19 in the second embodiment and the processing in steps S5b to S26 in the third embodiment.

  When the depth is calculated for the feature point, the process proceeds to step S32, where it is determined whether or not the feature point for which the depth is not obtained remains. Of the feature points, a predetermined number of feature points are added (step S36), and the process returns to step S30 to calculate the depth of the feature points sequentially. If it is determined in step S32 that the depths of all feature points have been obtained, the process proceeds to step S34, the final candidate pair is output, and the process ends. As a result, in the above-described example, the depth of the feature point can be obtained by performing 3 types × 10 times = 30 times in the minimum case, and the calculation time can be greatly shortened. Become.

  In the above description, the case where the present invention is applied to an in-vehicle face recognition camera has been described as an example. However, the present invention is not limited to this, and can be applied to, for example, object recognition in a robot or a monitoring device. It is.

1 is a block diagram showing a configuration of a first embodiment of an image processing apparatus according to the present invention. It is a flowchart which shows the process of 1st Embodiment of the image processing method concerning this invention. It is a figure explaining the image sequence input. The figure which shows the feature point position in an input image (Fig.4 (a)), and the example of the extracted feature point periphery image (FIG.4 (b)). It is a figure explaining the face model used by this invention. It is a figure which shows the relationship between an initial image and the image after the exercise | movement corresponding to this. It is a figure which shows an example of a face image database. An example of the verification image sequence is a bell. It is a block diagram which shows the structure of 2nd Embodiment of the image processing apparatus concerning this invention. It is a flowchart which shows the process of 2nd Embodiment of the image processing method concerning this invention. It is a graph which shows the change of the cumulative value of an evaluation result. It is a block diagram which shows the structure of 3rd Embodiment of the image processing apparatus concerning this invention. It is a flowchart which shows the process of 3rd Embodiment of the image processing method concerning this invention. It is a flowchart which shows the process of 4th Embodiment of the image processing method concerning this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image acquisition means, 2 ... Feature point image holding means, 3 ... Depth candidate pair production | generation means, 4 ... Tracking means, 5 ... Tracking condition evaluation means, 6 ... Depth candidate pair selection means

Claims (32)

Means for acquiring a plurality of images including three or more feature points on an object whose depth is to be estimated;
Feature point image holding means for holding each feature point from a predetermined viewpoint and its peripheral image;
Depth candidate pair generation means for generating a depth candidate pair that is a set of depth candidate values for feature points;
Tracking means for tracking the position of the feature point in the acquired image using the image held by the feature point image holding means and the generated depth candidate pair;
Tracking status evaluation means for evaluating the tracking status by the tracking means;
An image processing apparatus comprising:   The tracking unit generates a feature point peripheral image candidate when the position / posture of the object is changed from the image held by the feature point image holding unit and the generated depth candidate pair, and the feature point peripheral image generated from the acquired image The image processing apparatus according to claim 1, wherein tracking is performed by searching for a candidate.   The depth candidate pair generation means further includes a depth candidate pair selection means for generating a plurality of sets of depth candidate pairs and selecting an appropriate depth candidate pair based on an evaluation result by the tracking situation evaluation means. The image processing apparatus according to claim 1, wherein:   The depth candidate pair generation unit generates a new depth candidate pair by changing the value of each element of the candidate pair based on the evaluation result by the tracking state evaluation unit for the generated depth candidate pair. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, wherein the depth candidate pair generation unit generates a plurality of depth candidate pairs by extracting the depth of each feature point from a set range under a predetermined condition.   Based on the depth candidate pair selected from the depth candidate pair generated by the depth candidate pair generation means, the depth candidate pair is generated by further restricting the range and extraction conditions of each depth, and the tracking means, tracking status evaluation means, depth 6. The image processing apparatus according to claim 5, wherein tracking, evaluation, and selection by the candidate pair selection unit are re-executed. Further comprising means for obtaining estimated information of the location of the feature point;
The image processing apparatus according to claim 1, wherein the depth candidate pair generation unit generates a depth candidate pair based on the acquired position estimation information of feature points.   The image processing apparatus according to claim 1, wherein when the evaluation result by the tracking state evaluation unit is equal to or less than a predetermined value, tracking for the depth candidate pair is terminated.   The image processing apparatus according to claim 1, wherein the tracking state evaluation unit performs evaluation by integrating tracking states of a plurality of images.   The image processing apparatus according to claim 1, wherein the tracking state evaluation unit performs determination based on a similarity between a feature point image held by the feature point image holding unit and an image near the tracking position.   The image processing apparatus according to claim 10, wherein the tracking state evaluation unit performs evaluation by adding similarity calculated for each feature point.   The image processing apparatus according to claim 10, wherein the tracking state evaluation unit performs evaluation based on a similarity greater than or equal to a predetermined value among similarities calculated for each feature point.   The tracking unit creates an image sequence according to the position and orientation based on the image of the feature point image holding unit and the depth candidate pair, and performs tracking by comparing the image sequence with the acquired image, The image processing apparatus according to claim 1, wherein the tracking state evaluation unit performs the evaluation based on a similarity between the image sequence and the acquired image.   The tracking unit creates an image sequence according to the position and orientation based on the image of the feature point image holding unit and the depth candidate pair, and performs tracking by comparing the image sequence with the acquired image, The image processing apparatus according to claim 1, wherein the tracking state evaluation unit performs evaluation by integrating evaluation results of image sequences in which a predetermined or higher evaluation result is obtained for the depth candidate pair.   A feature point selecting unit that selects a feature point for depth estimation from among the feature points, wherein the depth candidate pair selecting unit generates a depth candidate pair for the selected feature point and performs depth estimation; The image processing apparatus according to claim 1.   16. The image processing apparatus according to claim 15, wherein when depth estimation is completed for the selected feature point, depth estimation is executed by adding another feature point for which depth estimation has not been completed. Acquire multiple images including 3 or more feature points on the object whose depth is to be estimated,
Each feature point from a predetermined viewpoint and a feature point image that is a peripheral image thereof are retained,
Generate depth candidate pairs that are pairs of depth candidate values for feature points;
An image processing method comprising: a step of tracking a position of a feature point in an acquired image using a feature point image held and a generated depth candidate pair, and evaluating a tracking state.   A feature point peripheral image candidate at the time of changing the position / orientation of the target object is generated from the retained feature point image and the generated depth candidate pair, and tracking is performed by searching the generated feature point peripheral image candidate from the acquired image. The image processing method according to claim 17, wherein the image processing method is performed.   18. The image processing method according to claim 17, further comprising a step of generating a plurality of pairs of depth candidate pairs and performing tracking, and selecting an appropriate depth candidate pair based on the evaluation result of the tracking state.   18. The image processing method according to claim 17, wherein a new depth candidate pair is generated by changing a value of each element of the candidate pair based on an evaluation result for the generated depth candidate pair.   The image processing method according to claim 17, wherein a plurality of depth candidate pairs are generated by extracting the depth of each feature point from a set range under a predetermined condition.   22. The image processing method according to claim 21, wherein a depth candidate pair is generated based on the selected depth candidate pair, and the depth range and extraction conditions are further tightened, and tracking, evaluation, and selection are performed again. .   18. The image processing method according to claim 17, further comprising: obtaining estimation information of feature point existence positions, and generating depth candidate pairs based on the obtained feature point position estimation information.   18. The image processing method according to claim 17, wherein when the tracking state evaluation result is equal to or less than a predetermined value, tracking for the depth candidate pair is terminated.   18. The image processing method according to claim 17, wherein the tracking status is evaluated by integrating tracking statuses of a plurality of images.   18. The image processing method according to claim 17, wherein the tracking state is evaluated based on the similarity between the feature point image held and the image near the tracking position.   27. The image processing method according to claim 26, wherein the tracking status is evaluated by adding the similarity calculated for each feature point.   27. The image processing method according to claim 26, wherein the tracking status is evaluated based on a similarity greater than or equal to a predetermined value among the similarities calculated for each feature point.   An image sequence corresponding to the position and orientation is created based on the stored feature point image and depth candidate pair, and tracking is performed by comparing the acquired image sequence with the acquired image sequence. 18. The image processing method according to claim 17, wherein the tracking status is evaluated based on a similarity with the image.   Create an image sequence according to the position and orientation based on the stored feature point image and depth candidate pair, and perform tracking by comparing the image sequence with the acquired image. 18. The image processing method according to claim 17, wherein the evaluation is performed by integrating the evaluation results of the image sequences from which a predetermined or higher evaluation result is obtained.   18. The image processing method according to claim 17, wherein a feature point for depth estimation is selected from the feature points, and a depth candidate pair is generated for the selected feature point to perform depth estimation.   32. The image processing method according to claim 31, wherein, when depth estimation is completed for the selected feature point, depth estimation is executed by adding another feature point for which depth estimation has not been completed.

Images