JP2008085491A - Image processor, and image processing method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a tracking image processor which can shorten the time required for getting an optimal reference image when tracking image processing for object tracking is performed by acquiring a reference image representing the appearance of the object and can evaluate the reference image three-dimensionally. <P>SOLUTION: Computation time is shortened by adding a semi-tracking mode between a detection mode and a tracking mode, and the probability of getting an optimal reference image increases because many frames can be processed. In addition, a success rate of tracking is enhanced. Furthermore, since the reference image is evaluated three-dimensionally for face direction, a face image with a wider facing range is obtained as a reference face image, thus enhancing the success rate of tracking. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that processes an image input from a camera, detects an image area of a target object, and efficiently shifts to a tracking process, and a method thereof.

  The method of detecting a specific area from a video as disclosed in Patent Document 1 and tracking the specific area in subsequent frames generally processes all frames that are input for a long processing time of the detection processing unit. In some cases, the optimal image to be used as the initial value of the tracking processing unit may be missed.

In addition, a method for evaluating an optimal image for use in tracking processing and recognition processing as disclosed in Patent Document 2 and Patent Document 3 is evaluated from the two-dimensional arrangement of face parts and the appearance of faces and face parts. However, an image evaluated only from such information could not provide a truly optimal image, and the tracking process using the image led to a decrease in the success rate.
Japanese Patent No. 3576734 JP 2002-49912 A Japanese Patent Laid-Open No. 2005-22795

  As described above, in the prior art, information given as prior knowledge is used as a dictionary, and a reference image that is an initial value in a detection mode in which detection processing is performed using the dictionary is acquired, and a tracking mode in which tracking processing is performed using the reference image is performed. There is a method for performing the process of migration.

  However, since the processing time in the detection mode is long and all frames cannot be processed, a frame having an optimal reference image may be missed.

  When handling a face image, the reference face image, which is the initial value in the detection mode, is evaluated as a face image facing the camera from the two-dimensional arrangement of face parts and the appearance of the face and face parts. It was. However, this evaluation method sometimes obtains a reference face image with a face orientation that is not optimal, and the tracking success rate is reduced.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus and method capable of processing more frames and improving the tracking success rate.

  The present invention provides an image input unit that inputs a time-series image, a dictionary storage unit that stores dictionary information related to a target object that is a detection target, a reference storage unit that stores a reference image and a reference similarity, and the input image The first corresponding area of the target object is detected from the dictionary information, and a first similarity between the dictionary information and the image of the first corresponding area is obtained, and the image of the first corresponding area and the first A detection mode execution unit that executes a detection mode for updating the reference image and the reference similarity with a similarity; a second target region is detected from the input image using the reference image; and the dictionary information and the second A second similarity with the image of the corresponding region is obtained, and when the obtained second similarity is higher than the similarity of the reference image, the reference image and the second image with the second corresponding region and the second similarity are obtained. Semi-tracking to update reference similarity A quasi-tracking mode execution unit that executes a tracking mode, a tracking mode execution unit that executes a tracking mode that detects the target region using the reference image from the input image, and (1) first switch to the detection mode, (2) When the first similarity is higher than a first threshold during execution of the detection mode, the mode is switched to the semi-tracking mode. (3) During the execution of the semi-tracking mode, the second similarity is When it becomes higher than the second threshold value higher than the first threshold value, the mode is switched to the tracking mode, and (4) the second similarity level is set to the first threshold value and the second threshold value during execution of the semi-tracking mode. And a switching unit that continues the semi-tracking mode.

  Compared to the conventional method, the calculation time is shorter and more frames can be processed, so that the probability of obtaining the optimum reference image increases, thereby improving the tracking success rate.

  In addition, when the target object is a face, it is possible to acquire a more accurate optimal face orientation image by considering the face orientation in three dimensions in the evaluation of the reference face image, thereby improving the tracking success rate. .

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
A tracking image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to FIGS. 1, 2, 8, and 9.

  When the tracking image processing apparatus 1 according to the present embodiment receives a time-series image (moving image, video) that captures the target object, the tracking image processing apparatus 1 detects the target object in the image by the processing method in the selected mode. Further, it is evaluated whether the detected image of the target area is suitable for the reference image, and the mode in the next frame is selected based on the evaluation value. That is, the tracking image processing device 1 is characterized by having a switching unit 5 that switches between three modes of a detection mode, a semi-tracking mode, and a tracking mode based on the evaluation value.

(1) Configuration of Tracking Image Processing Device 1 FIG. 1 is a block diagram showing a tracking image processing device 1 according to the present embodiment.

  The tracking image processing apparatus 1 includes an image input unit 2, a detection unit 3, a reference image evaluation unit 4, a switching unit 5, a dictionary storage unit 6, a reference image storage unit 7, and an output unit 8.

  Each function described below of each unit 2 to 7 can also be realized by a program stored in a computer.

(1-2) Image input unit 2
The image input unit 2 receives a time-series image obtained by imaging the target object, and sends the input image to the detection unit 3.

(1-3) Dictionary storage unit 6
The dictionary storage unit 6 records dictionary information for detecting the target object in the most general state learned in advance, and stores the dictionary information in the detection unit 3 and the reference image evaluation unit 4 according to the mode. send.

(1-4) Detection unit 3
The detection unit 3 detects a target region of the target object from the input image from the image input unit 2 and obtains an image of the target region as a detection image. At this time, the process for detecting the target region differs depending on the mode switched by the switching unit 5. As this mode, there are a detection mode, a semi-tracking mode, and a tracking mode.

  In the detection mode, the detected image is detected using the dictionary information in the dictionary storage unit 6. The similarity between the detected image and the dictionary information is sent to the reference image evaluation unit 4. Further, the detected image and the position information are output to the output unit 8. Here, subspace method (Fukui, Yamaguchi, “Face feature point extraction by combination of shape extraction and pattern matching”), IEICE Journal, Aug. 1997, J80-D-II, No. 8, p.2170-2177). However, the detection method may be any method that can identify the target region in the image using dictionary information.

  In the semi-tracking mode and the tracking mode, the target area is detected using the reference image in the reference image storage unit 7. The similarity between the detected image and the reference image information is sent to the reference image evaluation unit 4. Further, the detected image and the position information are output to the output unit 8. As a method of detecting the region of the target object using the reference image, QTR method (Osamu Yamaguchi, Kazuhiro Fukui. Image matching based on qualitative ternary expression. IEICE technical report, PRMU2002-34, pp. 23 -30, June 2002.). However, any method can be used as long as it can identify an area most similar to the reference image in the image.

  Further, since the similarity obtained in the detection mode and the similarity obtained in the semi-tracking mode and the tracking mode are acquired by different methods, the scales are different. Therefore, in order to facilitate the subsequent processing, each similarity is normalized so that the scales are the same.

(1-5) Reference image evaluation unit 4
The reference image evaluation unit 4 obtains an evaluation value from the detected image detected by the detection unit 3 and the position information, and determines whether or not to update the reference image from this evaluation value.

  This evaluation value is obtained as follows. First, the similarity between the dictionary information in the dictionary storage unit 6 and the detected image is set as a first evaluation value. Further, a second evaluation value obtained by evaluating the orientation of the target object with respect to the camera by the geometrical arrangement of the specific part of the target object is obtained. Then, the first evaluation value and the second evaluation value are synthesized to obtain a final evaluation value. Note that the second evaluation value is set to be high when the target object is oriented in a specific direction. For example, when the target object is a face, the second evaluation value is increased as it faces the front.

  The method for obtaining the first evaluation value may be any method that can quantify how similar the two images are. However, similar to the detection unit 3, different processing is performed depending on the mode switched by the switching unit 5.

  In the detection mode and the semi-tracking mode, an evaluation value as a reference image is obtained using dictionary information in the dictionary storage unit 6. The evaluation value is not obtained in the tracking mode.

  The obtained evaluation value is sent to the switching unit 5. If the obtained evaluation value is higher than the evaluation value of the current reference image, the detected image and its position information are sent to the reference image storage unit 7 as a new reference image.

(1-6) Switching unit 5
The switching unit 5 switches the mode according to the evaluation value from the reference image evaluation unit 4. First, the initial value is the detection mode. Then, when the evaluation value exceeds the first mode switching threshold S1, the mode is switched to the semi-tracking mode, and when the evaluation value exceeds the second mode switching threshold S2 (where S1 <S2), the mode is switched to the tracking mode. In addition, when detection fails for a certain period for the target object in all modes, the mode shifts to the detection mode.

(1-7) Reference image storage unit 7
The reference image storage unit 7 records the detected image and its position information sent from the reference image evaluation unit 4 as reference image information, and sends it as reference image information to the detection unit 3 and the reference image evaluation unit 4 according to the mode.

(1-8) Output unit 8
The output unit 8 outputs the detected image and its position information sent from the detection unit 3 to the outside of the apparatus.

(2) Contents of Mode In the detection mode, the detection unit 3 detects the image (= detected image) of the target region in the image using the dictionary information in the dictionary storage unit 6. Next, the reference image evaluation unit 4 uses the dictionary information in the dictionary storage unit 6 to evaluate the optimality of the detected image as a reference image.

  In the semi-tracking mode, the detection unit 3 detects the target area (= detected image) using the reference image information in the reference image storage unit 7. Next, the reference image evaluation unit 4 uses the dictionary information in the dictionary storage unit 6 to evaluate the optimality of the detected image as a reference image.

  In the tracking mode, the detection unit 3 detects the target area using the reference image information in the reference image storage unit 7. The reference image evaluation unit 4 does not perform evaluation.

(3) Tracking Image Processing Procedure FIG. 2 is a flowchart showing an example of the tracking image processing procedure according to this embodiment. The processing procedure is as follows.

  First, in step 1, one frame of the input time-series image is input to the image input unit 2. If there is an image that has already been input, an image for the next one frame is input.

  Next, in step 2, it is determined whether the detection mode or the semi-tracking mode and the tracking mode are set. The initial value is determined as the detection mode.

  Next, when it is determined in step 2 that the mode is the detection mode, in step 3, the detection unit 3 uses the dictionary information in the dictionary storage unit 6 to detect the target region, and the detected image, the dictionary information, Are sent to the reference image evaluation unit 4 (= first evaluation value).

  Next, when the semi-tracking mode and the tracking mode are determined in step 2, in step 4, the detection unit 3 detects the target region using the reference image of the reference image recording unit 7, and the detected image and the reference mode are detected. The similarity (= first evaluation value) with the image is sent to the reference image evaluation unit 4.

  Next, in step 5, it is determined whether similarity (= first evaluation value)> = threshold value 1 based on the similarity (= first evaluation value) received from step 3 or step 4. If yes, go to Step 6, otherwise go to Step 11.

  Next, in step 6, it is determined whether the detection mode and the semi-tracking mode or the tracking mode is set. When it is determined that the detection mode and the semi-tracking mode are selected, the process proceeds to Step 7, and when the tracking mode is selected, the process proceeds to Step 10.

  Next, in step 7, since the detection mode and the semi-tracking mode are determined, the reference image evaluation unit 4 uses the dictionary information to evaluate the detection image (= first evaluation value + second evaluation value). ) And go to step 8.

  Next, in step 8, it is determined whether or not evaluation value> = threshold value 2. If satisfied, the process proceeds to step 9, and if not satisfied, the process proceeds to step 10.

  Next, in step 9, the switching unit 5 switches the mode based on the evaluation value. If the evaluation value is higher than the evaluation value of the current reference image, the detected image is recorded as a reference image in the reference image storage unit 7, and the process proceeds to step 10.

  Next, in step 10, the detected image and position information are output to the outside of the apparatus by the output unit 8, and the process proceeds to step 11.

  Next, in step 11, it is determined whether or not to end the process. If not, the process returns to step 1, and if it ends, the tracking image process is ended.

(4) Effects Hereinafter, effects of introducing the semi-tracking mode will be described.

  FIG. 8 shows changes in the amount of calculation processing between the present embodiment and the prior art.

  The present embodiment shifts to the quasi-tracking mode at time t1 and shifts to the tracking mode at time t2, whereas the conventional technology shifts to the tracking mode at time t3.

  In general, the detection mode requires a large amount of calculation processing and the calculation time is slower than the tracking mode. However, in the tracking mode, since the reference image is not acquired and the tracking success rate is not improved, the detection mode for evaluating the reference image is important.

  Therefore, by using a detection method similar to that in the tracking mode, a quasi-tracking mode in which only the calculation time of the evaluation portion is slower than in the tracking mode is introduced. The frame rate becomes higher, and more images can be processed.

  Next, FIG. 9 shows the operation contents of this embodiment and the prior art until an optimum reference image is acquired. In addition, the length of the arrow in a figure is calculation time.

  In the present embodiment, a reference image that is close to optimum is acquired at time t1, and the mode shifts to the quasi-tracking mode with a fast calculation time. The optimal reference image is acquired at time t2 and the mode is shifted to the tracking mode.

  On the other hand, according to the conventional technique, since the calculation time of the detection mode is slow, the reference image cannot be acquired until time t3, and the tracking mode cannot be entered. In addition, although it is desired to output the face orientation in more frames, in the case of the conventional technique, only 3 times can be output until time t3, whereas this embodiment outputs 13 times. It is effective because it can.

  The difference between the detection mode and the quasi-tracking mode is that the detection mode uses dictionary information for detection processing, so that detection can be performed on a general basis. However, the semi-tracking mode is that only a specific target object can be detected because the reference image is used for the detection process.

  The reason why the tracking mode is necessary is that since the initial value in the tracking process is evaluated in the detection mode and the semi-tracking mode, it is not necessary to update the reference image when the optimum initial value is obtained. In the tracking mode, the reference image is constant, but the tracking success rate is improved by, for example, updating a template used for the tracking process.

(5) Modification Example In the above embodiment, the first evaluation value + the second evaluation value is obtained as the evaluation value, but only the first evaluation value (similarity) may be used instead. .

(Second Embodiment)
A tracking image processing apparatus 101 according to a second embodiment of the present invention will be described with reference to FIGS.

  The tracking image processing apparatus 101 adjusts the processing amount of the reference image evaluation unit 104 based on the parameters recorded in the parameter storage unit 109 in the semi-tracking mode.

(1) Configuration of Tracking Image Processing Device 101 FIG. 3 is a configuration diagram showing the tracking image processing device 101 according to the present embodiment.

  The tracking image processing apparatus 101 includes an image input unit 102, a detection unit 103, a reference image evaluation unit 104, a switching unit 105, a dictionary storage unit 106, a reference image storage unit 107, an output unit 108, and a parameter storage unit 109. .

  Of the operation of the tracking image processing apparatus 101, the description of the same processing as that of the tracking image processing apparatus 1 of the above embodiment will be omitted.

  The reference image evaluation unit 104 evaluates the detection image from the detection unit 103 and the evaluation value of the position information and the optimality of the detection image as the reference image.

  This evaluation value is obtained as follows. First, the similarity between the dictionary image in the dictionary storage unit 106 and the detected image is set as the first evaluation value. Further, a second evaluation value obtained by evaluating the orientation of the target object with respect to the camera by the geometrical arrangement of the specific part of the target object is obtained. Then, the first evaluation value and the second evaluation value are synthesized to obtain a final evaluation value.

  The method for obtaining the first evaluation value (= similarity) may be any method that can quantify how similar two images are and can adjust the calculation cost. Here, the calculation cost indicates the calculation processing amount and frequency. For example, in order to adjust the processing amount of the calculation for evaluation, in the case of the subspace method, the number of dimensions of the space to be used may be changed. Further, in order to adjust the frequency of performing the evaluation, the processing may be performed at regular intervals instead of performing the processing every frame. Different processing is performed depending on the mode switched by the switching unit 105.

  In the detection mode, optimality as a reference image is evaluated using dictionary information in the dictionary storage unit 106.

  In the quasi-tracking mode, based on the parameters in the parameter storage unit 109, processing is performed in which the number of subspace dimensions is reduced and the amount of calculation is reduced as the parameters approach the threshold value. As the frequency approaches, the frequency of evaluation is reduced to reduce the amount of calculation, and the optimality as a reference image is evaluated. The evaluation value acquired at this time is recorded in the parameter storage unit 109 as the next parameter.

  No evaluation is performed in the tracking mode.

  As the evaluation value increases, the semi-tracking mode process is the same as the tracking mode process.

  The obtained evaluation value is sent to the switching unit 105. If the obtained evaluation value is higher than the evaluation value of the current reference image, the detected image and its position information are sent to the reference image storage unit 107 as a new reference image.

  The parameter storage unit 109 stores the evaluation value obtained from the reference image evaluation unit in the previous process as a parameter. That is, the parameter storage unit 109 receives and records the parameter from the reference image evaluation unit 104, and sends the previous parameter to the reference image evaluation unit 104 in response to a request.

(2) Tracking Image Processing Procedure FIG. 4 is a flowchart showing an example of the tracking image processing procedure according to this embodiment. The processing procedure is as follows. Description of processing similar to the tracking image processing procedure described in FIG. 2 is omitted.

  In step 106, it is determined whether the detection mode, the semi-tracking mode, or the tracking mode. When in the detection mode, the process proceeds to step 7; when in the semi-tracking mode, the process proceeds to step 112; when in the tracking mode, the process proceeds to step 110.

  Next, since the detection mode is determined in step 107, an evaluation value is obtained from the detected image and position information using the dictionary information in the dictionary storage unit 106, and the process proceeds to step 108.

  Next, in step 112, parameters in the parameter storage unit 109 are received.

  Next, in step 113, an evaluation value is obtained from the detected image and the position information using the dictionary information in the dictionary storage unit 106. The amount of calculation is changed based on the parameters received at this time.

  Next, in step 114, the evaluation value is set as the next parameter, recorded in the parameter storage unit 109, and the process proceeds to step 108.

(Third embodiment)
A face direction estimation apparatus 201 according to the third embodiment of the present invention will be described with reference to FIGS. The face orientation estimation apparatus 201 according to the present embodiment is such that the target object is a human face in the tracking image processing apparatus 1 of the above embodiment. Therefore, among the operations of the face orientation estimation device 201, the description of the same processing as that of the tracking image processing device 1 described above will be omitted.

(1) Configuration of Face Orientation Estimation Device 201 FIG. 5 is a configuration diagram showing the face orientation estimation device 201 according to the present embodiment.

  The face orientation estimation apparatus 201 includes an image input unit 202, a detection unit 203, a reference image evaluation unit 204, a switching unit 205, a dictionary storage unit 206, a reference image storage unit 207, an output unit 208, a face detection unit 210, and a feature point detection unit. 211, a similarity evaluation unit 212, a face direction evaluation unit 213, a face direction estimation unit 14, and a general face shape storage unit 215.

(1-1) Detection unit 203
The detection unit 203 includes a face detection unit 210 and a feature point detection unit 211.

  The face detection unit 210 detects a face area from the input image from the image input unit 202.

  The feature point detection unit 211 detects the coordinates of the face feature points in the face area.

  Then, the detection unit 203 acquires a neighborhood image of the face area image and the feature point coordinates. However, the detection process differs depending on the mode switched by the switching unit 205.

  In the detection mode, the coordinates of the face area and the face feature point in the image are detected using the dictionary information in the dictionary storage unit 206.

  In the semi-tracking mode and the tracking mode, the coordinates of the face area and the face feature point are detected using the reference face image information in the reference image storage unit 207. The detected face image, its position information, the feature point coordinates, and its neighboring image are sent to the reference image evaluation unit 204, and the face position estimation unit 214 is sent the face position information and the feature point coordinates.

(1-1-1) Face detection unit 210
The face detection unit 210 detects a face area from the input image and obtains the face image. At this time, the process for detecting an area differs depending on the mode switched by the switching unit 205.

  In the detection mode, the face area in the input image is detected using the dictionary information in the dictionary storage unit 206.

  In the semi-tracking mode and the tracking mode, the face area is detected using the reference face image information in the reference image storage unit 207.

(1-1-2) Feature point detection unit 211
The feature point detection unit 211 detects the coordinates of the feature points using the input image, and obtains a neighborhood image of the coordinates. At this time, the detection process differs depending on the mode switched by the switching unit 205.

  In the detection mode, the coordinates of the feature points in the input image are detected using the dictionary information in the dictionary storage unit 206.

  In the semi-tracking mode and the tracking mode, the feature point coordinates are detected using the reference face image information in the reference image storage unit 207.

(1-2) Reference image evaluation unit 204
The reference image evaluation unit 204 includes a similarity evaluation unit 212 and a three-dimensional face direction evaluation unit 213.

  The reference image evaluation unit 204 uses the face image detected by the detection unit 203, the image of the feature point, and the position information, and the first evaluation value obtained from the similarity evaluation unit 212 and the three-dimensional face direction evaluation. By combining the second evaluation values obtained from the unit 213, the optimality as a reference face image is evaluated.

  This evaluation method performs different processing depending on the mode switched by the switching unit 205.

  In the detection mode and the semi-tracking mode, the first evaluation value obtained from the similarity evaluation unit 212 and the second evaluation obtained from the three-dimensional face direction evaluation unit 213 are used using the dictionary information in the dictionary storage unit 206. The optimality as a reference face image is evaluated by combining the values.

  Note that no evaluation is performed in the tracking mode.

  The obtained evaluation value is sent to the switching unit 205. If the obtained evaluation value is higher than the evaluation value of the current reference image, the detected image and its position information are sent to the reference image storage unit 207 as a new reference image.

  That is, the reference image evaluation unit 204 uses the first evaluation value f (x) obtained from the similarity evaluation unit 212 and the second evaluation value g (x) from the three-dimensional face orientation evaluation unit 213, The reference face image is evaluated by synthesizing with an expression such as αf (x) + βg (x). Where x is a frame number.

(1-2-1) Similarity evaluation unit 212
The similarity evaluation unit 212 calculates the similarity between the face and feature point images received from the detection unit 203 and the dictionary information in the dictionary storage unit 206 in order to use a general and non-occlusion image as a reference image. Earn individually.

  The dictionary information includes dictionary information (correct answer dictionary information) having a plurality of correct answer information and dictionary information (error dictionary information) having a plurality of error information. Then, the similarity between the two dictionary information is obtained for each image.

  First. Among the correct face dictionary information of a plurality of faces, the highest value obtained by subtracting each similarity from the face correct answer threshold is set as the first face correct similarity.

  Next, among the plurality of face error dictionary information, the highest value obtained by subtracting each similarity from the face error threshold is set as the second face correct similarity.

  Next, a value obtained by subtracting the second face correct similarity from the first face correct similarity is set as the face correct / incorrect similarity.

  For all feature points, among the correct answer dictionary information of a plurality of feature points, the highest value among subtracted similarities from the feature point correct answer threshold is set as the first feature point correct similarity.

  Next, among the plurality of feature point error dictionaries, the highest value obtained by subtracting each similarity from the feature point error threshold is set as the second feature point positive similarity.

  Next, a value obtained by subtracting the second feature point correct similarity from the first feature point correct similarity is set as the feature point correct similarity.

  For each feature point, the greater of the feature point correct similarity and the feature point correct / incorrect similarity is defined as the feature point similarity.

  In addition, a value obtained by adding four of the feature point correct similarity, the feature point correct / incorrect similarity, the face correct answer dictionary, and the face correct / incorrect similarity of the feature points having the lowest feature point similarity among the feature points is the first. The evaluation value f (x) is 1.

(1-2-2) Three-dimensional face orientation evaluation unit 213
The three-dimensional face orientation evaluation unit 213 receives the face and feature point position information received from the detection unit 203.

  Here, FIG. 7 is a conceptual diagram showing a range of face orientations that can be tracked with respect to the front face, and a face image that faces the front direction in the center can be traced when the face orientation is changed. Is indicated by a dotted line.

  If the reference face image is the front direction as much as possible, the range that can be tracked is wider than the traceable face direction range shown by the solid line in FIG. That is, the second evaluation value is increased as the image faces the front direction.

  Therefore, in order to evaluate the face facing the front direction, the position information of the face and the feature point is sent to the face direction estimation unit 214, and the lateral direction angle, the vertical direction angle, and the neck angle of the face direction with respect to the camera are received and evaluated.

  Further, in order to evaluate an outlier regarding the image coordinates of each feature point, the face orientation estimation unit 214 receives and evaluates a shape fitting error and an error from an average position learned in advance for each feature point with respect to the face. The shape fitting error is caused by a feature point detection error or the like, and is an error value when fitting the observed feature point to the general face shape. A second evaluation value g (x), which is a three-dimensional face orientation evaluation value, is based on the evaluation of the angle and the evaluation of the displacement of the feature point.

(1-3) General face shape storage unit 215
The general face shape storage unit 215 records the general face shape learned in advance and the average position of each feature point with respect to the face, and sends it to the face direction estimation unit 214 if requested.

(1-4) Face orientation estimation unit 214
The face orientation estimation unit 214 uses the general face shape recorded in the general face shape storage unit 215 for the image coordinate information of the face and the feature point received from the detection unit 203, and the horizontal orientation angle, the vertical orientation angle, and the neck curl. A corner and shape fitting error is obtained and sent to the output unit 208.

  Also, the position information of the face and feature points received from the three-dimensional face direction evaluation unit 213 is used for the horizontal angle, the vertical angle, the neck angle, and the shape fitting using the general face shape of the general face shape storage unit 215. The error and the error from the average position of each feature point with respect to the face are obtained, and these are returned to the three-dimensional face orientation evaluation unit 213. The average position of each feature point is recorded in the general face shape storage unit 215.

  The general face shape model is defined in advance by a factorization method or the like. In this general face shape model, the current face orientation is the initial value. Non-patent literature (Kanade, Poelman, Morita, “Restoring Object Shape and Camera Motion by Factorization”) IEICE Transactions D-II No.8, pp-1947-1505, Aug .1993) is used.

(2) Face Orientation Estimation Procedure FIG. 6 is a flowchart showing an example of a face orientation estimation procedure according to this embodiment. The processing procedure is as follows. Description of processing similar to the tracking image processing procedure described in FIG. 2 is omitted.

  Step 203 will be described. Step 203 is composed of Step 215 and Step 216.

  In step 215, the face area in the image is detected using the dictionary information in the dictionary storage unit 206, and the similarity between the detected image, the position information, and the used dictionary information is sent to the reference image evaluation unit 204. Next, in step 216, the image information of the feature point in the image is detected using the dictionary information, and the similarity between the detected image coordinate and its neighboring image and the used dictionary information is sent to the reference image evaluation unit 204.

  Step 204 will be described. Step 204 includes step 217 and step 218.

  In step 217, the face area in the image is detected using the reference face image information in the reference image storage unit 207, and the similarity between the detected image, the position information, and the used dictionary information is sent to the switching unit 205. Next, in step 218, the image coordinates of the feature points in the image are detected using the reference face image information, and the similarity between the detected image coordinates and the neighboring image and the dictionary information used is sent to the switching unit 205.

  Step 207 will be described. Step 207 includes step 219 and step 220.

  In step 219, first, the dictionary information in the dictionary storage unit 206 is used to evaluate the three-dimensional face direction from the face and feature point position information received from the detection unit 203, and the value of the image facing the front direction is high. The first evaluation value is obtained as follows. Next, in step 220, a second evaluation value is obtained based on the similarity between the face received from the detection unit 203 and the dictionary of feature point images. Next, the acquired first evaluation value and second evaluation value are synthesized, and the process proceeds to step 208.

  In step 221, the face orientation is estimated from the received face and feature point position information, and the process proceeds to step 210.

(Example of change)
The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof.

(1) Modification 1
In the above embodiment, one reference image is used, but a plurality of images may be used instead. For example, this is the case.

  First, when an image of the first to fifth frames is input as a time series image, a reference image is obtained for each frame.

  Next, when the image of the sixth frame is input, the reference image of the sixth frame is obtained.

  Next, the reference image having the lowest evaluation value (for example, the reference image of the first frame) among the five reference images is compared with the evaluation value of the reference image of the sixth frame.

  Next, when the evaluation value of the reference image of the sixth frame is higher than the evaluation value of the reference image of the first frame, the reference image of the sixth frame and the reference image of the first frame are switched.

  As a result, five reference images are always recorded.

(2) Modification example 2
In the above embodiment, only one target object detection area is detected from the input image. However, the present invention is not limited to this, and a plurality of areas may be detected. For example, in the third embodiment, a plurality of faces are shown in one input image.

(3) Modification 3
Although the human face is used as the target object in the third embodiment, the present invention is not limited to this, and other target objects such as a human (the entire human body) and a vehicle such as an automobile may be used.

It is a block diagram which shows the structure of the tracking image processing apparatus concerning the 1st Embodiment of this invention. It is a flowchart which shows operation | movement of 1st Embodiment. It is a block diagram which shows the structure of the tracking image processing apparatus concerning the 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of 2nd Embodiment. It is a block diagram which shows the structure of the face direction estimation apparatus concerning the 3rd Embodiment of this invention. It is a flowchart which shows operation | movement of 3rd Embodiment. It is a conceptual diagram which shows the traceable face direction range with respect to a front face. It is a conceptual diagram which shows the change of the frame rate of a conventional method and this embodiment. It is a conceptual diagram which shows the behavior of this invention and a conventional method until it acquires an optimal image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,101 ... Tracking image processing apparatus 2,102,202 ... Image input part 3,103,203 ... Detection part 4,104,204 ... Reference image evaluation part 5,105,205 ... Switching unit 6, 106, 206 ... Dictionary storage unit 7, 107, 207 ... Reference image storage unit 8, 108, 208 ... Output unit 109 ... Parameter storage unit 201 ... Face orientation estimation Device 210: Face detection unit 211 ... Feature point detection unit 212 ... Similarity evaluation unit 213 ... Three-dimensional face direction evaluation unit 214 ... Face direction estimation unit 215 ... General face Shape memory

Claims (8)

An image input unit for inputting time-series images;
A dictionary storage unit that stores dictionary information related to a target object that is a detection target;
A reference storage unit for storing a reference image and a reference similarity;
A first corresponding area of the target object is detected from the input image using the dictionary information, a first similarity between the dictionary information and the image of the first corresponding area is obtained, and an image of the first corresponding area and A detection mode execution unit that executes a detection mode for updating the reference image and the reference similarity with the first similarity;
A second target area is detected from the input image using the reference image, a second similarity between the dictionary information and the image of the second corresponding area is obtained, and the obtained second similarity is the reference image. A semi-tracking mode execution unit that executes a semi-tracking mode for updating the reference image and the reference similarity with the second corresponding region and the second similarity when the similarity is higher than
A tracking mode execution unit that executes a tracking mode for detecting the target region using the reference image from the input image;
(1) switch to the detection mode at first, (2) switch to the semi-tracking mode when the first similarity becomes higher than a first threshold during the execution of the detection mode, and (3) the semi-tracking mode When the second similarity is higher than the second threshold higher than the first threshold during the execution of (2), switching to the tracking mode, (4) during the execution of the semi-tracking mode, the second similarity is the A switching unit that continues the semi-tracking mode when it is between a first threshold and the second threshold;
An image processing apparatus. When the first similarity is lower than the first threshold, the switching unit continues the detection mode for the next input image,
The detection mode execution unit detects a next first corresponding area of the target object using the dictionary information from the next input image, and performs the next first from the reference image and the next first corresponding area. The similarity is obtained, and when the next first similarity is higher than the first similarity of the reference image, the next first corresponding region is set as a new reference image. The image processing apparatus according to 1. In addition to the first similarity, the detection mode execution unit uses the dictionary information and the first corresponding area as a direction evaluation value that increases as the target object in the target area faces a specific direction. And using
The semi-tracking mode execution unit obtains the direction evaluation value in addition to the second similarity using the reference image and the second corresponding region,
The switching unit compares the first similarity or the value obtained by adding the direction evaluation value to the second similarity with the first threshold or the second threshold.
The image processing apparatus according to claim 1. In the detection mode and the semi-tracking mode,
Having the reference image for each of the time-series images;
The reference image having the lowest evaluation value among the plurality of reference images is compared with the evaluation value of the newly obtained reference image,
The new reference image and the reference image with the lowest evaluation value are switched when the evaluation value of the new reference image is higher than the evaluation value of the reference image with the lowest evaluation value. Item 6. The image processing apparatus according to Item 1. The image processing apparatus according to claim 1, wherein, in the detection mode and the semi-tracking mode, the calculation cost of the similarity is reduced as the first similarity or the second similarity increases. The target object is a human face;
Obtaining facial part position information from the target area in each input image,
Obtaining three-dimensional face orientation information from the face component position information of each input image;
The image processing apparatus according to claim 3, wherein the direction evaluation value is increased as the face orientation information faces the front. Enter a time-series image,
Stores dictionary information about the target object that is the detection target,
Memorize the standard image and standard similarity,
A first corresponding area of the target object is detected from the input image using the dictionary information, a first similarity between the dictionary information and the image of the first corresponding area is obtained, and an image of the first corresponding area and Executing a detection mode for updating the reference image and the reference similarity with the first similarity;
A second target area is detected from the input image using the reference image, a second similarity between the dictionary information and the image of the second corresponding area is obtained, and the obtained second similarity is the reference image. A semi-tracking mode for updating the reference image and the reference similarity with the second corresponding region and the second similarity when the similarity is higher than
Performing a tracking mode for detecting the target region using the reference image from the input image;
(1) switch to the detection mode at first, (2) switch to the semi-tracking mode when the first similarity becomes higher than a first threshold during the execution of the detection mode, and (3) the semi-tracking mode When the second similarity is higher than the second threshold higher than the first threshold during the execution of (2), switching to the tracking mode, (4) during the execution of the semi-tracking mode, the second similarity is the When in between the first threshold and the second threshold, the quasi-tracking mode is continued;
Image processing method. Image input function to input time-series images,
A dictionary storage function for storing dictionary information relating to a target object to be detected;
A reference storage function for storing a reference image and a reference similarity;
A first corresponding area of the target object is detected from the input image using the dictionary information, a first similarity between the dictionary information and the image of the first corresponding area is obtained, and an image of the first corresponding area and A detection mode execution function for executing a detection mode for updating the reference image and the reference similarity with the first similarity;
A second target area is detected from the input image using the reference image, a second similarity between the dictionary information and the image of the second corresponding area is obtained, and the obtained second similarity is the reference image. A semi-tracking mode execution function for executing a semi-tracking mode for updating the reference image and the reference similarity with the second corresponding region and the second similarity when the similarity is higher than
A tracking mode execution function for executing a tracking mode for detecting the target region using the reference image from the input image;
(1) switch to the detection mode at first, (2) switch to the semi-tracking mode when the first similarity becomes higher than a first threshold during the execution of the detection mode, and (3) the semi-tracking mode When the second similarity is higher than the second threshold higher than the first threshold during the execution of (2), switching to the tracking mode, (4) during the execution of the semi-tracking mode, the second similarity is the A switching function to continue the semi-tracking mode when between the first threshold and the second threshold;
An image processing program that realizes

Images