JP2012105130A - Image processing method and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce as much as possible an operation amount by activating a function that optimizes the operation amount to detect specific area such as a face area based on a picture image condition even in a back-and-forth movement of a subject and an operation of zoom-up and zoom-out.SOLUTION: In an image processing method, a distance calculation step n10 calculates distance information based on image data sequentially inputted. A distance comparison step n20 compares current frame image data with previous frame image data in the respective distance information obtained in the distance calculation step. A parameter setting step n30 for specific area detection sets parameters for the specific area detection based on comparison result of the distance comparison step. A specific area detection step n40 detects the specific area based on the parameters for the specific area detection set in the parameter setting step for the specific area detection.

Description

  The present invention relates to an image processing method and an image processing apparatus for detecting a specific area such as a face area for sequentially input image data. Specifically, the calculation amount required for detection of a specific area is optimized and specified according to the image situation. The present invention relates to a technique for shortening the time required for area detection as much as possible.

  In recent years, an image processing apparatus in an image pickup apparatus such as a digital camera has become more intelligent, and a face area detection function is generally installed. An example of the face area detection method is a template matching method. The outline of template matching in the conventional technique will be described below with reference to FIG.

  As shown in FIG. 17 (a), a feature representing facialness is extracted in advance from a large amount of sample face image data Is as reference face image data It. Next, as shown in FIG. 17 (b), while the partial area is scanned over the entire area of the face area detection target image data Id, the similarity with the reference face image data It is calculated, and the partial area becomes the face area. The face area is detected while determining whether or not it is.

  Further, as shown in FIG. 17C, the face area detection target image data Id is enlarged or reduced, and the same process is performed to detect face areas of various sizes.

  In this prior art, the number of matching operations increases, and the amount of calculation becomes very large.

  FIG. 18 shows a configuration diagram and a conceptual diagram of control of the target image search device disclosed in Patent Document 1 of the prior art. In Patent Document 1, face area detection is performed on image data sequentially input by the face detection unit 21, and coordinate information indicating the face area is output. The output coordinate information is stored in the face detection result storage unit 23. Next, in the detection target area setting unit 22, the face area coordinates C (N-2) and (N-1) frame image data in the (N-2) frame image data accumulated in the face detection result accumulation unit 23 are used. The face area coordinates C (N) in the N frame image data are predicted on the basis of the face area coordinates C (N-1), and the predicted area is set as a detection target range. By calculating the face area coordinates C (N), the calculation amount is optimized in accordance with the image state, and the calculation amount is reduced.

JP 2007-33479A (pages 5-7, FIGS. 1 and 3)

  In the above-described conventional technique, when the subject moves back and forth with respect to the imaging apparatus, or when the photographer performs a zoom-up or zoom-out operation, the specific area (face area) coordinates of the past frame image data are used. It is difficult to predict the specific area coordinates of the current frame image data, and it is difficult to reduce the amount of calculation through optimization of the amount of calculation according to the image situation.

  The present invention has been created in view of such circumstances, and has a function for optimizing the amount of calculation required for detection of a specific area such as a face area according to the image situation. The purpose of this is to make it effective at the time of the operation and to make the calculation amount reduction effect as high as possible.

  The present invention solves the above problems by taking the following measures. Since numbers in parentheses such as << 1 >>, << 2 >>, << 3 >>, etc. correspond to claim numbers in [Claims], they are not necessarily serial numbers in the following description. May fly.

  << 1 >> The image processing method according to the present invention is configured to include a plurality of steps having the following contents. It is at least a distance calculation step, a distance comparison step, a specific region detection parameter setting step, and a specific region detection step.

  In the distance calculation step, distance information is calculated in the sequentially input image data. Here, the distance information is information related to the distance from the imaging device to the subject or a physical quantity corresponding to the distance.

  In the distance comparison step, the distance information obtained in the distance calculation step is compared for each of the current frame image data and the past frame image data. The comparison result may be in any form such as increase / decrease, change amount (difference), and ratio.

  In the specific region detection parameter setting step, the specific region detection parameter is set based on the comparison result in the distance comparison step. The parameter for detecting the specific area is information that defines the size of the specific area to be detected, the detection range, or a physical quantity corresponding to the size, and may have any format.

  In the specific region detection step, the specific region is detected based on the specific region detection parameter set in the specific region detection parameter setting step.

  The effect | action by said structure is as follows. The distance information in the current frame image data is compared with the distance information in the past frame image data, a parameter for detecting a specific area is set according to the comparison result, and the specific area is detected. The distance information, which is information about the distance from the imaging device to the subject or the physical quantity corresponding to the distance, can be calculated quickly and easily as compared with the template matching method and the prediction method of the prior art. This also applies to the image data of each frame when the subject moves back and forth or zooms in and out. In addition, a specific area detection parameter is set from the comparison result of the distance information between the current frame image data and the past frame image data. This setting can also be performed quickly and easily compared to the template matching method and the prediction method. is there.

  Therefore, the specific area based on the specific area detection parameters set by the above-described method of the present invention is a specific area based on the optimum parameters that are not excessive or insufficient in the image status of the current frame image data each time. Setting the specific area detection parameter based on the comparison result of the distance information means that the specific area detection parameter is not set for all sizes but is set for a limited size. In other words, this corresponds to narrowing down the specific area to be detected in that size. This narrowing down makes it possible to greatly reduce the amount of calculation.

  Furthermore, as described above, by setting a specific area with optimal parameters without excess or deficiency, the amount of calculation required for detecting the specific area during the forward / backward movement of the subject or the zoom-up / zoom-out operation is the current frame image data. The optimal amount of computation according to the distance information at.

  Due to the above synergy, according to the present invention, it is possible to avoid the difficulty of imaging due to the excessive amount of calculation, and it is possible to enhance the effect of reducing the time required for detecting a specific area such as a face area.

  <11> In the image processing method configured as described in <1> above, an operation analysis step for analyzing the presence / absence of movement of a specific region, and a branch control step for controlling a branch to the next step based on the analysis result Including image processing methods are also useful. In the above configuration <1>, the basis for setting the parameter for detecting the specific area is the result of comparison of the distance information between the current frame image data and the past frame image data. However, the difference between the distance information in the current frame image data and the distance information in the past frame image data is not necessarily limited to the front-rear movement of the subject and the zoom-up / zoom-out operation. Even when the optical axis of the image pickup apparatus swings left and right or up and down and when the subject moves left and right or up and down, the distance information is different. The reason for setting the parameters for specific area detection from the comparison results of the distance information between the current frame image data and the past frame image data is the calculation required for the specific area detection regardless of the subject's back-and-forth movement and zoom-up / zoom-out operations. This is because the amount is set to an optimum calculation amount corresponding to the distance information in the current frame image data, and the difficulty of imaging due to the excessive calculation amount is avoided. However, if this calculation amount adjustment is applied even when there is a difference in both distance information due to the optical axis shake of the image pickup device or the left / right / upper / lower movement of the subject, it will hinder optimization of the calculation amount. End up. Therefore, in such a case, adjustment is performed so that the function set by the parameter for detecting the specific region is not used from the comparison result of the distance information between the current frame image data and the past frame image data.

  The motion analysis step is set between the distance calculation step and the distance comparison step. In this motion analysis step, it is analyzed whether or not there is a left / right / up / down movement in the specific area based on the distance information in the past frame image data and the distance information in the current frame image data.

  The branching control step proceeds to the distance comparison step when the analysis result in the motion analysis step does not involve the left / right / up / down movement in the specific region, and the analysis result indicates that the specific region has the left / right / up / down motion. The distance comparison step is skipped and the process proceeds to a specific region detection parameter setting step.

  Further, when the analysis result indicates that the specific area is accompanied by a horizontal and vertical movement, the specific area detection parameter setting step sets the specific area detection parameter for all sizes. The setting for all sizes does not mean that the specific area to be detected is limited to that size as in << 1 >>. Here, in determining whether the specific area has left / right / up / down movement, the distance information of the past and current frame image data is used, as in the case of setting the parameter for detecting the specific area in << 1 >>. This is one technical point.

  According to the configuration of this section, when there is a change in the distance information between the current frame image data and the past frame image data, and the cause is the back-and-forth movement of the subject or the zoom-up / zoom-out operation, << 1 >> On the other hand, when there is a horizontal movement of the specific area due to the optical axis shake of the imaging device or the horizontal and vertical movement of the subject, the process of << 1 >> is disabled. Parameters for detecting a specific area are set for all sizes. As a result, it is possible to prevent detection omissions in the case of an optical axis shake of the imaging device or a left / right / up / down movement of the subject.

  The technical contents of this section can also be applied to << 2 >> to << 10 >> described later.

  << 14 >> In the image processing method configured as described in the above << 1 >>, there is another system different from the above << 11 >> in analyzing whether or not the specific area has a horizontal and vertical movement. Here, a motion vector is used instead of the comparison result of the distance information. In this section, the image processing method having the above configuration <1> further includes a motion vector generation step and a motion vector analysis step.

  The motion vector generation step generates motion vector information in the input image data.

  The motion vector analyzing step analyzes whether there is a left / right / up / down motion in the specific area based on the motion vector information. This motion vector analysis step also has a branch control function. If the analysis result indicates that the specific area does not involve left / right / up / down movement, the process proceeds to the distance comparison step. If the analysis result indicates that the specific area includes left / right / up / down movement, the distance comparison step is skipped. The process proceeds to a parameter setting step for detecting a specific area.

  Further, when the analysis result indicates that the specific area is accompanied by a horizontal and vertical movement, the specific area detection parameter setting step sets the specific area detection parameter for all sizes. Setting for all sizes means that the specific area to be detected is not limited and limited in size. Here, different from the above << 1 >> and << 11 >>, one technical point is to use motion vector information in determining whether there is a left / right / up / down movement in a specific area.

  According to the configuration of this section, when there is a change in the motion vector and the cause is the back-and-forth movement of the subject or the zoom-up / zoom-out operation, the processing of << 1 >> is performed, while the cause is However, when there is left / right / up / down movement of the specific area due to the optical axis shake of the image pickup device or the left / right / up / down movement of the subject, the processing of << 1 >> is disabled, and the parameters for detecting the specific area are set to all sizes. It is supposed to be set against. As a result, it is possible to prevent detection omissions in the case of an optical axis shake of the imaging device or a left / right / up / down movement of the subject.

  The technical contents of this section can also be applied to << 2 >> to << 10 >> described later.

  << 21 >> As an image processing apparatus corresponding to the image processing method configured as described in <1> above, a distance calculation unit that calculates distance information in sequentially input image data, and current frame image data and past frame image data, respectively. A distance comparison unit that compares distance information obtained by the distance calculation unit; a specific region detection parameter setting unit that sets a specific region detection parameter based on a comparison result in the distance comparison unit; and the specific region detection An image processing apparatus including a specific area detection unit that detects a specific area based on the specific area detection parameter set in the parameter setting unit is effective.

  According to the image processing apparatus of the present invention having this configuration, it is possible to avoid the difficulty of imaging due to an excessive amount of calculation, and it is possible to improve the effect of reducing the time required to detect a specific area such as a face area. Become.

  << 22 >> As an image processing apparatus corresponding to the image processing method having the above configuration <11>, in the image processing apparatus having the above structure <21>, further, distance information in past frame image data and distance information in current frame image data Based on the motion analysis unit for analyzing whether the specific region has left and right and up and down movement, and if the analysis result in the motion analysis unit is that the specific region does not have a horizontal and vertical movement, the distance comparison unit A branch control unit that operates the specific region detection parameter setting unit after disabling the distance comparison unit when the analysis result is that the specific region is accompanied by left / right / up / down movement. However, in the case where the specific area is accompanied by left / right / up / down movements, the specific area detection parameter setting unit sets the specific area detection parameters for all sizes. Configured image processing apparatus as it is effective.

  According to the image processing apparatus having this configuration, it is possible to prevent omission of detection in the case of an optical axis shake of the imaging apparatus and a left / right / up / down movement of the subject.

<< 23 >> As an image processing apparatus corresponding to the image processing method having the above configuration <14>, in the image processing apparatus having the above structure <21>, a motion vector for generating motion vector information in input image data Based on the motion vector information, the generation unit analyzes whether the specific area has left / right / up / down movement, and if the analysis result indicates that the specific area does not have left / right / up / down movement, operates the distance comparison unit. And a motion vector analysis unit that operates the parameter setting unit for detecting the specific region after the distance comparison unit is inoperable when the specific result is that the specific region is accompanied by left / right / up / down motion. However, in the case where the specific area is accompanied by left / right / up / down movement, the specific area detection parameter setting unit sets the specific area detection parameters for all sizes. Configured image processing apparatus is effective.

  According to the image processing apparatus having this configuration, it is possible to prevent omission of detection in the case of an optical axis shake of the imaging apparatus and a left / right / up / down movement of the subject.

  Further, the image processing methods << 1 >>, << 11 >>, and << 14 >> can be developed as programs for causing a computer to execute each image processing method.

  << 27 >> Further, the image processing apparatuses of the above << 21 >> to << 23 >> are developed as an imaging apparatus having the following configuration. An image sensor that receives subject light incident via an optical lens, converts the light into an image signal, and outputs the image signal; an A / D converter that converts the image signal output from the image sensor into a digital signal; A digital signal processing circuit including an image processing circuit that performs image processing on a digital signal output from the A / D converter, a recording circuit that records image data output from the digital signal processing circuit on a medium, and A reproduction circuit for reproducing a signal, and the image processing circuit according to any one of the above << 21 >> to << 23 >>.

  According to the present invention, the specific area detection parameter is set based on the result of comparing the distance information in the current frame image data with the distance information in the past frame image data, and the size of the specific area such as the face area in the current frame is predicted. In addition, the function of optimizing the amount of calculation required for detecting a specific area such as a face area according to the image state can be validated even when the subject moves back and forth or zooms in and out. That is, the calculation amount reduction effect in detecting a specific region can be made as high as possible while avoiding the difficulty of imaging due to excessive calculation amount.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. 1 is a flowchart (part 1) illustrating an operation (overall flow) of the image processing apparatus according to the first exemplary embodiment of the present invention. 2 is a flowchart showing an initial flow of the image processing apparatus according to the first embodiment of the present invention. Explanatory drawing about the comparison with the distance information in the calculation of the distance to a to-be-photographed object in Example 1 of this invention, and past frame image data Explanatory drawing of the transition from the past frame image data to the present frame image data in Embodiment 1 of the present invention Explanatory drawing of the setting of the parameter for face area detection in the Example 1 of this invention, and the process of face area detection (the 1) Explanatory drawing of the setting of the parameter for face area detection in the Example 1 of this invention and the process of face area detection (the 2) 2 is a flowchart (part 2) illustrating the operation (overall flow) of the image processing apparatus according to the first exemplary embodiment of the present invention. FIG. 3 is a block diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the present invention. 7 is a flowchart showing the operation (overall flow) of the image processing apparatus according to the second embodiment of the present invention. Explanatory drawing of the analysis method of the presence or absence of the right-and-left up-and-down movement in Example 2 of this invention Explanatory drawing of the setting and detection process of a specific area | region when the face area | region in Example 2 of this invention does not accompany a left-right up-down operation. FIG. 6 is a block diagram illustrating a configuration of an image processing apparatus according to a third embodiment of the present invention. 9 is a flowchart showing the operation (overall flow) of the image processing apparatus according to the third embodiment of the present invention. Explanatory drawing of the analysis method of the presence or absence of the right-and-left up-and-down movement in Example 3 of this invention 1 is a block diagram showing a configuration of an imaging apparatus equipped with an image processing apparatus according to an embodiment of the present invention. Schematic diagram of template matching in the prior art Configuration diagram and control conceptual diagram of target image search device disclosed in prior art

  The image processing method of the present invention having the above configuration <1> can be further advantageously developed in the following embodiment.

  << 2 >> In the image processing method configured as described in <1> above, the specific area detection parameter in the specific area detection parameter setting step is a parameter related to the size or detection range of the specific area to be detected. preferable. That is, when the distance information in the current frame image data is larger than the distance information in the past frame image data, the specific area detection parameters are set so as to reduce the size of the specific area or the detection range. Conversely, when the distance information in the current frame image data becomes smaller than the distance information in the past frame image data, the parameters for detecting the specific area are set so that the size of the specific area or the detection range is expanded.

  <3> In the image processing method configured as described in <1>, the distance comparison step includes a mode in which a difference between distance information in current frame image data and distance information in past frame image data is compared with a threshold value. . This is one of the simplest methods for setting parameters for detecting a specific area.

  <4> In the image processing method having the above configuration <3> mentioned in the comparison of the difference between the distance information and the threshold value, the distance comparison step includes the inside or the periphery of the specific area detected in the past frame image data. Comparing the distance information with the distance information at the same position in the current frame image data is one of the preferable modes.

  The position coordinates of the specific area detected in the past frame image data are fixed. On the other hand, in the current frame image data, since the specific area is determined from now on, the position coordinates of the specific area are not yet determined. Therefore, distance information in the current frame image data is obtained at the same position as the inside or the periphery of the specific area detected in the past frame image data whose position coordinates have already been determined. By doing so, it is possible to increase the accuracy of comparison of the distance information of both the past and current frame image data even when the subject moves back and forth or zooms in and out.

  << 5 >> In the image processing method configured as described in <3> above, which refers to the comparison between the difference in distance information and the threshold value, the distance comparison step has priority over the size of the specific area detected based on the comparison result. In the specific area detection parameter setting step, a parameter based on the priority is set, and in the specific area detection step, the priority is set. It is one of preferred embodiments to detect specific areas in order.

  A plurality of enlarged / reduced image data are generated for the input image data. These multiple enlarged / reduced image data are candidates. Now, it is assumed that the specific area is detected from one enlarged / reduced image data among a plurality of enlarged / reduced image data of the past frame image. Candidates are several enlarged / reduced image data continuous with respect to the enlarged / reduced image data in which the specific area is detected. Priorities are assigned to the parameters of some enlarged / reduced image data selected in this way. The priority is determined by narrowing down to some of all candidates, and the specific area is detected from the current frame image data according to the priority order. This reduces the amount of calculation required for detecting the specific area, resulting in excessive calculation amount. It becomes possible to avoid the difficulty of imaging even better.

  << 6 >> In the image processing method configured as described in <3> above, which refers to the comparison between the distance information difference and the threshold value, the specific area detection parameter setting step includes a determination criterion for the distance information difference. When a threshold value P (P> 0) having an appropriate value is determined, and the difference between the distance information is equal to or larger than the threshold value P, a size smaller than the size of the specific region detected in the past frame image data is specified. It is a preferable aspect to set a parameter for detecting a region, and to detect the small-sized specific region in the specific region detecting step.

  If the distance information in the current frame image data is S and the distance information in the past frame image data is R, the difference in distance information is (S−R). The difference (S−R) of the distance information is compared with a threshold value P (P> 0).

0 <P ≦ (S−R)
It can be seen that the distance information in the current frame image data is increasing. In this case, the specific region detection parameters may be set so that the detection target size of the specific region is small.

  << 7 >> In the image processing method configured as described in <3> above, which refers to the comparison between the difference in distance information and the threshold value, the determination criterion for the difference in distance information is set as the specific region detection parameter setting step. A threshold value Q (Q <0) having an appropriate value is set as follows, and when the difference in the distance information is smaller than the threshold value Q, the size of the specified region larger than the size of the specific region detected in the past frame image data is specified. It is a preferable aspect to set a parameter for detecting a region and to detect the large-sized specific region in the specific region detecting step.

  If the distance information in the current frame image data is S and the distance information in the past frame image data is R, the difference in distance information is (S−R). The difference (S−R) in the distance information is compared with a threshold value Q (Q <0).

(S−R) <Q <0
It can be seen that the distance information in the current frame image data is decreasing. In this case, the specific area detection parameters may be set so that the detection target size of the specific area is large.

  <8> In the image processing method having the above configuration <3> mentioned in the comparison between the difference in distance information and the threshold value, the difference in the distance information is the threshold value P ( When P> 0) and a threshold Q (Q <0) or more, a parameter for detecting a specific area having a size equivalent to the size of the specific area detected in the past frame image data is set. In the specific area detecting step, it is one of preferred modes to detect the specific area having the same size.

  If the distance information in the current frame image data is S and the distance information in the past frame image data is R, the difference in distance information is (S−R). The distance information difference (S−R) is compared with a threshold value P (P> 0) and a threshold value Q (Q <0).

Q ≦ (S−R) <P
It can be understood that it is difficult to say that the distance information in the current frame image data is increasing or decreasing. In this case, the specific area detection parameters may be set so that the detection area size of the specific area is equivalent to the detection area size of the specific area in the past frame image data.

  <9> Configuration of the above <6>, in which a parameter for detecting a specific area having a size smaller than the past frame image data is set when the difference in distance information is equal to or greater than a threshold value P (P> 0). In the image processing method, the threshold value P is subdivided into a plurality of threshold values P1 to Pn (n is a positive integer, P1 <P2 <... <Pn). In the specific area detection parameter setting step, parameters for detecting a specific area of a specific size are set according to the magnitude relationship between the difference in the distance information and the subdivided threshold values P1 to Pn. Furthermore, in the specific area detecting step, it is one of preferred embodiments to detect the specific area of the specific size.

  If the distance information in the current frame image data is S and the distance information in the past frame image data is R, the difference in distance information is (S−R). This distance information difference (S−R) is compared with threshold values P1, P2,..., Pn (0 <P1 <P2 <... <Pn).

  When P1 ≦ (S−R) <P2, parameters are set for the enlarged / reduced image data corresponding to the minimum size among the selected candidates, and the detection process is performed.

  When P2 ≦ (S−R) <P3, a parameter is set for the enlarged / reduced image data corresponding to the next largest size among the selected candidates, and the detection process is performed.

  When Pn−1 ≦ (S−R) <Pn, a parameter is set for the enlarged / reduced image data corresponding to the next smallest size among the selected candidates, and the detection process is performed. To do.

  When Pn ≦ (S−R), parameters are set for the enlarged / reduced image data corresponding to the maximum size among the selected candidates, and the detection process is performed.

  Since fine parameter setting is performed in this way, it is possible to increase the accuracy of detection of a specific area even when the subject moves back and forth or zooms in and out.

  <10> Configuration of the above <7>, in which a parameter for detecting a specific area having a size larger than the past frame image data is set when the difference in the distance information becomes smaller than the threshold value Q (Q <0). In the image processing method, the threshold value Q is subdivided into a plurality of threshold values Q1 to Qn (n is a positive integer, Qn <... <Q2 <Q1). In the specific area detection parameter setting step, parameters for detecting a specific area of a specific size are set according to the magnitude relationship between the difference in the distance information and the subdivided threshold values Q1 to Qn. Furthermore, in the specific area detecting step, it is one of preferred embodiments to detect the specific area of the specific size.

  If the distance information in the current frame image data is S and the distance information in the past frame image data is R, the difference in distance information is (S−R). The distance information difference (S−R) is compared with threshold values Q1, Q2,..., Qn (Qn <... <Q2 <Q1 <0).

  When Q2 ≦ (S−R) <Q1, parameters are set for the enlarged / reduced image data corresponding to the maximum size among the selected candidates, and the detection process is performed.

  When Q3 ≦ (S−R) <Q2, parameters are set for the enlarged / reduced image data corresponding to the next smallest size among the selected candidates, and the detection process is performed.

  When Qn ≦ (S−R) <Qn−1, a parameter is set for the enlarged / reduced image data corresponding to the next largest size among the selected candidates, and the detection process is performed. To do.

  When (S−R) <Qn, parameters are set for the enlarged / reduced image data corresponding to the minimum size among the selected candidates, and the detection process is performed.

  Since fine parameter setting is performed in this way, it is possible to increase the accuracy of detection of a specific area even when the subject moves back and forth or zooms in and out.

  <12> In the image processing method having the above-described configuration <11> referring to the motion analysis step, the motion analysis step includes distance information inside or around a specific area detected from past frame image data and the current frame image. It is one of the preferable modes to analyze whether or not there is a left / right / up / down movement in the specific area based on the distance information of the same position in the data.

  The position coordinates of the specific area detected in the past frame image data are fixed. On the other hand, in the current frame image data, since the specific area is determined from now on, the position coordinates of the specific area are not yet determined. Therefore, the distance information in the current frame image data is obtained at the same position as the inside or the periphery of the specific area detected in the past frame image data whose position coordinates have already been determined, and the specific area is affected by the distance information. Let us analyze whether there is vertical movement. By doing so, it is possible to make the determination of the left / right / up / down movement of the specific region due to the optical axis shake of the imaging apparatus and the left / right / up / down movement of the subject with high accuracy.

  << 13 >> In the image processing method configured as described in <12> above, the motion analysis step includes the distance information of the inside or the periphery of the specific area detected in the past frame image data and the distance at the same position in the current frame image data. It is one of the preferable modes to calculate the magnitude relationship with the information, and to determine that the specific area is accompanied by the horizontal and vertical movements when one or more different magnitude relationships appear.

  When there is left / right / up / down movement of the specific area due to the optical axis shake of the image pickup device or the left / right / up / down movement of the subject, the difference between the distance information in the current frame image data and the distance information in the past frame image data The sign of positive / negative is reversed, and the magnitude relationship between the two distance information becomes irregular. If such irregularities appear, it is determined that the specific area is accompanied by a horizontal and vertical movement, and the processing of << 1 >> is disabled, and the specific area detection parameters are set for all sizes. I decided to. According to this, it is possible to make the determination of the left / right / up / down movement of the specific region highly accurate.

  <15> In the image processing method having the above configuration <14>, in which a motion vector is used to analyze whether there is a left / right / up / down motion in a specific region, the motion vector analysis step is detected from past frame image data There is a mode in which it is analyzed whether or not there is a horizontal and vertical movement in the specific area based on the motion vector in the current frame image data at the same position as or inside the specific area.

  The position coordinates of the specific area detected in the past frame image data are fixed. On the other hand, in the current frame image data, since the specific area is determined from now on, the position coordinates of the specific area are not yet determined. Therefore, distance information in the current frame image data is obtained at the same position as the inside or the periphery of the specific area detected in the past frame image data whose position coordinates have already been determined. By doing so, it is possible to increase the accuracy of comparison of the distance information of both the past and current frame image data even when the subject moves back and forth or zooms in and out.

  <16> In the image processing method configured as described in <15> above, in the motion vector analysis step, the motion vector in the input image data at the same position as the inside or the periphery of the specific area detected in the past frame image data There is a mode in which a total value of information is calculated, and it is determined that the specific area is accompanied by a vertical movement when the total value is equal to or greater than a threshold value.

  When the specific area is accompanied by left / right / up / down movement, the motion vector information is not regular, and the total value thereof is relatively large. In such a case, it is determined that there is a left / right / up / down motion, and a parameter for detecting a specific area of all sizes is set.

  On the other hand, the specific area does not move left / right / up / down during the subject's back-and-forth movement or zoom-up / zoom-out operation, but in such a case, the motion vector information is regularly calculated and the total value of the vectors Becomes relatively small. In such a case, it is determined that there is no left / right / up / down motion, and a parameter for detecting a specific area with a limited size is set.

  <17> In the image processing method configured as described in <11>, which refers to the motion analysis step, when the motion analysis step determines that the specific region does not move left and right and up and down, the specific region detection parameter is set. In the step, the specific area detection parameter is set for the size of the specific area to be detected.

  <18> In the image processing method having the above configuration <14> that refers to the motion vector analysis step, when it is determined in the motion vector analysis step that the specific region does not move left and right and up and down, the specific region detection parameter In the setting step, the specific area detection parameter is set for the size of the specific area to be detected.

  << 19 >> In the image processing method configured as described in the above << 1 >> to << 18 >>, it is typical to set a face area as the specific area. However, in the present invention, the specific area is not limited to the face area, but can be widely applied to areas of any form.

  <20> In the image processing method configured as described in << 1 >> to << 19 >>, parallax information may be set as the distance information. The parallax between two images obtained using two imaging elements changes according to the distance between the subject and the imaging device. Using this property, it is possible to convert the parallax into a distance.

  Embodiments of an image processing method and an image processing apparatus according to the present invention will be described below with reference to the drawings. Hereinafter, the specific area will be described as a face area.

Example 1
FIG. 16 is a block diagram showing a configuration of an image pickup apparatus (for example, a digital still camera) equipped with an image processing apparatus according to an embodiment of the present invention.

  In FIG. 16, 1 is an optical lens, 2 is an image sensor such as a CMOS image sensor that receives subject light incident through the optical lens 1, converts it into an image signal, and outputs it, and 3 is output from the image sensor 2. An A / D conversion unit 4 that converts an imaging signal into a digital signal, 4 is a digital signal processing circuit that performs image processing on the digital signal output from the A / D conversion unit 3, and 5 is provided in the digital signal processing circuit 4. 6 is a recording circuit for recording image data output from the digital signal processing circuit 4 on a medium, 7 is a reproduction circuit for reproducing a signal recorded in the recording circuit 6, 8 is an image sensor 2, and A / D. A timing control circuit that controls the timing of driving the conversion unit 3 and the digital signal processing circuit 4, and a system control circuit 9 that controls the entire imaging apparatus CA.

  Next, the operation of the imaging apparatus CA in FIG. 16 will be described. The image light incident through the optical lens 1 is imaged on the image sensor 2. The imaging device 2 is driven by the timing control circuit 8 to accumulate the imaged image light and photoelectrically convert it into an electrical signal. The electrical signal read from the image sensor 2 is converted into a digital signal by the A / D converter 3 and then input to the digital signal processing circuit 4 including the image processing circuit 5. The digital signal processing circuit 4 performs image processing such as Y / C processing, edge processing, image data enlargement / reduction, and image compression / expansion processing according to the present invention. The image-processed signal is recorded on a medium in the recording circuit 6. The recorded signal is reproduced by the reproduction circuit 7.

  FIG. 1 is a block diagram showing the configuration of the image processing apparatus in Embodiment 1 of the present invention. This image processing apparatus corresponds to the image processing circuit 5 in FIG. Note that FIG. 16 is applied mutatis mutandis when the image processing circuit 5 has any of the functions of the image processing methods of Embodiments 2 and 3 described later.

  In FIG. 1, 10 is a distance calculation unit that calculates distance information in sequentially input image data, and 20 is a distance that compares distance information obtained by the distance calculation unit 10 for current frame image data and past frame image data. The comparison unit 30 is a specific region detection parameter setting unit for setting a specific region detection parameter based on the comparison result in the distance comparison unit 20, and 40 is a specific region detection parameter set in the specific region detection parameter setting unit 30. It is a specific area detection part which detects a specific area based on these parameters.

  FIG. 2 is a flowchart showing the operation (overall flow) of the image processing apparatus in Embodiment 1 of the present invention. Hereinafter, the operation of the image processing apparatus will be described. The specific area below is a face area.

  In step n10, the distance from the imaging device CA to the subject in the sequentially input image data data is calculated (distance calculation step). This operation is executed in the distance calculation unit 10. The distance calculation unit 10 accompanies a storage unit for accumulating input image data and accumulating calculated distance information. This storage unit is generally placed outside, but may be inside.

  Next, in step n20, the distance information in the current frame image data is compared with the distance information in the past frame image data (distance comparison step). This operation is executed in the distance comparison unit 20. The distance comparison unit 20 simultaneously reads the distance information in the current frame image data and the distance information in the past frame image data from the storage unit, compares the two, and compares the comparison results with the parameter setting unit 30 for detecting a specific region. To pass. The aspect of comparison will be described later.

  Next, in step n30, parameters used in specific area detection are set based on the comparison result in the distance comparison step n20 (specific area detection parameter setting step). This operation is executed in the specific area detection parameter setting unit 30. The mode of the parameter for detecting the specific area will be described later.

  Next, in step n40, the specific area is detected based on the parameters set in the specific area detection parameter setting step n30 (specific area detection step). This operation is executed in the specific area detection unit 40.

  Here, as a method for calculating the distance from the imaging device to the subject in the present invention, two methods will be described as examples.

  One is an active method in which a subject is irradiated with laser light, electromagnetic waves, or the like, and the distance to the subject is calculated based on the time delay of the reflected light.

  The other is a passive method in which a correlation between two image data obtained using two image sensors is calculated, and a distance is calculated based on parallax information between the two image data.

  In both methods, it is possible to calculate distance information in block units or pixel units obtained by dividing image data into N × M blocks (N, M: positive integers).

  Although two distance calculation methods have been described as examples, the distance calculation method in the present invention is not limited to these.

  FIG. 3 shows an initial flow in the first embodiment. If the specific area detection process is performed on the sequentially input image data data in the specific area detection step n2, and it is determined that the specific area (face area) is detected in the specific area detection presence / absence determination step n4, the distance calculation is performed. In step n6, the distance to the subject is calculated, and the detection result is output. Further, the flow shown in FIG. 2 is performed on the image data of the subsequent frames. If it is determined that the specific area (face area) is not detected, the initial flow process is performed again.

  The reason why the initial flow of FIG. 3 is set prior to the flow of FIG. 2 is that distance information in the past frame image data is required for distance information in the current frame image data in the distance comparison step n20 of FIG. .

  The calculation of the distance to the subject in the distance calculation step n10 and the comparison with the distance information in the past frame image data in the distance comparison step n20 will be described with reference to FIG. Iin is input image data. A typical example of the specific area A is a face area.

  As for the position in the image data to be compared in the comparison of distance information, the block or pixel in or around the specific area A detected in the past frame may be compared. That is, in the distance calculation step n10, the distance information is calculated in block unit B0 when the input image data is divided into N × M blocks (N, M: positive integers). In the distance comparison step n20, the distance information in the block Bc (the block including the center coordinates CC of the face area) inside the specific area A detected from the past frame image data is compared.

Next, an example of the input image data shown in FIG. 5A will be described with reference to FIG. FIG. 5A shows a case where the specific area of the current frame image data F is larger than the specific area detected in the past frame image data F ⁻ .

Previous frame image data F ^- distance information in R1, compares the current distance information in the frame image data F as S1 (S1 <R1), the distance difference of (S1-R1) a predetermined threshold value Q and (Q <0).

In the case of (S1-R1) <Q, this means that the imaging device CA and the subject are approaching, and it is assumed that the specific area size becomes large. In this case, a parameter for detecting a specific region having a size larger than the specific region size detected in the past frame image data F ⁻ is set in the specific region detection parameter setting step n30, and in the specific region detection step n40. The detection process is performed according to the set parameters. Here, the specific area detection parameter is a parameter related to the size or detection range of the specific area to be detected.

  Next, the above-described parameter setting for specific area detection and specific area detection processing will be described with reference to FIG.

In the specific area detection step n40, the input image data is enlarged / reduced i times (i: positive integer), and each enlarged / reduced image data is represented by I _{0 to} I _i−1 .

Past frame F ^- If the specific area in the enlarged or reduced image data I _i-4 in is detected, for the current frame image data F, in the setting step n30 of the specific area detection parameter, the specific area is detected A parameter for executing the specific area detection process is set for a plurality of enlarged / reduced image data I _i-3 , I _i-2 , I _i-1 continuous to the enlarged / reduced image data I _i-4 . Furthermore, in the specific area detection step n40, detection processing is performed according to the set parameters.

Regarding the order of the enlarged / reduced image data to be subjected to the detection process, priority is determined as follows for a plurality of continuous enlarged / reduced image data I _i-3 , I _i-2 , I _i-1 . Also good.

Compared with the distance difference (S1-R1) and a predetermined threshold value X (X <Q), if (S1-R1) <X, this means that the imaging device CA and the subject are approaching, and the approach speed is compared. Therefore, detection processing is performed in the order of enlarged / reduced image data I _i-1 → I _i-2 → I _i-3 .

On the other hand, in the case of X <(S1-R1), this means that the imaging apparatus CA is approaching and the subject is approaching and the approaching speed is relatively slow. _The detection process is performed in the order of _i-3 → I _i-2 → I _i-1 .

  In the above processing, the detection processing is not performed for all the enlarged / reduced image data, but only for a part of the detection processing, so that the calculation amount of the specific area detection processing is reduced. can do.

  The detection process is performed in accordance with the priority order of the enlarged / reduced image data described above. If a specific area is detected during the process, the remaining processes are terminated. On the other hand, when a specific area is not detected in the enlarged / reduced image data that is the detection target, the detection process is performed on the enlarged / reduced image data that is not the detection target.

Next, an example of the input image data shown in FIG. 5B will be described with reference to FIG. FIG. 5B shows a case where the specific area of the current frame image data F is smaller than the specific area detected in the past frame image data F ⁻ .

In the distance calculation step n10, the distance to the subject is calculated with respect to the input current frame image data F, and in the distance comparison step n20, comparison with the distance information in the past frame image data F ⁻ is performed. The position in the image data to be compared in the distance information comparison has already been described with reference to FIG.

The distance information in the past frame image data F ⁻ is R2, the distance information in the current frame image data F is S2 (R2 <S2), and the distance difference (S2−R2) is compared with a predetermined threshold P (P> 0).

In the case of P ≦ (S2-R2), this means that the imaging apparatus CA and the subject are being separated from each other, and it is assumed that the specific area size becomes small. In this case, in the setting step n30 of the specific region detection parameters, past frame image data F ^- set the parameters for detecting a specific area of a size smaller than the detected specific region size, in the specific region detection step n40 The detection process is performed according to the set parameters.

The parameter setting for specific area detection and the specific area detection process have already been described with reference to FIG. 5A and will not be described in detail. However, the enlarged / reduced image data I _{0 to} I _i-5 are not described in detail. The same processing is performed.

  In the above processing, the detection processing is not performed for all the enlarged / reduced image data, but only for a part of the detection processing, so that the calculation amount of the specific area detection processing is reduced. can do.

Next, an example of the input image data shown in FIG. 5C will be described with reference to FIG. FIG. 5C shows a case where the specific area detected in the past frame image data F ⁻ is equivalent to the specific area of the current frame image data F −.

In the distance calculation step n10, the distance to the subject is calculated with respect to the input current frame image data F, and in the distance comparison step n20, comparison with the distance information in the past frame image data F ⁻ is performed. The position in the image data to be compared in the distance information comparison has already been described with reference to FIG.

Previous frame image data F ^- distance information in R3, the current distance information in the frame image data F as S3 (R3 = S3), the distance difference (S3-R3) a predetermined threshold value P, Q (P> 0, Q < Compare with 0).

In the case of (S3-R3) <P and Q ≦ (S3-R3), this means that the imaging apparatus CA and the subject are hardly approaching or separating from each other, and it is assumed that the specific area size is not substantially changed. The In this case, in the setting step n30 of the specific region detection parameters, past frame image data F ^- set the parameters for detecting the specific area of the specific region size detected in the same size, in the specific region detection step n40 The detection process is performed according to the set parameters.

For the process of setting and the specific region detection parameters for a particular region detection, omitted detail because you have already described in FIGS. 5 (a) is, past frame F ^- scaling the specific area in is detected A similar process is performed on the image data I _i-4 .

  In addition, in order to allow the calculation error in the distance calculation step n10, the detection process may be performed on the front and rear enlarged / reduced image data.

  In the above processing, the detection processing is not performed for all the enlarged / reduced image data, but only for a part of the enlarged / reduced image data, so that the calculation amount of the specific area detection processing is reduced. can do.

  Next, a method for setting parameters in more detail than the setting method of FIG. 6 will be described with reference to FIG.

If the image data is input as shown in FIG. 5 (a), in the distance calculation step n10, the current is calculated the distance to the object with respect to the frame image data F, at a distance comparing step n20, the previous frame image data F ^- in Compare with distance information. The position in the image data to be compared in the distance information comparison has already been described with reference to FIG.

Previous frame image data F ^- distance information in R1, the distance information currently in the frame image data F S1 (S1 <R1), the distance difference (S1-R1) with a predetermined threshold value Q1, Q2, ..., Qn ( Qn < ... <Q2 <Q1 <0) are compared with each other.

In the case of Q2 ≦ (S1-R1) <Q1 <0, this means that the imaging apparatus CA and the subject are approaching and the approaching speed is relatively slow, and it is assumed that the specific area size gradually increases. Is done. In this case, a parameter for executing the specific area detection process is set for the enlarged / reduced image data I _i-3 in the specific area detection parameter setting step n30, and detection is performed according to the set parameter in the specific area detection step n40. Perform the process.

In the case of Q3 ≦ (S1-R1) <Q2, this means that the imaging apparatus CA and the subject are approaching and the approaching speed is in an intermediate state, and it is assumed that the specific area size increases at the intermediate speed. Is done. In this case, a parameter for executing the specific area detection process is set for the enlarged / reduced image data I _i-2 in the specific area detection parameter setting step n30, and detection is performed according to the set parameter in the specific area detection step n40. Perform the process.

In the case of (S1-R1) <Q3, this means that the imaging device CA and the subject are approaching and the approaching speed is relatively fast, and it is assumed that the specific area size increases rapidly. In this case, a parameter for executing the specific area detection process is set for the enlarged / reduced image data I _i-1 in the specific area detection parameter setting step n30, and detection is performed according to the set parameter in the specific area detection step n40. Perform the process.

  A plurality of enlarged / reduced image data may be assigned to one condition.

Similarly, when the image data shown in FIG. 5B is input, the distance information in the past frame image data F ⁻ is R2, and the distance information in the current frame image data F is S2 (R2 <S2). The distance difference (S2-R2) and predetermined threshold values P1 to Pn (0 <P1 <P2 <... <Pn) are respectively compared, and in the specific region detection parameter setting step n30, enlargement / reduction corresponding to each condition is performed. A parameter for executing the specific area detection process is set for the image data, and the detection process is executed according to the set parameter in the specific area detection step n40.

  In addition, it is also possible to replace the distance information in this embodiment with disparity information as in the flow shown in FIG.

(Example 2)
The second embodiment of the present invention relates to an image processing method in which the presence / absence determination of the left / right / up / down motion of a specific area is further introduced in the first embodiment. The presence / absence determination of the left / right up / down motion of the specific area is performed based on distance information in the current frame image data and the past frame image data data.

  FIG. 9 is a block diagram showing the configuration of the image processing apparatus in Embodiment 2 of the present invention. This image processing apparatus corresponds to the image processing circuit 5 in FIG. In FIG. 9, the same reference numerals as those in FIG. 1 of the first embodiment indicate the same components, and thus detailed description thereof is omitted. Reference numeral 12 denotes an operation analysis unit that analyzes whether there is a left / right / up / down movement in a specific area based on the distance information in the past frame image data obtained by the distance calculation unit 10 and the distance information in the current frame image data. 14, the distance comparison unit 20 is operated when the analysis result in the motion analysis unit 12 indicates that the specific region does not involve horizontal and vertical movements, and the distance comparison is performed when the analysis result indicates that the specific region includes horizontal and vertical movements. The branch control unit operates the specific region detection parameter setting unit 30a with the unit 20 inoperative. The specific area detection parameter setting unit 30a is configured to set the specific area detection parameters for all sizes when the analysis result indicates that the specific area is accompanied by a horizontal and vertical movement.

  FIG. 10 is a flow executed by the image processing apparatus of FIG. 9 and shows a flow in which determination of presence / absence of left / right up / down motion of a specific region is introduced based on distance information in current frame image data and past frame image data data. An embodiment for the input image data shown in FIG. 5D will be described with reference to FIG.

FIG. 5D shows a case where the specific area has left and right up and down movements in the past frame image data F ⁻ and the current frame image data F.

  In the distance calculation step n10, the distance to the subject is calculated with respect to the input current frame image data F.

Next, at step n12, the presence / absence of the left / right up / down motion is analyzed based on the distance information in the past frame image data F- ^and the distance information in the current frame image data F (operation analysis step). This operation is executed in the operation analysis unit 12. The motion analysis unit 12 simultaneously reads the distance information in the current frame image data and the distance information in the past frame image data from the storage unit in the distance calculation unit 10, analyzes the presence / absence of the left / right up / down motion, and branches the analysis result It passes to the control unit 14.

  Next, at step n14, when the analysis result at the motion analysis step n12 indicates that the specific area does not move left and right and up and down, the process proceeds to the distance comparison step n20, and when the analysis result indicates that the specific area includes left and right and up and down movement. The distance comparison step n20 is skipped and the process proceeds to a specific region detection parameter setting step n30a (branch control step). This operation is executed in the branch control unit 14.

  In the subsequent specific area detection parameter setting step n30a, the same processing as in the first embodiment is executed in the flow from step n20, and the specific area of all sizes is detected in the flow from step S14. Set the parameters. The latter operation is executed in the specific area detection parameter setting unit 30a.

  Next, in a specific area detection step n40, detection processing is performed according to the set parameters.

  Here, a method for analyzing the presence / absence of left / right up / down motion will be described with reference to FIG.

  Similar to the first embodiment, the distance information is calculated in units of blocks when the input image data is divided into N × M blocks (N, M: positive integers) in the distance calculation step n10.

In operation analysis step n12, the previous frame image data F ^- may be compared the distance information of the peripheral blocks or interior blocks of the distance information and the block of the current the same position in the frame image data F in a specific area in each, but in this embodiment Peripheral blocks (shaded blocks in FIG. 11) are respectively compared, and the magnitude relationship is calculated.

In FIG. 11 (a), the previous frame image data F ^- both more current frame image data F of the distance information is smaller blocks B2 ^- more current frame image data F of the distance information is larger blocks B1 and past frame image data F Existing. Therefore, it is determined in the branch control step n14 that there is a left / right up / down motion, and a parameter for detecting specific regions of all sizes is set in a specific region detection parameter setting step n30a.

In FIG. 11 (b), only the block B1 in which the distance information of the current frame image data F is larger than the past frame image data F ⁻ is present. Therefore, it is determined that there is no left / right / up / down motion in the motion analysis step n12, the process branches from the branch control step n14 to step n20, and thereafter the same processing as in the first embodiment is performed.

  By performing the above processing, it is possible to prevent omission of detection even if the specific area is accompanied by a left / right / up / down motion.

Also, as shown in FIG. 12, when the specific area does not involve left / right and up / down movements, the specific area A ^* to be detected in the current frame image data F is set by the specific area detection parameter, and in the specific area detection step n40. By performing the detection process according to the set parameters, it is possible to reduce the calculation amount of the specific area detection process. The specific area A ^* to be detected is desirably set based on the prediction by predicting the specific area size in the current frame image data F from the enlarged / reduced image data described in the first embodiment. Previous frame image data F ^- If the specific area currently in the frame image data F from the specific region in the larger is the same.

(Example 3)
The third embodiment of the present invention relates to an image processing method in which the presence / absence determination of the left / right / up / down motion of the specific region based on the motion vector information is further introduced in the first embodiment.

  FIG. 13 is a block diagram showing the configuration of the image processing apparatus in Embodiment 3 of the present invention. This image processing apparatus corresponds to the image processing circuit 5 in FIG. In FIG. 13, the same reference numerals as those in FIG. 1 of the first embodiment indicate the same components, and thus detailed description thereof is omitted. Reference numeral 16 denotes a motion vector generation unit that generates motion vector information in the input image data. 18 analyzes whether or not the specific area has left and right and up and down movement based on the motion vector information by the motion vector generation unit 16, and if the analysis result indicates that the specific area does not have left and right and up and down movement, the distance When the comparison unit 20 is operated and the analysis result indicates that the specific region is accompanied by left and right and up-and-down movements, the motion vector analysis unit that operates the specific region detection parameter setting unit 30a after disabling the distance comparison unit 20 ( With branch control function). The specific area detection parameter setting unit 30a is configured to set the specific area detection parameters for all sizes when the analysis result indicates that the specific area is accompanied by a horizontal and vertical movement.

  FIG. 14 is a flow executed by the image processing apparatus of FIG. 13 and introduces a flow in which determination of presence / absence of left / right / up / down motion of a specific region based on motion vector information is introduced. An embodiment for the input image data shown in FIG. 5D will be described with reference to FIG.

  In the distance calculation step n10, the distance to the subject is calculated with respect to the input current frame image data F.

  Next, in step n16, a motion vector is calculated (motion vector generation step). This operation is executed in the motion vector generator 16.

  The motion vector information is calculated in units of blocks when the input image data is divided into N × M blocks (N, M: positive integers).

  Next, in step n18, the presence / absence of left / right / up / down motion is analyzed based on the motion vector information. If the analysis result is that the specific region does not involve left / right / up / down motion, the process proceeds to distance comparison step n20. If it is accompanied by left / right / up / down movement, the distance comparison step n20 is skipped and the process proceeds to a specific region detection parameter setting step n30a (motion vector analysis step). This operation is executed in the motion vector analysis unit 18.

  In the subsequent specific region detection parameter setting step n30a, the same processing as in the first embodiment is executed in the flow from step n20, and in the flow from step S18, all the sizes are processed as in the second embodiment. Set parameters for detecting specific areas. The latter operation is executed in the specific area detection parameter setting unit 30a.

  Next, in a specific area detection step n40, detection processing is performed according to the set parameters.

  Here, a method for analyzing the presence / absence of left / right up / down motion will be described with reference to FIG.

In the motion vector analysis step n18, the motion vector information of the peripheral block or the internal block of the specific area in the past frame image data F ⁻ may be analyzed. In this embodiment, the motion vector of the peripheral block (arrow block in FIG. 15) is analyzed. To analyze. In the analysis method, a total value of motion vector information in the peripheral blocks is calculated and compared with a predetermined threshold MV.

  In FIG. 15A, since the motion vector information of the peripheral blocks is not regular and the total value of the vectors is equal to or greater than a predetermined threshold MV, it is determined that there is a left / right up / down motion, and a specific region detection parameter setting step In n30a, parameters for detecting specific areas of all sizes are set.

  In FIG. 15B, the motion vector information of the neighboring blocks is regularly calculated, and the total value of the vectors is smaller than a predetermined threshold MV, so that it is determined that there is no left / right up / down motion. Perform the process.

  By performing the above processing, it is possible to prevent omission of detection even if the specific region is accompanied by up / down and up / down movements with respect to the imaging apparatus CA.

Similarly to the second embodiment, as shown in FIG. 12, when the specific area does not move left and right and up and down, the specific area A ^* to be detected in the current frame image data F is set by the specific area detection parameter. In the specific area detecting step n40, the amount of calculation of the specific area detecting process can be reduced by performing the detecting process according to the set parameters. The specific area A ^* to be detected is preferably set based on the prediction by predicting the specific area size in the current frame image data F from the enlarged / reduced image data described in the first embodiment. Previous frame image data F ^- If the specific area currently in the frame image data F from the specific region in the larger is the same.

  The image processing in the image processing circuit 5 is not necessarily applied only to a signal based on the image light imaged on the image pickup device 2 via the optical lens 1, and is input as an electrical signal from an external device, for example. Needless to say, the present invention is also applicable when processing an incoming image data signal.

  The first to third embodiments described above are merely examples, and various modifications can be made thereto.

  The present invention is a digital camera (digital still camera, digital video camera, camera-equipped mobile phone, etc.), a network camera, a surveillance camera, a TV built-in camera, an interphone camera, and other intelligent cameras having a detection function for a specific area such as a face area. This imaging device is useful for achieving the effect of reducing the amount of computation in detecting a specific area, while avoiding the difficulty of imaging due to the excessive amount of computation, even when the subject moves back and forth or zooms in and out. It is.

n4 Specific region detection presence / absence determination step n6, n10 Distance calculation step n10a Parallax calculation step n12 Motion analysis step n14 Branch control step n16 Motion vector generation step n18 Motion vector analysis step n20 Distance comparison step n20a Parallax comparison step n30 Specific region detection parameter Setting step n40 Specific area detection step CA imaging device 1 optical lens 2 imaging device 3 A / D conversion unit 4 digital signal processing circuit 5 image processing circuit 10 distance calculation unit 12 motion analysis unit 14 branch control unit 16 motion vector generation unit 18 motion Vector analysis unit 20 Distance comparison unit 30, 30a Parameter setting unit for specific region detection 40 Specific region detection unit

Claims (27)

A distance calculating step for calculating distance information in sequentially input image data;
A distance comparison step for comparing the distance information obtained in the distance calculation step for each of the current frame image data and the past frame image data;
A specific region detection parameter setting step for setting a specific region detection parameter based on the comparison result in the distance comparison step;
An image processing method comprising: a specific area detecting step of detecting a specific area based on the specific area detecting parameter set in the specific area detecting parameter setting step.   The image processing method according to claim 1, wherein the specific area detection parameter in the specific area detection parameter setting step is a parameter related to a size or a detection range of the specific area to be detected.   The image processing method according to claim 1, wherein in the distance comparison step, a difference between distance information in the current frame image data and distance information in the past frame image data is compared with a threshold value.   4. The image processing method according to claim 3, wherein in the distance comparison step, distance information in or around a specific area detected from past frame image data is compared with distance information at the same position in the current frame image data.   In the distance comparison step, priority is given to the size of the specific area to be detected based on the comparison result, and in the specific area detection parameter setting step, a parameter based on the priority is set, and the specific area is detected. The image processing method according to claim 3, wherein in the region detection step, specific regions are detected in the order of priority.   In the specific region detection parameter setting step, when the difference in the distance information is greater than or equal to a threshold value P (P> 0), the specific region having a size smaller than the size of the specific region detected in the past frame image data The image processing method according to claim 3, wherein a parameter for detecting an image is set, and the specific region of the small size is detected in the specific region detection step.   In the specific region detection parameter setting step, when the difference in the distance information is smaller than a threshold value Q (Q <0), the specific region having a size larger than the size of the specific region detected in the past frame image data 4. The image processing method according to claim 3, wherein a parameter for detecting an image is set, and the large specific area is detected in the specific area detecting step.   In the specific region detection parameter setting step, when the difference in the distance information is smaller than a threshold value P (P> 0) and greater than or equal to a threshold value Q (Q <0), the specific region detected in the past frame image data 4. The image processing method according to claim 3, wherein a parameter for detecting a specific area having a size equivalent to the size of the area is set, and the specific area having the same size is detected in the specific area detecting step.   The threshold value P is subdivided into a plurality of threshold values P1 to Pn (n is a positive integer, P1 <P2 <... <Pn), and in the specific region detection parameter setting step, the difference between the distance information and The parameter for detecting a specific area of a specific size is set according to the size relationship with the subdivided thresholds P1 to Pn, and the specific area of the specific size is detected in the specific area detection step. An image processing method described in 1.   The threshold value Q is subdivided into a plurality of threshold values Q1 to Qn (n is a positive integer, Qn <... <Q2 <Q1). In the specific region detection parameter setting step, the difference between the distance information and 8. A parameter for detecting a specific area of a specific size is set according to the size relationship with the subdivided thresholds Q1 to Qn, and the specific area of the specific size is detected in the specific area detecting step. An image processing method described in 1. Between the distance calculating step and the distance comparing step,
An operation analysis step for analyzing whether there is a left-right up-down movement in the specific area based on the distance information in the past frame image data and the distance information in the current frame image data;
If the analysis result in the motion analysis step indicates that the specific area does not move left and right and up and down, the process proceeds to the distance comparison step. If the analysis result indicates that the specific area includes movement left and right and up and down, the distance comparison step is performed. A branch control step skipping and proceeding to the step of setting the specific area detection parameter,
11. The specific region detection parameter is set for all sizes in the specific region detection parameter setting step when the analysis result indicates that the specific region is accompanied by horizontal and vertical movements. The image processing method according to any one of the above.   In the motion analysis step, whether or not the specific area has a horizontal and vertical movement based on distance information inside or around the specific area detected in the past frame image data and distance information at the same position in the current frame image data. The image processing method according to claim 11, wherein the image is analyzed.   In the motion analysis step, the magnitude relationship between the distance information in or around the specific area detected in the past frame image data and the distance information at the same position in the current frame image data is calculated, and even one or more different magnitude relationships The image processing method according to claim 12, wherein when the image appears, it is determined that the specific area is accompanied by a horizontal and vertical movement. Between the distance calculating step and the distance comparing step,
A motion vector generation step for generating motion vector information in the input image data;
Based on the motion vector information, it is analyzed whether the specific area has left / right / up / down movement, and if the analysis result is that the specific area does not have left / right / up / down movement, the process proceeds to the distance comparison step. A motion vector analysis step that skips the distance comparison step and proceeds to the parameter setting step for the specific region detection when the specific region is accompanied by left and right and up and down movements,
11. The specific region detection parameter is set for all sizes in the specific region detection parameter setting step when the analysis result indicates that the specific region is accompanied by horizontal and vertical movements. The image processing method according to any one of the above.   In the motion vector analysis step, it is analyzed whether or not there is a left / right / up / down movement in the specific area based on a motion vector in the current frame image data at the same position as or inside the specific area detected in the past frame image data. The image processing method according to claim 14.   In the motion vector analysis step, a total value of motion vector information in the input image data in the same position as the inside or the periphery of the specific area detected in the past frame image data is calculated, and the total value is equal to or greater than a threshold value. The image processing method according to claim 15, wherein it is determined that the specific area is accompanied by a vertical movement.   When it is determined in the motion analysis step that the specific area does not move left and right and up and down, the specific area detection parameter is set with respect to the size of the specific area to be detected in the specific area detection parameter setting step. The image processing method according to claim 11.   When it is determined in the motion vector analysis step that the specific area does not move left and right, up and down, the specific area detection parameter is set with respect to the size of the specific area to be detected in the specific area detection parameter setting step. The image processing method according to claim 14.   The image processing method according to claim 1, wherein a face area is set as the specific area.   The image processing method according to claim 1, wherein parallax information is set as the distance information. A distance calculation unit that calculates distance information in sequentially input image data;
A distance comparison unit that compares the distance information obtained by the distance calculation unit for each of the current frame image data and the past frame image data;
A specific region detection parameter setting unit for setting a specific region detection parameter based on the comparison result in the distance comparison unit;
An image processing apparatus comprising: a specific area detection unit configured to detect a specific area based on the specific area detection parameter set in the specific area detection parameter setting unit. further,
An action analysis unit that analyzes whether or not there is a left / right / up / down movement in the specific area based on the distance information in the past frame image data and the distance information in the current frame image data;
When the analysis result in the motion analysis unit indicates that the specific region does not move left and right and up and down, the distance comparison unit is operated, and when the analysis result indicates that the specific region includes left and right and up and down movement, the distance comparison unit A branch control unit for operating the specific region detection parameter setting unit,
The image processing apparatus according to claim 21, wherein when the analysis result is that the specific area is accompanied by a horizontal and vertical movement, the specific area detection parameter setting unit sets a specific area detection parameter for all sizes. . further,
A motion vector generation unit for generating motion vector information in the input image data;
Based on the motion vector information, it is analyzed whether the specific area has left / right / up / down movement, and if the analysis result is that the specific area does not have left / right / up / down movement, the distance comparison unit is operated, and the analysis result When the specific area is accompanied by left and right and up and down movement, the distance comparison unit is disabled, and the motion vector analysis unit that operates the specific area detection parameter setting unit,
The image processing apparatus according to claim 21, wherein when the analysis result is that the specific area is accompanied by a horizontal and vertical movement, the specific area detection parameter setting unit sets a specific area detection parameter for all sizes. . A procedure for calculating distance information in sequentially inputted image data;
A procedure for comparing the distance information obtained in the distance calculation procedure for each of the current frame image data and the past frame image data;
A procedure for setting a parameter for detecting a specific region based on a comparison result in the procedure for comparing the distance information;
An image processing program for causing a computer to execute a procedure of detecting a specific area based on the parameter for detecting the specific area set in the procedure of setting the parameter for detecting the specific area. Between the step of calculating the distance information and the step of comparing the distance information,
A procedure for analyzing whether or not there is a left / right / up / down movement in the specific area based on the distance information in the past frame image data and the distance information in the current frame image data;
If the analysis result in the procedure for analyzing whether there is left / right / up / down motion is that the specific area does not involve left / right / up / down motion, the process proceeds to the procedure for comparing the distance information. Including the step of skipping the step of comparing the distance information and proceeding to the step of setting the specific region detection parameter.
25. The image according to claim 24, wherein the specific region detection parameter is set for all sizes in the procedure of setting the specific region detection parameter when the analysis result is that the specific region is accompanied by a horizontal and vertical movement. Program for processing. Between the step of calculating the distance information and the step of comparing the distance information,
A procedure for generating motion vector information in input image data;
Based on the motion vector information, analyze whether the specific area has left, right, up and down movement, and if the analysis result is that the specific area does not involve left, right, up and down movement, proceed to the procedure of comparing the distance information, If the analysis result is that the specific area is accompanied by left and right and up and down movement, the procedure of comparing the distance information and skipping the procedure for setting the specific area detection parameter,
25. The image according to claim 24, wherein the specific region detection parameter is set for all sizes in the procedure of setting the specific region detection parameter when the analysis result is that the specific region is accompanied by a horizontal and vertical movement. Program for processing.   An image sensor that receives subject light incident through an optical lens, converts it into an image signal, and outputs the image signal; an A / D converter that converts an image signal output from the image sensor into a digital signal; A digital signal processing circuit including an image processing circuit that performs image processing on the digital signal output from the D converter; a recording circuit that records image data output from the digital signal processing circuit on a medium; and a recorded signal An image pickup apparatus comprising: a reproduction circuit for reproduction; and the image processing circuit according to any one of claims 21 to 23 as the image processing circuit.

Images