JP4692006B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for detecting changes in each part of a face from a captured image.

  In image processing, a face is imaged by a camera, and the orientation of the face and translational motion are estimated from the captured image. As a technique for estimating the movement of the face in this way, for example, by setting a plurality of feature points on the face such as the ends of the eyes and mouth, and tracking those feature points in the captured image, Estimating movement. When estimating facial motion using feature points in this way, when facial expressions change, such as when laughing or angry, the positions of feature points such as eyes and mouth are not only facial motion but also facial expressions. But it will change. For this reason, when the facial motion is estimated without considering the facial expression, the estimation accuracy decreases.

Also, some image processing detects deformation of each part of a face from a captured image in order to recognize a change in facial expression. As a method for detecting the deformation of the face, relatively conspicuous points on the face are detected and tracked, and detection is performed based on the time-series two-dimensional or three-dimensional movement of these points. In addition, there is also a technique that detects the degree of opening and closing of eyes based on a change in the position of the eyes relative to the eyebrows (see Patent Document 1).
JP 9-44685 A

  In the conventional detection of the deformation of each part, since the number of points that are conspicuous on the face is originally small, the number of points that can be feature points is small. Therefore, high detection accuracy cannot be expected even if the deformation of each part is detected from the movement of the few points. Therefore, when feature points are used even if they are not so conspicuous in order to increase the number of feature points, there is a high possibility that the detection and tracking of those points will fail, and eventually the detection accuracy will decrease. In addition, the shape and position of the eyebrows may also change due to changes in facial expressions for detecting the degree of opening / closing of the eyes. In such a case, high accuracy cannot be expected.

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method for detecting changes in each part of the face with high accuracy.

An image processing apparatus according to the present invention uses an imaging unit that captures a face, a feature point detection unit that detects a feature point of a face from a captured image captured by the imaging unit, and a feature point detected by the feature point detection unit. Detecting at least one of a luminance dispersion value, a standard deviation value, and an average value as a statistic value of luminance in the existence area set by the existence area setting means, and an existence area setting means for setting an existence area including the entire face part The image feature detection means for comparing, and the brightness statistical value in the existing area detected by the image feature detection means and a threshold value corresponding to the statistical value are compared, and the judgment for determining the open / closed state of the facial part based on the comparison result Means.

  In this image processing apparatus, a face is imaged by an imaging unit, and a feature point of the face is detected from each captured image by a feature point detection unit. In the image processing apparatus, the image feature detection unit detects the image feature around the feature point. When each part of the face changes (deforms) such as when the mouth opens and closes, there is a clear difference in luminance, color information, etc. before and after the change in the area including each part. Therefore, the image processing apparatus determines the state around the feature point from the image feature around the feature point detected by the determination unit. As described above, since the image processing apparatus detects changes in each part from the image features of the region including each part on the face, the change in each part can be detected with high accuracy, and the number of feature points is small. Can be detected. Therefore, when high-level processing such as face motion estimation and face recognition is performed using this detection result, it is possible to perform processing by eliminating local changes in the face, so that the processing accuracy is improved. Can do.

  The image feature is information about brightness, saturation, color (RGB), and the like. A feature point is a point that can be easily detected in a part of the face that changes itself, and is, for example, each end of eyes, mouth, eyebrows, and the like. The state around the feature point is, for example, opening / closing when the feature point periphery is an eye or mouth, and changing the position or shape when the feature point periphery is eyebrows.

An image processing method according to the present invention uses an imaging step for imaging a face, a feature point detection step for detecting a feature point of a face from a captured image captured at the imaging step, and a feature point detected at the feature point detection step. An existence area setting step for setting an existence area including the entire face part, and detecting at least one of a luminance dispersion value, a standard deviation value, and an average value as a luminance statistical value in the existence area set in the existence area setting step The image feature detection step to be performed, and the determination of the open / closed state of the facial part based on the comparison result by comparing the statistical value of the luminance in the existing area detected in the image feature detection step with a threshold value corresponding to the statistical value And a step. This image processing method has the same effects as the above-described image processing apparatuses.

  According to the present invention, a change in each part of the face can be detected with high accuracy.

  Embodiments of an image processing apparatus and an image processing method according to the present invention will be described below with reference to the drawings.

  In the present embodiment, the present invention is applied to an image processing apparatus that uses a target object as a human face and estimates a facial motion (rotational motion or translational motion). The image processing apparatus according to the present embodiment uses feature points at both ends of the eyes and mouth, and estimates facial motion from changes in the positions of the feature points. In particular, the image processing apparatus according to the present embodiment detects deformations in the eyes and mouth, which are parts where deformation occurs among the feature points.

  The image processing apparatus 1 will be described with reference to FIGS. FIG. 1 is a configuration diagram of an image processing apparatus according to the present embodiment. FIG. 2 is an example of a captured image of the camera of FIG. FIG. 3 is an example of an image showing a face candidate area extracted from the captured image. FIG. 4 is an example of an image showing a candidate area for each feature point set in the face candidate area. FIG. 5 is an example of an image showing each feature point detected from each candidate area. FIG. 6 is an example of an image showing a prediction area of each feature point in the current frame predicted from the position of each feature point in the previous frame. FIG. 7 is an example of an image showing each feature point tracked from each prediction region. FIG. 8 is a histogram of the luminance of the mouth area, where (a) shows a case where the mouth is open and (b) shows a case where the mouth is closed.

  In order to estimate the facial motion with high accuracy, the image processing apparatus 1 estimates the facial motion without using the feature point of the part when the eyes or the mouth that are the feature points are deformed. For this purpose, the image processing apparatus 1 detects the eye area and the mouth area from the captured image, and detects the deformation of the eyes and mouth from the luminance characteristics of each area. The image processing apparatus 1 includes a camera 2 and an image ECU [Electronic Control Unit] 3. In the image ECU 3, a feature point detection / tracking unit 10, a feature point peripheral luminance feature measurement unit 11, a face part deformation detection unit 12, A face motion estimation unit 13 is configured.

  In the present embodiment, the camera 2 corresponds to the imaging means described in the claims, and the feature point detection / tracking unit 10 corresponds to the feature point detection means described in the claims, and around the feature points. The luminance feature measurement unit 11 corresponds to the image feature detection unit described in the claims, and the facial part deformation detection unit 12 corresponds to the determination unit described in the claims.

  As the output of the facial motion estimation result in the image processing apparatus 1, an image showing the facial motion may be displayed on a monitor, or the facial motion state may be output by voice. When such an image processing device 1 is mounted on a vehicle, the motion of the driver's face may be estimated, and the estimation result may be provided as a signal to a travel control device, an alarm device, or the like. For example, in the case of a device that automatically activates the brake according to the distance between the vehicle and the front vehicle, when the driver can estimate that the line of sight has been removed from the front from the estimation result of the movement of the driver's face, the brake is applied earlier than usual. Control such as operating. In the case of an alarm device, an alarm may be output when it can be estimated that the driver is looking away. Or it is good also as a structure incorporated in the robot etc. which communicate with people other than a vehicle.

  The camera 2 is a digital camera provided with an image sensor such as a CCD [Charge coupled device], and transmits a captured image composed of digital image data to the image ECU 3 as an image signal. At this time, the camera 2 continuously captures the imaging target in time and outputs temporally continuous image (moving image) data. Since the image ECU 3 can perform processing if there is at least luminance information, the camera 2 may be a color camera or a monochrome camera. Incidentally, when the image processing apparatus 1 is mounted on a vehicle, the camera 2 is disposed at a position in the passenger compartment where the face of the driver sitting in the driver's seat can be imaged from the front.

  The image ECU 3 is an ECU for image processing, and includes a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like. In the image ECU 3, when the image processing apparatus 1 is activated, a dedicated application program stored in the ROM is loaded into the RAM, and each process described in the program is executed by the CPU. 13 is configured. The image ECU 3 holds in advance a reference image of the face area created from the front face image and a reference image of each feature point. The image ECU 3 continuously captures captured images (for example, every 1/30 seconds) from the camera 2 after the start of processing, detects each feature point in the captured image of the frame in which the face is first detected, and thereafter Each feature point is tracked in the captured image of the frame. Then, the image ECU 3 estimates each area of the eyes and mouth from the positions of the detected and tracked feature points, and obtains luminance characteristics of the eye area and mouth area. Further, the image ECU 3 detects the deformed state of the eyes and mouth from each luminance feature, and when the state of the eyes and mouth differs from the state of the eyes and mouth in the front face image when the reference image is created Does not use the feature points for facial motion estimation.

  The facial feature points are, for example, points that are borders with the skin, such as the corners of both eyes, the heads of each eye, the right and left holes of the nose, and the left and right edges of the mouth. It is assumed that there is a place. Among these feature points, the deformation state is detected only for the feature points related to the eyes and mouth, which are parts that are deformed by conversation, facial expressions, and the like. 4 to 7 show only the feature points related to the eyes and mouth (only the feature points of the part for detecting the deformation state) as an example of the images related to the feature points. There may be feature points other than eyes and mouths for estimation.

  The reference image of the face area and the reference image of each feature point are created in advance and held in the image ECU 3. As a method for creating a reference image, first, an image of a person's face facing the front is prepared. The face state at this time is a normal face state with the mouth closed and the eyes open. The prepared face image may be an image of a person's face to be estimated for facial motion, but is an average face image. The reference image of the face area is created by cutting out a rectangular area including the entire face from the front face image. The reference image of each feature point is created by cutting out a rectangular area centered on each feature point from the front face image.

  Each time the captured image PP is taken in from the camera 2 (see FIG. 2), the image ECU 3 holds the captured image PP in a predetermined area of the RAM and converts it into a luminance image. Incidentally, when the camera 2 is a black and white camera, the conversion into the luminance image is not performed.

  The feature point detection / tracking unit 10 performs feature point detection processing on the captured image PP of each frame until each feature point (face region) can be detected from the captured image PP. Specifically, the feature point detection / tracking unit 10 extracts the face candidate area FA from the captured image (luminance image) PP by template matching using the reference image of the face area (see FIG. 3). Then, the feature point detection / tracking unit 10 sets candidate areas CA1 to CA6 in which each feature point may exist in the face candidate area FA (see FIG. 4). As a method for setting the candidate area, for example, the area within the maximum range that can be assumed from the relative positional relationship of each feature point in the front face image used when creating the reference image of the face area and the reference image of the feature point is used. Set. Further, the feature point detection / tracking unit 10 searches for the feature points CP1 to CP6 from the candidate areas CA1 to CA6 by template matching using the reference images of the feature points (see FIG. 5).

As a method of searching for a face area or feature point, for each reference image of the face area or reference image of each feature point, an area having a size corresponding to each reference image is selected from the brightness image of the captured image or the brightness image of each candidate area. Sequentially cut out, the degree of similarity between the luminance image of the cut-out area and each reference image is calculated sequentially using Equation (1), and the position where the degree of similarity is the highest (that is, the value of corr in Equation (1) is the smallest). Search for (x, y)). In the face area search, if the corr value of Equation (1) does not fall below the threshold even at the position where the similarity is the largest, it is determined that no face exists in the captured image, and the subsequent processing is stopped. The process proceeds to the process for the captured image of the next frame.

  In Equation (1), I (u, v) is a luminance value of the captured image or a luminance value at a position (u, v) on the luminance image of each candidate region, and T (u, v) is on the reference image. , U is the definition area of u (horizontal width of the reference image), V is the definition area of v (vertical width of the reference image), and x is The horizontal reference position when the region is extracted from the luminance image of the captured image or the luminance image of each candidate region, and y is the vertical reference when the region is extracted from the luminance image of the captured image or the luminance image of each candidate region Position.

  The feature point detection / tracking unit 10 performs feature point tracking processing on the captured image PP of each frame after detection of each feature point (face region). Specifically, in the feature point detection / tracking unit 10, there is a possibility that each feature point exists in the captured image (luminance image) PP of the current frame based on the position of each feature point detected or tracked in the previous frame. Certain prediction areas PA1 to PA6 are calculated (see FIG. 6). Then, the feature point detection / tracking unit 10 searches for the feature points CP1 'to CP6' from the prediction regions PA1 to PA6 by template matching using the reference images of the feature points (see FIG. 7).

  As the calculation method of the prediction area, it is considered that the movement of the face between frames is small, so predict the pixel by adding the maximum area that can move between frames from the previous frame to the current frame around each feature point This is an area. That is, (the position of each feature point in the previous frame ± the maximum amount of movement on the inter-frame image) is calculated and set as the prediction region. For example, if the position of a certain feature point in the previous frame is (x0, y0), and there is a possibility that the feature point moves in the horizontal and vertical directions at most (s, t), the prediction region of this feature point The horizontal direction is a range of x0-s to x0 + s, and the vertical direction is a range of y0-t to y0 + t. The feature point search method is the same method as described above, but in tracking feature points, if the value of corr in equation (1) does not fall below the threshold even at the position where the similarity is the highest, tracking is being performed in the captured image. It is determined that there is no feature point (and thus a face), the subsequent processing is stopped, and the process returns to the feature point (face) detection process for the captured image of the next frame.

  When the feature point peripheral luminance feature measurement unit 11 can detect or track each feature point, each region including the entire right eye, the entire left eye, and the entire mouth using the position of the feature point related to the eyes and the mouth among the feature points. Is set as the region existing area. Regarding the setting of this part existence area, the horizontal direction is set based on the position of the feature point, and the vertical direction is sufficient whether it is open or closed in consideration of the case where the eyes and mouth are opened and closed. Set the range to include. Then, the feature point peripheral luminance feature measurement unit 11 obtains the luminance feature of each part existence region using the luminance value of each pixel in the luminance image of each part existence region. Examples of luminance characteristics include a histogram of pixel values indicating a luminance distribution, an average of luminance values, a standard deviation, and a variance.

  FIG. 8 shows a histogram of luminance values as an example of the luminance characteristics of the region where the mouth part is present. FIG. 8A shows a case where the mouth is opened, and FIG. 8B shows a case where the mouth is closed. . When the mouth is opened, a dark portion in the mouth appears, so the histogram is distributed more in places with lower brightness values than when the mouth is closed. Therefore, the average luminance value is smaller when the mouth is opened. Also, as can be seen from the histogram, the distribution range of the luminance values is wider when the mouth is opened, and therefore the variance and standard deviation of the luminance values are larger when the mouth is opened. As for the eyes, when the eyes are closed, the skin color is almost skin color. Therefore, there is a difference in the luminance characteristics between when the eyes are closed and when the eyes are opened. For example, when the eyes are closed, the skin color information occupies most of the information. Therefore, the variance and standard deviation of the luminance values are smaller than when the eyes are opened, and the distribution range of the luminance values in the histogram is also narrowed. Thus, the state of each part can be detected by using each luminance feature, and the detection accuracy is improved by combining several luminance features.

  The face part deformation detection unit 12 detects the state of the left eye, right eye, and mouth based on the luminance characteristics of each part existence region. Then, in the facial part deformation detection unit 12, whether the detected state of the left eye, right eye, and mouth is deformed from the left eye, right eye, and mouth states (closed state) in the front face image when the reference image is created. Determine whether. When there is a deformed part, the face part deformation detection unit 12 specifies a feature point related to the part and provides the information to the face motion estimation unit 13.

  As a method for detecting the state of each part, there are a method of detecting every frame and a method of detecting between temporally consecutive frames. As a method of detecting every frame, there are a method of detecting from a distribution state of a histogram, a method of detecting by comparing an average with a threshold value, and a method of detecting by comparing a variance or standard deviation with a threshold value. One of these methods may be detected, or two or more methods may be combined and detected. In the case of the mouth, for example, a histogram when the mouth is closed is stored in advance, and the opening / closing state of the mouth is detected by obtaining the similarity to the histogram, or a threshold value A is set, and the average is If it is greater than A, it is determined to be closed, if it is less than A, it is determined to be open, or a threshold value B is set, and if the standard deviation is greater than B, it is determined to be open, If it is B or less, it is determined that it is closed. As a method of detecting between successive frames, it is detected whether or not each part has been deformed by detecting one or more temporal changes of a histogram, an average, and a standard deviation (variance). In this case, for example, whether or not a change has occurred in a certain frame can be determined by grasping the phenomenon of whether it has become smaller or larger than a preset ratio compared to the average and standard deviation values in the previous frame. It can be determined, and can also be determined from changes in the distribution state of the histogram.

  When the facial part deformation detection unit 12 determines that all parts are not deformed, the face motion estimation unit 13 uses all the feature points detected or tracked by the feature point detection / tracking unit 10 and uses the feature points. The facial motion (rotational motion and translational motion) is calculated from the time change of the point position. Further, when the face motion estimation unit 13 determines that there is a deformation of a part in the face part deformation detection unit 12, the deformation is selected from all the feature points detected or tracked by the feature point detection / tracking unit 10. Using the feature points excluding the feature points related to the part, the face motion is calculated from the temporal change in the position of the feature points.

  The operation of the image processing apparatus 1 will be described with reference to FIG. In particular, the feature point detection process in the image ECU 3 will be described with reference to the flowchart of FIG. 9, and the feature point tracking process will be described with reference to the flowchart of FIG. FIG. 9 is a flowchart showing a flow of feature point detection processing in the image ECU of FIG. FIG. 10 is a flowchart showing the flow of the feature point tracking process in the image ECU of FIG.

  The camera 2 captures images continuously in time, and sequentially transmits the captured image data to the image ECU 3. The image ECU 3 sequentially takes the captured images, holds the captured images in the RAM, and converts them into luminance images (S10 in FIG. 9 and S20 in FIG. 10).

  If the face candidate area has not yet been extracted, the image ECU 3 searches the captured image for a position similar to the reference image for the face area, and extracts the face candidate area (S11). If the face candidate area cannot be extracted, the image ECU 3 stops the subsequent processing and moves to the processing for the captured image of the next frame. On the other hand, when the face candidate area can be extracted, the image ECU 3 sets each candidate area where each feature point may exist in the face candidate area (S12). Further, the image ECU 3 searches for a position similar to the reference image of each feature point in the candidate area for each feature point, and detects each feature point (S13).

  When the face candidate area has already been extracted, the image ECU 3 determines a predicted area where each feature point may exist in the captured image of the current frame from the position of each feature point detected or tracked in the captured image of the previous frame. Calculate (S21). Then, the image ECU 3 searches for a position similar to the reference image of each feature point in the prediction area for each feature point, and tracks each feature point (S22). If the feature point cannot be tracked, the image ECU 3 stops the subsequent processing, and again moves to the extraction process of the face candidate region for the captured image of the next frame.

  When the feature points can be detected or tracked, the image ECU 3 uses the feature points for the eyes and the mouth among all the feature points to set each part existing region including each eye and the whole mouth. Then, the image ECU 3 obtains a luminance feature for each part existence region from the luminance image of each part existence region. Furthermore, the image ECU 3 detects the state of each eye and mouth based on the luminance characteristics of each region, and determines whether the state of each eye and mouth in the current frame is deformed from the state at the time of the reference image. to decide.

  When it is determined that both eyes and mouth are not deformed, the image ECU 3 uses all the feature points detected or tracked to estimate the facial motion from the temporal change in the position of each feature point. On the other hand, when it is determined that a certain part of both eyes and mouth is deformed, the image ECU 3 uses the feature points excluding the feature points related to the deformed part from all the detected or tracked feature points. The movement of the face is estimated from the time change of the position of each feature point.

  According to this image processing apparatus 1, since the deformation of each part is detected from the luminance characteristics of the region including the eyes and mouth, it can be detected with high accuracy whether or not each part is deformed. Therefore, when estimation of face motion is performed using this detection result, processing can be performed while eliminating local changes in the face, so that the estimation accuracy can be improved.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in the present embodiment, the change in each part of the face is used for estimation of facial motion, but the configuration may be used for other processes such as face recognition and facial expression detection.

  In the present embodiment, each processing unit is configured by executing an application program (software) in the image ECU. However, each processing unit may be configured on a computer such as a personal computer, or hardware. Each processing unit may be configured by wear.

  In this embodiment, the luminance information is used to detect feature points and determine the deformation of each part. However, other image information such as saturation and RGB color information may be used.

  In the present embodiment, the configuration is such that the deformation of the eyes and mouth of the face is detected, but the configuration may be such that the deformation of other parts such as eyebrows is detected.

It is a block diagram of the image processing apparatus which concerns on this Embodiment. It is an example of the captured image of the camera of FIG. It is an example of the image which shows the face candidate area | region extracted from the captured image. It is an example of the image which shows the candidate area | region of each feature point set in the face candidate area | region. It is an example of the image which shows each feature point detected from each candidate field. It is an example of the image which shows the prediction area | region of each feature point in the present frame estimated from the position of each feature point of the previous frame. It is an example of the image which shows each feature point tracked from each prediction area. It is a brightness | luminance histogram of a mouth area | region, (a) is a case where the mouth is open, (b) is a case where the mouth is closed. It is a flowchart which shows the flow of the feature point detection process in image ECU of FIG. It is a flowchart which shows the flow of the feature point tracking process in image ECU of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2 ... Camera, 3 ... Image ECU, 10 ... Feature point detection and tracking part, 11 ... Feature point periphery luminance feature measurement part, 12 ... Face part deformation | transformation detection part, 13 ... Face motion estimation part

Claims (2)

Imaging means for imaging the face;
Feature point detection means for detecting feature points of a face from a captured image captured by the imaging means;
Presence area setting means for setting an existence area including the entire part of the face using the feature points detected by the feature point detection means;
Image feature detection means for detecting at least one of a variance value, a standard deviation value, and an average value of luminance as a statistical value of luminance in the presence area set by the presence area setting means;
A determination unit that compares a statistical value of luminance in the presence area detected by the image feature detection unit with a threshold value corresponding to the statistical value, and determines an open / closed state of the facial part based on the comparison result. A featured image processing apparatus. An imaging step for imaging the face;
A feature point detection step of detecting a feature point of the face from the captured image captured in the imaging step;
An existing area setting step for setting an existing area including the entire face part using the feature points detected in the feature point detecting step;
An image feature detection step of detecting at least one of a variance value, a standard deviation value, and an average value of luminance as a statistical value of luminance in the presence region set in the presence region setting step;
A determination step of comparing a statistical value of luminance in the existing region detected in the image feature detection step with a threshold value corresponding to the statistical value, and determining an open / closed state of a facial part based on the comparison result. A featured image processing method.

Images