JP2010250419A - Image processing device for detecting eye condition - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently detect the condition of an eye contained in an image. <P>SOLUTION: An image processing device for detecting a predetermined eye condition from an image of an eye contained in a target image includes: a facial area detecting unit for detecting an area containing, at least a part of a face image as a facial area from the target image; an eye area detecting unit for detecting an eye area which is an area containing an eye image from the face area; and an eye condition detecting unit for detecting the predetermined eye condition from the eye area, by using an image vector of a converted image obtained by converting an image of the eye area detected by the eye area detecting unit and a predetermined specific vector which is a specific vector, in relation to the predetermined eye condition of one or more specific vectors used for indicating an optional eye condition obtained, by calculating the eye condition, based on a plurality of sample images including a known eye image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that detects a predetermined eye state from an eye image included in a target image.

  Conventionally, a technique for detecting an eye state from an image including a face image is known. For example, a technique is known in which a region including the eyes is identified by extracting a predetermined feature amount such as a density value distribution from the detected face image, and the state of the eyes is detected by applying a template to the identified region. (Patent Document 1).

JP 2001-229368 A

  However, regarding the technique for detecting the state of the eye included in the image, there is room for efficient detection even by a method different from the above.

  The present invention has been made to solve at least a part of the above-described conventional problems, and an object thereof is to efficiently detect the state of an eye included in an image.

  In order to solve at least a part of the above problems, the present invention employs the following aspects.

  A first aspect provides an image processing apparatus that detects a predetermined eye state from an eye image included in an image of interest. An image processing apparatus according to a first aspect of the present invention includes a face area detection unit that detects an area including at least a part of a face image from the attention image as a face area, and the eye image from the face area. A plurality of eye regions including an eye region detecting unit that detects an eye region that is a region, an image vector of a converted image obtained by converting the image of the eye region detected by the eye region detecting unit, and an eye image having a known eye state The eye region using a predetermined eigenvector that is an eigenvector related to the predetermined eye state among one or more eigenvectors used to represent an arbitrary eye state calculated based on the sample image of And an eye state detector for detecting the predetermined eye state.

  According to the image processing device according to the first aspect, in order to detect the eye state using the image vector of the converted image obtained by converting the image of the detected eye region and the eigenvector related to the eye state, The state of the eyes included in the image can be detected efficiently.

  In the image processing apparatus according to the first aspect, the eye state detection unit may detect the predetermined eye state based on a parameter calculated by an inner product of the image vector and the predetermined eigenvector. In this case, since the eye state is detected by the inner product of the image vector of the converted image obtained by converting the image of the detected eye region and the eigenvector related to the eye state, the eye state included in the image is efficiently determined. Can be detected.

  In the image processing device according to the first aspect, the conversion is such that the detected shape of the eye region becomes a reference shape that is a reference of the shape of the eye region defined based on the plurality of sample images. Conversion may be possible. In this case, since the eye state is detected using the image vector of the converted image converted so that the shape of the detected eye region becomes the reference shape and the eigenvector related to the eye state, it is included in the image. The eye state can be detected efficiently.

  In the image processing device according to the first aspect, the eigenvector is calculated by performing principal component analysis on an image vector of a converted image obtained by performing the conversion on the image of the eye region in the plurality of sample images. Also good. In this case, since the eye state is detected using the eigenvector calculated by principal component analysis of the image vector of the converted image obtained by converting the image of the eye region in the plurality of sample images, the eye state included in the image is determined. It can be detected efficiently.

  In the image processing apparatus according to the first aspect, the predetermined eye state may be a state where the eyes are meditated. In this case, the eye state is detected using the eigenvector related to whether or not the eye is meditated and the image vector of the converted image obtained by converting the image of the detected eye region. It is possible to efficiently detect whether or not the eyes are meditating.

  The image processing apparatus according to the first aspect may include an eye state determination unit that determines whether or not an eye image included in the attention image is in a meditated state using the parameter. In this case, it can be determined whether or not the eye image included in the target image is in a state where the eyes are meditated.

  In the image processing device according to the first aspect, the eye area detection unit detects a position of a facial feature part including at least an outer periphery position of the eye image with respect to the target image based on the face area. An initial position setting unit for setting an initial position of the feature point set in the target image, and the setting position of the feature point set in the target image is updated so as to approach the position of the feature part of the face, A feature position detection unit that detects, as the eye region, an area surrounded by feature points for detecting the position of the outer periphery of the eye image at the updated set position. In this case, the eye region included in the image is efficiently detected by detecting the region surrounded by the feature point for detecting the outer periphery of the eye image at the updated feature point setting position. be able to.

  In the image processing device according to the first aspect, the initial position setting unit includes an eye area as a part, and is an area around the eye that is defined based on a position of the detected face area with respect to the target image The initial position may be set using a region. In this case, the eye area included in the image can be efficiently detected by setting the feature points for the eye peripheral area.

  Note that the present invention can be realized in various modes, for example, a printer, a personal computer, a digital video camera, or the like. Further, an image processing method and apparatus, an eye state detection method and apparatus, a facial expression determination method and apparatus, a computer program for realizing the functions of these methods or apparatuses, a recording medium on which the computer program is recorded, and the computer program Including a data signal embodied in a carrier wave.

1 is an explanatory diagram schematically showing a configuration of a digital still camera 100 as an image processing apparatus in a first embodiment of the present invention. FIG. It is a flowchart which shows the flow of the AAM setting process in 1st Example. It is explanatory drawing which shows an example of sample image SI. It is explanatory drawing which shows an example of the setting method of the feature point CP in the sample image SI. It is explanatory drawing which shows an example of the coordinate of the feature point CP set to sample image SI. Is an explanatory diagram showing an example of the average shape s _0. It is explanatory drawing which illustrated the relationship between shape vector si and shape parameter pi, and face shape s. It is explanatory drawing which shows an example of the method of the warp W of the sample image SI. It is explanatory drawing which shows an example of average face image _A0 (x). It is explanatory drawing for demonstrating an eye state determination information setting process. It is a flowchart which shows the flow of the dozing detection process in 1st Example. It is a flowchart which shows the flow of the eye area | region detection process in 1st Example. It is explanatory drawing which shows an example of the detection result of the face area FA in the attention image OI. It is a flowchart which shows the flow of the initial position setting process of the feature point CP in 1st Example. It is explanatory drawing which shows an example of the temporary setting position of the feature point CP by changing the value of a global parameter. It is explanatory drawing which shows an example of the average shape image I (W (x; p)). It is a flowchart which shows the flow of the feature point CP setting position correction process in 1st Example. It is explanatory drawing which shows an example of the result of a feature point CP setting position correction process. It is explanatory drawing which illustrated the relationship between the norm of a difference image, and characteristic part reliability. It is explanatory drawing which illustrated the initial position of the feature point in a to-be-detected image and an attention image. It is explanatory drawing which illustrated the face area | region in a to-be-detected image and an attention image. It is explanatory drawing for demonstrating the eye area | region detection process in 2nd Example. It is explanatory drawing for demonstrating the eye area | region detection process in a modification.

  Hereinafter, a digital still camera which is an aspect of an image processing apparatus according to the present invention will be described based on examples with reference to the drawings.

A. First embodiment:
A1. Configuration of image processing device:
FIG. 1 is an explanatory diagram schematically showing the configuration of a digital still camera 100 as an image processing apparatus according to the first embodiment of the present invention. The digital still camera (hereinafter also referred to as “DSC”) 100 according to the present embodiment functions as an imaging device (image generation device) that captures an imaging target as a subject and generates image data representing the subject image. It also functions as an image processing apparatus that performs image processing on the processed image data.

  The DSC 100 includes a lens 102, a lens driving unit 104 that drives the lens 102 to adjust a focus position and a focal length, a lens driving control unit 106 that controls the lens driving unit 104, and the lens 102. An image sensor 108 that converts light input to the light receiving surface into an electrical signal, an A / D converter 110 that performs A / D conversion on the electrical signal output from the image sensor 108, and exchange of information with external devices Interface unit (I / F unit) 112, a display unit 114 configured by a liquid crystal display, an operation unit 116 configured by buttons and a touch panel, a CPU 118 for controlling each unit of the DSC 100, and a ROM and a RAM Internal memory 200. The image sensor 108 is configured using, for example, a CCD. Each component of the DSC 100 is connected via a bus 122 so that bidirectional communication is possible.

  The internal memory 200 stores an image generation unit 210, a display processing unit 220, and an image processing unit 230. The image generation unit 210 is a computer program for controlling the lens 102, the image sensor 108, and the like to image an imaging target as a subject and generating image data representing the subject image. The display processing unit 220 is a display driver that controls the display unit 114 to display a processing menu, a message, an image, and the like on the display unit 114, and implements a function by being executed by the CPU 118. The image processing unit 230 is a computer program that performs doze detection processing by being executed by the CPU 118 under a predetermined operating system. The dozing detection process is a process for detecting whether or not a specific face image is dozing. The dozing detection process will be described in detail later.

  The image processing unit 230 includes a face region detection unit 231, an eye region detection unit 232, a detection information calculation unit 235, a processing determination unit 236, an eye state detection unit 237, and an eye state determination unit 238 as program modules. , A doze determination unit 239. The eye area detection unit 232 includes an initial position setting unit 233 and a feature position detection unit 234. The functions of these units will be described in detail in the description of the dozing detection process described later.

  The CPU 118 implements the functions of these units by reading and executing these programs (the image generation unit 210, the display processing unit 220, and the image processing unit 230) from the internal memory 200.

  The internal memory 200 also stores AAM information AMI and eye state determination information IJI. The AAM information AMI is information set in advance by an AAM setting process described later, and is referred to in a dozing detection process described later. The contents of the AAM information AMI will be described in detail in the description of AAM setting processing described later. The eye state determination information IJI is used for detection of an eye state by an eye state detection unit 237 described later. The contents of the eye state determination information IJI will be described in detail in the description of the eye state determination information setting process described later.

A2. AAM setting process:
FIG. 2 is a flowchart showing the flow of AAM setting processing in the first embodiment. The AAM setting process is a process for setting a shape model and a texture model used for modeling an image called AAM (Active Appearance Model). In this embodiment, the AAM setting process is performed by the user.

  First, the user prepares a plurality of images including a person's face as sample images SI (step S110). FIG. 3 is an explanatory diagram showing an example of the sample image SI. As shown in FIG. 3, the sample image SI is related to various attributes such as personality, race / gender, facial expression (anger, laughter, trouble, surprise, etc.), direction (front direction, upward, downward, rightward, leftward, etc.). It is prepared to include different face images. If the sample image SI is prepared in this way, any face image can be accurately modeled by AAM, and accurate face feature position detection processing (described later) for any face image can be executed. It becomes. The sample image SI is also called a learning image.

  A feature point CP is set to the face image included in each sample image SI (step S120). FIG. 4 is an explanatory diagram showing an example of a method for setting the feature point CP in the sample image SI. The feature point CP is a point indicating the position of a predetermined feature part in the face image. In this embodiment, as predetermined feature parts, a predetermined position on the eyebrows in a person's face (for example, an end point or a four-divided point, the same applies hereinafter), a predetermined position on the eye contour, and a predetermined position on the nose and nose contours 68 parts such as a predetermined position on the contour of the upper and lower lips and a predetermined position on the contour of the face (face line) are set. That is, in the present embodiment, a predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour that are commonly included in the human face are set as the characteristic parts. As shown in FIG. 4, the feature points CP are set (arranged) at positions representing 68 feature parts designated by the operator in each sample image SI. Since each feature point CP set in this way corresponds to each feature part, the arrangement of the feature points CP in the face image can be expressed as specifying the shape of the face.

  The position of the feature point CP in the sample image SI is specified by coordinates. FIG. 5 is an explanatory diagram showing an example of the coordinates of the feature point CP set in the sample image SI. In FIG. 5, SI (j) (j = 1, 2, 3,...) Indicates each sample image SI, and CP (k) (k = 0, 1,..., 67) indicates each feature. Point CP is shown. CP (k) -X represents the X coordinate of the feature point CP (k), and CP (k) -Y represents the Y coordinate of the feature point CP (k). The coordinates of the feature point CP include predetermined reference points (for example, in the sample image SI normalized with respect to the size of the face, the inclination of the face (inclination in the image plane), and the position of the face in the X direction and the Y direction) The coordinates with the origin at the lower left point of the image are used. Further, in the present embodiment, a case where a plurality of human face images are included in one sample image SI is allowed (for example, two sample face images are included in the sample image SI (2)). Each person in one sample image SI is specified by a person ID.

  Subsequently, the user sets an AAM shape model (step S130). Specifically, a principal component analysis is performed on a coordinate vector (see FIG. 5) composed of the coordinates (X coordinate and Y coordinate) of 68 feature points CP in each sample image SI, and specified by the position of the feature point CP. The face shape s to be modeled is expressed by the following equation (1). The shape model is also referred to as a feature point CP arrangement model.

In the above formula (1), s ₀ is an average shape. FIG. 6 is an explanatory diagram showing an example of the average shape s ₀ . As shown in FIGS. 6A and 6B, the average shape s ₀ is a model representing the average face shape specified by the average position (average coordinate) for each feature point CP of the sample image SI. is there. In the present embodiment, in the average shape s ₀ , a region surrounded by a straight line connecting feature points CP (face lines, eyebrows, feature points CP corresponding to the eyebrows, see FIG. 4) located on the outer periphery (FIG. 6 ( b) is indicated by hatching) and is referred to as “average shape region BSA”. In the average shape s ₀ , as shown in FIG. 6A, a plurality of triangular areas TA having the feature points CP as vertices are set so as to divide the average shape area BSA into a mesh shape.

In the above equation (1) representing the shape model, s _i is a shape vector, and p _i is a shape parameter representing the weight of the shape vector s _i . The shape vector s _i is a vector representing the characteristics of the face shape s and is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. As shown in the above equation (1), in the shape model in the present embodiment, the face shape s representing the arrangement of the feature points CP is modeled as the sum of the average shape s ₀ and the linear combination of the n shape vectors s _i. It becomes. By appropriately setting the shape parameter p _i in the shape model, the face shape s in any image can be reproduced.

FIG. 7 is an explanatory diagram illustrating the relationship between the shape vector s _{i, the} shape parameter p _i, and the face shape s. As shown in FIG. 7A, in order to specify the face shape s, the number n set based on the cumulative contribution rate in order from the eigenvector corresponding to the principal component having the larger contribution rate (in FIG. 7, n = The eigenvector of 4) is adopted as the shape vector s _i . Each of the shape vectors s _i corresponds to the movement direction / movement amount of each feature point CP, as indicated by the arrows in FIG. In this embodiment, most first shape vector s ₁ corresponding to the first principal component having a large contribution rate is a vector that is approximately correlated with the left and right appearance of a face, by the magnitude of the shape parameter p _1, 7 As shown in (b), the horizontal face direction of the face shape s changes. Second shape vector s ₂ corresponding to the large second principal component of the contribution rate to the second is a vector that is approximately correlated with the vertical appearance of a face, by the magnitude of the shape parameter p _2, FIG. 7 (c) As shown in FIG. 4, the vertical face direction of the face shape s changes. The third shape vector s ₃ corresponding to the third principal component having the third largest contribution ratio is a vector that is substantially correlated with the aspect ratio of the face shape, and the fourth principal component having the fourth largest contribution ratio. The fourth shape vector s ₄ corresponding to is a vector that is substantially correlated with the degree of mouth opening. As described above, the value of the shape parameter represents the feature of the face image such as facial expression and face orientation.

The average shape s ₀ and shape vector s _i set in the shape model setting step (step S130) are stored in the internal memory 200 as AAM information AMI (FIG. 1).

Subsequently, an AAM texture model is set (step S140). Specifically, first, for each sample image SI, image conversion (hereinafter referred to as “warp”) is performed so that the setting position of the feature point CP in the sample image SI is equal to the setting position of the feature point CP in the average shape s ₀ . W ”).

FIG. 8 is an explanatory diagram showing an example of a method of warping W of the sample image SI. In each sample image SI, similarly to the average shape s ₀ , a plurality of triangular regions TA that divide the region surrounded by the feature points CP located on the outer periphery into mesh shapes are set. The warp W is a set of affine transformations for each of the plurality of triangular areas TA. That is, in the warp W, an image of a certain triangular area TA in the sample image SI is affine transformed into an image of the corresponding triangular area TA in the average shape s ₀ . With the warp W, a sample image SI (hereinafter referred to as “sample image SIw”) in which the set position of the feature point CP is equal to the set position of the feature point CP in the average shape s ₀ is generated.

  Each sample image SIw has a rectangular frame containing an average shape area BSA (shown with hatching in FIG. 8) as an outer periphery, and an area other than the average shape area BSA (hereinafter also referred to as “mask area MA”). Generated as a masked image. An image area in which the average shape area BSA and the mask area MA are combined is referred to as a reference area BA. Each sample image SIw is normalized as an image having a size of 56 pixels × 56 pixels, for example.

  Next, a principal component analysis is performed on an image vector composed of pixel values in each pixel group x of each sample image SIw, and a facial texture (also referred to as “appearance”) A (x) is expressed by the following equation (2): ). The pixel group x is a set of pixels located in the average shape area BSA. Here, the pixel value is a luminance value, and the image vector is a luminance value vector composed of luminance values.

In the above equation (2), A ₀ (x) is an average face image. FIG. 9 is an explanatory diagram showing an example of the average face image A ₀ (x). The average face image A ₀ (x) is an image representing the average face of the sample image SIw (see FIG. 8) after the warp W. That is, the average face image A ₀ (x) is an image calculated by taking the average of the pixel values (luminance values) of the pixel group x in the average shape area BSA of the sample image SIw. Therefore, the average face image A ₀ (x) is a model representing the average face texture (appearance) in the average face shape. Note that the average face image A ₀ (x) is composed of the average shape area BSA and the mask area MA, similarly to the sample image SIw, and is calculated as an image having a size of 56 pixels × 56 pixels, for example.

In the above equation (2) representing the texture model, A _i (x) is a texture vector, and λ _i is a texture parameter representing the weight of the texture vector A _i (x). The texture vector A _i (x) is a vector representing the characteristics of the facial texture A (x), and specifically, is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. That is, the number m of eigenvectors set based on the cumulative contribution rate is adopted as the texture vector A _i (x) in order from the eigenvector corresponding to the principal component having the larger contribution rate. In the present embodiment, the first texture vector A ₁ (x) corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with a change in face color (also regarded as a gender difference).

As shown in the above equation (2), in the texture model in the present embodiment, the facial texture A (x) representing the appearance of the face is the average face image A ₀ (x) and m texture vectors A _i (x ) And the linear combination. By appropriately setting the texture parameter λ _i in the texture model, it is possible to reproduce the facial texture A (x) in any image. Note that the average face image A ₀ (x) and the texture vector A _i (x) set in the texture model setting step (step S140 in FIG. 2) are stored in the internal memory 200 as AAM information AMI (FIG. 1). .

By the AAM setting process described above (FIG. 2), a shape model for modeling the face shape and a texture model for modeling the face texture are set. By combining the set shape model and the texture model, that is, the synthesized texture A (x) is converted from the average shape s ₀ to the shape s (inverse conversion of the warp W shown in FIG. 8). Thus, it is possible to reproduce the shape and texture of any face image.

A3. Eye state determination information setting process:
The eye state determination information setting process is a process of setting a texture model for an eye image. In the present embodiment, the eye state determination information setting process is also performed by the user in the same manner as the AAM setting process.

FIG. 10 is an explanatory diagram for explaining eye state determination information setting processing. FIG. 10A is an explanatory diagram illustrating an eye region. In the sample image SI in which the feature point CP is set (FIG. 4), as shown in FIG. 10A, the region surrounded by the feature point CP set on the eye contour of the sample image SI is “eye This is referred to as “area EA”. Image conversion (warp W) is performed on the eye area EA of each sample image SI so that the setting position of the feature point CP forming the eye area EA is equal to the setting position of the feature point CP in the average shape s ₀ . The warp W generates a sample eye image SEIw including an average shape eye area BEA that is an area surrounded by the feature point CP at a position equal to the set position of the feature point CP in the average shape s ₀ .

  FIG. 10B is an explanatory diagram illustrating a sample eye image. As shown in FIG. 10B, each sample eye image SEIw is an image in which a rectangular frame containing the average shape eye area BEA is used as the outer periphery, and an area other than the average shape eye area BEA (mask area MA) is masked. Generated. An image area in which the average shape eye area BEA and the mask area MA are combined is referred to as a reference area BA. Each sample eye image SIw is normalized as an image having a size of 56 pixels × 56 pixels, for example.

Next, a principal component analysis is performed on an image vector composed of pixel values in each pixel group x of each sample eye image SEIw, and the texture of the eyes is modeled by the above equation (2). Here, the pixel group x is a set of pixels located in the average shape eye area BEA. The pixel value is a luminance value, and the image vector is a luminance value vector composed of luminance values. In order to distinguish from the texture model set in the AAM setting process, the texture model set in the eye state determination information setting process will be described below with a dash ('). For example, the texture of the average shape eye area BEA is referred to as texture A ′ (x). A ′ ₀ (x) is an average eye image. The average eye image A ′ ₀ (x) is an image in which the average eye appearance of the sample eye image SEIw (see FIG. 10B) after the warp W is expressed. The average eye image A ′ ₀ (x) is composed of an average shape eye area BEA and a mask area MA, as with the sample eye image SEIw, and is calculated as an image having a size of, for example, 56 pixels × 56 pixels.

In the eye texture model, the eye texture A ′ (x) representing the appearance of the eyes is modeled as the sum of the average eye image A ′ ₀ (x) and a linear combination of m texture vectors A ′ _i (x). It becomes. By appropriately setting the texture parameter λ ′ _i in the eye texture model, it is possible to reproduce any eye image with the texture A ′ (x), for example, by changing the eye opening and closing, the position of the black eye, etc. Is possible. The set average eye image A ′ ₀ (x) and texture vector A ′ _i (x) are stored in the internal memory 200 as eye state determination information IJI (FIG. 1).

FIG. 10C is an explanatory diagram illustrating the relationship between the first texture vector and the sample eye image. In the present embodiment, the first texture vector A ′ ₁ (x) corresponding to the first principal component having the largest contribution rate is a sample to be subjected to principal component analysis so that the first texture vector A ′ ₁ (x) is a vector that is substantially correlated with the opening and closing of the eyes. An eye image SEIw is selected. Specifically, by including in the sample eye image SEIw which is the subject of principal component analysis, an eye image in a closed state and an eye image in an open state, a principal component that is intentionally correlated with the opening and closing of the eyes is formed. I try to do it. Thus, by changing the value of the texture parameter λ ′ ₁ that is a coefficient of the first texture vector A ′ ₁ (x) that is substantially correlated with the opening and closing of the eye, as shown in FIG. A change in the open / closed state of the eye can be expressed. In the present embodiment, a description is given on the assumption that a texture vector correlating with opening and closing of the eye is formed in the first principal component. However, it is not always necessary to form the texture vector in the first principal component. You may form as main components other than this, such as a main component and a 3rd main component.

A4. Dozing detection processing:
FIG. 11 is a flowchart showing the flow of the dozing detection process in the first embodiment. The dozing detection process in the present embodiment is a process for detecting whether or not a target face (hereinafter also referred to as “target face”) represents a dozing state. Specifically, the DSC 100 shoots the target face, generates image data representing an image including the target face, and detects whether or not the target face represents a sleeping state from the image data.

  The image generation unit 210 generates image data that is a target of the dozing detection process (step S210). Specifically, the DSC 100 displays a region to be photographed on the display unit 114, and photographs a region including the target face in response to an operation of the operation unit 116 by the user. The image generation unit 210 generates image data of an area including the captured target face and stores the image data in a predetermined area of the internal memory 200. In the present embodiment, the DSC 100 continuously captures the target face at predetermined time intervals and generates respective image data. Of the generated image data, image data to be described below is particularly referred to as attention image data, and an image represented by the attention image data is referred to as attention image OI.

  The DSC 100 performs eye area detection processing (step s220). The eye area detection process is a process for detecting the eye area EA from the target image OI. FIG. 12 is a flowchart showing the flow of eye area detection processing in the first embodiment. The eye region detection processing in the present embodiment uses AAM to detect the position of the feature part in the face image by determining the arrangement of the feature points CP in the face image included in the target image, and the detected feature part The eye area is specified from the position of. As described above, in the present embodiment, in the AAM setting process (FIG. 2), a total of 68 predetermined positions in the human facial organs (eyebrows, eyes, nose, mouth) and facial contour are set as the characteristic parts. (See FIG. 4). Therefore, the position of the characteristic part in the present embodiment is detected by specifying the positions of 68 characteristic points CP representing predetermined positions in the facial organs and facial contours of the person.

  In the present embodiment, an image that has been subjected to eye region detection processing temporally before the target image OI is also referred to as a detected image DI. The detected image DI has the same image size as the target image OI. Of the detected images DI, in particular, an image taken immediately before the target image OI and subjected to eye region detection processing is defined as a detected image DI1, and an image captured immediately before the detected image DI1 and subjected to eye region detection processing. Is a detected image DI2. Information related to detection of the eye area EA in the detected image DI is referred to as detection information DCI. In the eye area detection process in the present embodiment, the eye area EA is detected from the attention image OI using the detection result of the eye area detection process for the detected image DI in accordance with the detection information DCI. The conditions for using the detection result of the detected image DI and the details of the detection result to be used will be described later.

  As shown in FIG. 12, the process determining unit 236 determines whether or not the detection information DCI exists in the internal memory 200 (step S310). As described above, the detection information DCI is information related to the detection of the eye area for the detected image DI. Specifically, the position of the facial feature detected by the eye area detection unit 232 is true. The coordinate position of the feature part of the face detected in each of the two or more detected images DI having different feature part reliability, which is an index representing the probability of the position of the feature part of the face, and the time when the eye region detection process is performed Including differences. In the present embodiment, description will be made using the characteristic part reliability as the detection information DCI.

  When it is determined that there is no detection information DCI in the internal memory 200 (step S310: NO), the face area detection unit 231 (FIG. 1) performs a face area FA detection process on the target image OI ( Step S320). That is, the face area detection unit 231 detects an area including at least a part of the face image as the face area FA based on the face image included in the target image OI. The detection of the face area FA can be performed using a known face detection method. Known face detection methods include, for example, pattern matching methods, skin color region extraction methods, learning using sample images (for example, learning using neural networks, learning using boosting, and support vector machines). For example, a method using learning data set by learning).

  FIG. 13 is an explanatory diagram illustrating an example of the detection result of the face area FA in the target image OI. FIG. 13 shows the face area FA detected in the target image OI. In the present embodiment, a face detection method is used in which a rectangular area including the vertical direction of the face from the forehead to the chin and the horizontal direction from the outside of both ears is detected as the face area FA.

After the face area FA detection process, the initial position setting unit 233 (FIG. 1) sets the initial position of the feature point CP with respect to the target image OI (step S330). FIG. 14 is a flowchart showing a flow of initial position setting processing of the feature point CP in the first embodiment. In this embodiment, the initial position setting unit 233 includes global parameter values representing the size, inclination, and position (vertical position and horizontal position) of the face image with respect to the face area FA, and shape parameters that are feature amounts. The feature point CP is set to a temporarily set position on the target image OI by variously changing the values of p ₁ and the shape parameter p ₂ (step S410).

FIG. 15 is an explanatory diagram showing an example of a temporary setting position of the feature point CP by changing the value of the global parameter. FIGS. 15A and 15B show a mesh formed by connecting feature points CP and feature points CP in the target image OI. As shown in the center of FIGS. 15A and 15B, the initial position setting unit 233 temporarily sets the feature point CP such that the average shape s ₀ is formed in the center of the face area FA ( Hereinafter, it is also referred to as “reference temporary setting position”.

  The initial position setting unit 233 also sets a plurality of temporary setting positions obtained by changing various global parameter values with respect to the reference temporary setting position. When the global parameters (size, inclination, vertical position and horizontal position) are changed, the mesh formed by the feature point CP in the target image OI is enlarged / reduced, the inclination is changed, and the mesh is moved in parallel. Therefore, as shown in FIG. 15A, the initial position setting unit 233 forms a temporary setting position (a reference temporary setting position diagram) that forms a mesh obtained by enlarging or reducing the mesh at the reference temporary setting position at a predetermined magnification. A temporary setting position (shown on the right and left in the drawing of the reference temporary setting position) that forms a mesh whose inclination is changed clockwise or counterclockwise by a predetermined angle. In addition, the initial position setting unit 233 forms a temporary setting position (in the drawing of the reference temporary setting position in the figure of the reference temporary setting position) that forms a mesh obtained by performing a combination of enlargement / reduction and inclination change on the reference temporary setting position mesh. Also set (upper left, lower left, upper right, lower right).

  Further, as shown in FIG. 15B, the initial position setting unit 233 forms a temporary setting position (reference temporary setting position) that forms a mesh that is translated upward or downward by a predetermined amount from the mesh of the reference temporary setting position. And a temporary setting position (shown on the left and right of the reference temporary setting position diagram) that forms a mesh moved in parallel to the left or right. In addition, the initial position setting unit 233 forms a temporary setting position (upper left of the figure of the reference temporary setting position) that forms a mesh obtained by performing a combination of vertical and horizontal parallel movements on the reference temporary setting position mesh. , Lower left, upper right, and lower right).

  The initial position setting unit 233 performs temporary setting in which up / down / left / right parallel movement illustrated in FIG. 15B is performed on the mesh at each of the eight temporary setting positions other than the reference temporary setting position illustrated in FIG. Also set the position. Therefore, in the present embodiment, the four global parameters (size, inclination, vertical position, horizontal position) are set in 80 ways (= 3 × 3 × 3) that are set as known three-level values. A total of 81 temporary setting positions of (3-1) temporary setting positions and reference temporary setting positions are set.

The initial position setting unit 233 generates an average shape image I (W (x; p)) corresponding to each set temporary setting position (step S420). FIG. 16 is an explanatory diagram showing an example of the average shape image I (W (x; p)). The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the input image is equal to the arrangement of the feature points CP in the average shape s ₀ .

Similar to the conversion for calculating the sample image SIw, the conversion for calculating the average shape image I (W (x; p)) is performed by a warp W that is a set of affine conversions for each triangular area TA. Specifically, an average shape that is an area surrounded by a straight line connecting feature points CP (feature points CP corresponding to face lines, eyebrows, and eyebrows) located on the outer periphery by feature points CP arranged in the target image OI The area BSA is specified, and the average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the average shape area BSA in the target image OI. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image A ₀ (x) and an average shape area BSA and a mask area MA, average face image A ₀ (x) And is calculated as an image of the same size.

As described above, the pixel group x is a set of pixels located in the average shape region BSA in the average shape s ₀ . The pixel group in the image (average shape area BSA of the target image OI) before execution of the warp W corresponding to the pixel group x in the image after execution of the warp W (face image having the average shape s ₀ ) is expressed as W (x; p). To express. Since the average shape image is an image composed of luminance values in each of the pixel groups W (x; p) in the average shape region BSA of the target image OI, it is represented as I (W (x; p)). FIG. 16 shows nine average shape images I (W (x; p)) corresponding to the nine temporarily set positions shown in FIG.

The initial position setting unit 233 calculates a difference image Ie between the average shape image I (W (x; p)) and the average face image A ₀ (x) corresponding to each temporary setting position (step S430). The difference image Ie is a difference between pixel values of the average shape image I (W (x; p)) and the average face image A ₀ (x), and is also referred to as a difference value in this embodiment. Since the difference image Ie does not appear when the setting position of the feature point CP matches the position of the feature part, the difference image Ie represents the difference between the setting position of the feature point CP and the position of the feature part. In this embodiment, since 81 types of temporarily set positions of the feature points CP are set, the initial position setting unit 233 calculates 81 difference images Ie.

  The initial position setting unit 233 calculates a norm from the pixel value of each difference image Ie, and pays attention to a temporary installation position corresponding to the difference image Ie having the smallest norm value (hereinafter also referred to as “norm minimum temporary setting position”). The initial position of the feature point CP in the image OI is set (step S440). The pixel value may be a luminance value or an RGB value. The feature point CP initial position setting process is thus completed.

  When the feature point CP initial position setting process is completed, the process returns to FIG. 12, and the feature position detector 234 (FIG. 1) corrects the set position of the feature point CP in the target image OI (step S340). FIG. 17 is a flowchart showing the flow of the feature point CP setting position correction process in the first embodiment.

  The feature position detection unit 234 calculates the average shape image I (W (x; p)) from the attention image OI (step S510). The calculation method of the average shape image I (W (x; p)) is the same as that in step S420 in the feature point CP initial position setting process.

The feature position detection unit 234 calculates a difference image Ie between the average shape image I (W (x; p)) and the average face image A ₀ (x) (step S520). The feature position detection unit 234 determines whether or not the feature point CP setting position correction process has converged based on the difference image Ie (step S530). The feature position detection unit 234 calculates the norm of the difference image Ie, determines that the norm value has converged if it is smaller than a preset threshold value, and has converged if the norm value is greater than or equal to the threshold value. Judge that there is no.

  The feature position detection unit 234 determines that the calculated norm value of the difference image Ie has converged when the value is smaller than the value calculated in the previous step S520, and still determines that the value is equal to or greater than the previous value. It is good also as what determines with not having converged. Alternatively, the feature position detection unit 234 may perform convergence determination by combining determination based on a threshold value and determination based on comparison with a previous value. For example, the feature position detection unit 234 determines that the calculated norm value is converged only when the calculated norm value is smaller than the threshold value and smaller than the previous value, and otherwise determines that it has not converged yet. It may be a thing.

When it is determined that the convergence has not yet been made in the convergence determination in step S530, the feature position detection unit 234 calculates the parameter update amount ΔP (step S540). The parameter update amount ΔP includes four global parameters (total size, inclination, X direction position, Y direction position), and n shape parameters p _i calculated by the AAM setting process (formula (1)) This means the amount of change in the value of (see). Note that immediately after the feature point CP is set to the initial position, the value determined in the feature point CP initial position setting process is set as the global parameter. Further, difference between the initial position of the characteristic point CP and sets the position of the characteristic points CP of the average shape s ₀ in this case, the overall size, the tilt, because it is limited to the difference in position, shape parameters in the shape model p The values of _i are all zero.

  The parameter update amount ΔP is calculated by the following equation (3). That is, the parameter update amount ΔP is a product of the update matrix R and the difference image Ie.

The update matrix R in the expression (3) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔP based on the difference image Ie, and the internal memory 200 is used as the AAM information AMI (FIG. 1). Stored in In this embodiment, the number of rows M in the update matrix R is equal to the sum ((4 + n)) of the number of global parameters (4) and the number of shape parameters p _i (n), and the number of columns N is , average face image a ₀ average number of pixels in the shape area BSA of (x) - is equal to (56 pixels × 56 pixels the number of pixels in the mask area MA). The update matrix R is calculated by the following formulas (4) and (5).

The feature position detection unit 234 updates parameters (four global parameters and n shape parameters p _i ) based on the calculated parameter update amount ΔP (step S550). Thereby, the setting position of the feature point CP in the target image OI is updated. The feature position detector 234 updates the difference image Ie so that the norm of the difference image Ie becomes small. After the parameter update, the average shape image I (W (x; p)) is calculated again from the target image OI in which the installation position of the feature point CP is corrected (step S510), and the difference image Ie is calculated (step S520), convergence determination based on the difference image Ie (step S530) is performed. If it is determined that the convergence has not occurred in the convergence determination again, the parameter update amount ΔP based on the difference image Ie is calculated (step S540), and the setting position correction of the feature point CP by the parameter update (step S550). ) Is performed.

  When the processing from step S510 to S550 in FIG. 17 is repeatedly executed, the position of the feature point CP corresponding to each feature part in the target image OI approaches the position of the actual feature part as a whole, and converges at a certain time. In the determination (step S530), it is determined that convergence has occurred. If it is determined in the convergence determination that convergence has been achieved, the eye area detection process is completed (step S560). The setting position of the feature point CP specified by the values of the global parameter and the shape parameter set at this time is specified as the setting position of the feature point CP in the final target image OI. Further, the eye area EA in the target image OI is specified from the specified feature point CP. Note that, by repeating the processes from step S510 to S550, the position of the feature point CP corresponding to each feature part in the target image OI may coincide with the position of the actual feature part.

  FIG. 18 is an explanatory diagram illustrating an example of a result of the feature point CP setting position correction process. FIG. 18 shows the setting positions of the feature points CP finally specified in the target image OI. Since the position of the feature point CP specifies the position of the facial feature part (predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour) included in the attention image OI. It is possible to detect the shape and position of the human face organ and the face contour shape in the image OI. An eye area EA is detected. Thus, the feature point CP setting position correction process is completed.

  When the feature point CP setting position correction process is completed, the process returns to FIG. 12, and the detection information calculation unit 235 (FIG. 1) updates the detection information DCI (step S350). In this embodiment, since the feature part reliability is used as the detection information DCI, the detection information calculation unit 235 calculates the feature part reliability for the attention image OI. The feature part reliability is an index calculated based on the norm of the converged difference image Ie, and represents the certainty that the position of the detected feature part is truly the position of the facial feature part. In the eye region detection process, a position that is not the position of the feature part of the face, that is, a position that does not overlap at all with the position of the feature part of the true face or overlaps with a part of the position of the feature part of the true face is accurate. There is a possibility that a position that is not the corresponding position is erroneously detected as the position of the facial feature part. The feature part reliability indicates the certainty that the detection of the position of the feature part of the face is not a false detection but a correct detection.

FIG. 19 is an explanatory diagram illustrating the relationship between the norm of the difference image and the characteristic part reliability. The detection information calculation unit 235 calculates the feature part reliability from the norm of the difference image Ie using the correspondence diagram shown in FIG. 19 prepared in advance. By using the correspondence diagram of FIG. 19, the norm of the difference image Ie is scaled so that the characteristic part reliability value is in the range of 0 to 100. Here, when the feature part reliability is 0, it means that there is a high possibility that the position of the facial feature part is not correctly detected. When the reliability is 100, the position of the facial feature part is correct. It means that the possibility of being detected is high. In order to remove the influence of illumination and the like from the norm of the difference image Ie, the luminance value of each pixel included in the average shape image I (W (x; p)) and the pixel value of the average face image A ₀ (x) ( Normalization may be performed so that the average value and variance of (luminance values) are uniform. The detection information calculation unit 235 stores the calculated characteristic part reliability of the target image OI in the internal memory 200.

  The case where there is no detection information DCI in the internal memory 200 (step S310: NO) has been described above. However, in the following, in the case where the detection information DCI is stored in the internal memory 200 in step S310 (step S310: YES). explain. The process determination unit 236 compares the detection information DCI for the detected image DI1 captured immediately before the target image OI with the first threshold value TH1 stored in the internal memory 200 in advance (step S360).

  When the detection information DCI is greater than or equal to the first threshold value TH1 (step S360: YES), the process determining unit 236 sets the initial position of the feature point CP in the target image OI as the initial feature point CP determined in the detected image DI1. It is determined to be determined based on the position (step S390). When the detection information DCI is equal to or higher than the first threshold TH1, that is, when the characteristic part reliability is equal to or higher than the first threshold TH1, the position of the characteristic part detected in the detected image DI1 is truly a face. Since there is a high possibility of the position of the feature part, the feature point determined in the detected image DI1 is not performed without performing the face area FA detection process (step S320) and the feature point CP initial position setting process (step S330). Even if the initial position of the feature point CP in the target image OI is determined based on the initial position of the CP, the feature point CP can be set at a position close to the facial feature part in the target image OI.

  FIG. 20 is an explanatory diagram illustrating the initial positions of feature points in the detected image and the target image. After the above determination by the processing determination unit 236, the initial position setting unit 233 has the same coordinate position as the initial position of the feature point CP set in the detected image DI1 with respect to the target image OI as shown in FIG. A feature point CP is set. Thereby, the mesh formed by connecting the feature points CP is also set to the same position and range in the target image OI and the detected image DI1. The feature position detection unit 234 corrects the set position of the feature point CP set in the attention image OI (step S340), and detects the eye area EA. Further, the detection information calculation unit 235 calculates the detection information DCI of the target image OI, and updates the detection information DCI in the internal memory 200 (step S350).

  When the detection information DCI is smaller than the first threshold value TH1 (step S360: NO), that is, when the characteristic part reliability is smaller than the first threshold value TH1, the process determination unit 236 further adds the detection information DCI and the first threshold value TH1. The threshold value TH2 of 2 is compared (step S370). The second threshold value TH2 is smaller than the first threshold value TH1. When the detection information DCI is greater than or equal to the second threshold TH2 (step S370: YES), the process determination unit 236 detects the detected image DI1 obtained by photographing the position and range of the face area FA in the target image OI immediately before the target image OI. Is determined to be set based on the position and range of the face area FA in (step S380). When the detection information DCI is greater than or equal to the second threshold TH2, that is, when the characteristic part reliability is greater than or equal to the second threshold TH2, the detection information DCI is inferior to the case where it is greater than or equal to the first threshold TH1. Since there is a high possibility that the position of the feature part detected in the image DI is truly the position of the feature part of the face, the face area FA detection process (step S320) is not performed, and the face set in the detected image DI1 Even if the position and range of the area FA are set as the position and range of the face area FA in the target image OI, the feature point CP can be set at a position close to the feature part of the face in the target image OI.

  FIG. 21 is an explanatory diagram illustrating face areas in the detected image and the target image. After the above determination by the processing determination unit 236, the face area detection unit 231 has the same position as the position and range of the face area FA specified on the detected image DI1 with respect to the target image OI as shown in FIG. The range is specified as the face area FA of the target image OI. The initial position setting unit 233 sets the initial position of the feature point CP based on the face area FA specified in the target image OI (step S330). Further, the feature position detection unit 234 corrects the set position of the feature point CP set in the attention image OI (step S340), and detects the eye area EA. Further, the detection information calculation unit 235 calculates the detection information DCI of the target image OI, and updates the detection information DCI in the internal memory 200 (step S350).

  If the detection information DCI is smaller than the second threshold value TH2 (step S370: NO), it is unlikely that the position of the feature part detected in the detected image DI is truly the position of the face feature part. The unit 236 determines that the face area FA of the target image OI is detected by the face area FA detection process. Thereafter, the initial position setting of the feature point CP (step S330), the setting position correction of the feature point CP (step S340), and the detection information DCI in the internal memory 200 are updated (step S350). Thus, the eye area detection process is completed.

When the eye region detection process is completed, the process returns to FIG. 11 and the image processing unit 230 performs eye meditation determination (step S230). The eye meditation determination is a determination as to whether or not the eye image included in the eye area EA of the target image OI detected by the eye area detection process is in the eye meditation state. Specifically, the eye state detection unit 237 calculates the average shape image I (W (x; p)) when the position of the facial feature part included in the target image OI is specified in the feature point CP setting position correction process. The value of the texture parameter λ ′ ₁ is obtained by the inner product of the image vector composed of the pixel values of the average shape eye area BEA in the first texture vector A ′ ₁ (x) that is a vector that is substantially correlated with the opening and closing of the eyes. calculate. Here, the calculated texture parameter λ ′ ₁ is a coefficient of the first texture vector A ′ ₁ (x), and represents whether or not the eye image included in the target image OI is in the eye-meditation state. Here, the average shape image I (W (x; p)) when the position of the feature part of the face included in the target image OI is specified in the feature point CP setting position correction process is “conversion”. The first texture vector A ′ ₁ (x) that corresponds to the “image” and is substantially correlated with the opening and closing of the eyes corresponds to the “predetermined eigenvector” in the claims.

The eye state determination unit 238 compares the calculated texture parameter λ ′ ₁ with a preset third threshold value TH3 to make eye meditation determination. For example, if the value of the texture parameter λ ′ ₁ is greater than the third threshold value TH3, it is determined that eyes are closed. If the texture parameter λ ′ ₁ is less than the third threshold value TH3, it is determined that eyes are not closed. To do. The eye meditation determination is thus completed.

  When the eye meditation determination is completed, the dozing determination unit 239 performs the dozing determination (step S240). The dozing determination is a determination as to whether or not the target face is dozing. Specifically, the dozing determination unit 239 includes a counter (not shown), and counts the number of eyes determined to be in an eye-meditation state in the eye-meditation determination for the detected image DI or the target image OI. Is reached, i.e., when the number of times that the eye is continuously determined to be in the closed state has reached a predetermined value, it is determined that the target face is dozing. When the DSC 100 determines that the target face is dozing by the dozing determination unit 239, the DSC 100 displays that the display unit 114 is dozing. Thus, the dozing determination process is completed.

As described above, the image processing apparatus according to the first embodiment uses the eigenvector related to the eye state and the image vector of the converted image obtained by converting the image of the detected eye region. Therefore, it is possible to efficiently detect the state of the eyes included in the image. Specifically, the first texture vector A ′ ₁ (x), which is an eigenvector that is substantially correlated with the opening and closing of the eyes, and the position of the feature part of the face included in the target image OI are specified in the feature point CP setting position correction process. The texture parameter λ ′ ₁ is calculated from the inner product of the average shape image area BEA in the average shape image I (W (x; p)) and the image vector formed by the pixel value of the texture parameter λ ′ ₁ Since the value detects whether or not the eye image included in the attention image OI is in the eye-meditation state, the eye state in the attention image OI can be detected efficiently.

  According to the image processing apparatus of the first embodiment, the face area FA of the target image OI is detected based on the face image included in the target image OI or based on the face area FA of the detected image. Since it is determined based on the detection information DCI, it is possible to improve the efficiency and speed of the process of detecting the position of the facial feature part included in the image. Specifically, when the feature part reliability is equal to or higher than the second threshold TH2, it is highly likely that the position of the feature part detected in the detected image DI is truly the position of the facial feature part. Even if the position and range of the face area FA set in the detected image DI1 is set as the position and range of the face area FA in the target image OI, the feature point CP is set at a position close to the facial feature part in the target image OI. Can do. Thereby, it is not necessary to perform the process of detecting the face area FA, and the efficiency and speed of the process of detecting the position of the facial feature part included in the target image can be improved.

  According to the image processing apparatus of the first embodiment, the initial position of the feature point CP set in the target image OI is determined based on the face area FA of the target image OI, or the initial position of the detected image is detected. Since it is determined based on the detection information DCI whether to determine based on the position, it is possible to improve the efficiency and speed of the process of detecting the position of the facial feature part included in the image. Specifically, when the feature part reliability is equal to or higher than the first threshold TH1, there is a higher possibility that the position of the feature part detected in the detected image DI1 is truly the position of the face feature part. Therefore, even if the initial position of the feature point CP in the target image OI is determined based on the initial position of the feature point CP determined in the detected image DI1, the feature point CP is located near the facial feature part in the target image OI. Can be set. This eliminates the need to perform the face area FA detection process and the feature point CP initial position setting process, thereby improving the efficiency and speed of the process of detecting the position of the facial feature part included in the target image. it can.

B. Second embodiment:
In the first embodiment, the shape model and the texture model for the entire face are set in the AAM setting process. However, in this embodiment, a part of the face image including the eye image described later (hereinafter referred to as “eye periphery”). A case will be described in which a shape model and a texture model are set for an image of “region EFA” and eye region detection processing is performed using the shape model and the texture model. Since the configuration of the digital still camera 100 is the same as that of the first embodiment, description thereof is omitted. In addition, in the dozing determination process by the digital still camera 100 of the present embodiment, image data generation (step S210), eye meditation determination (step S230), and dozing determination (step S240) are the same as in the first embodiment. Is omitted.

  FIG. 22 is an explanatory diagram for explaining eye region detection processing in the second embodiment. FIG. 22A is an explanatory diagram illustrating the face area and the eye peripheral area in the second embodiment. In this embodiment, as shown in FIG. 22A, the eye peripheral area EFA is a part of the face area FA, and is described as an area including eyebrows, eyes, and nose among facial feature parts. . In the face area FA detection process (step S320), the face area detection unit 231 identifies the eye peripheral area EFA by detecting the face area FA. That is, the position and range of the eye peripheral area EFA in the target image OI are defined in advance with respect to the position and range of the detected face area FA in the target image OI. The position and range are uniquely defined. The method for defining the position and range of the eye peripheral area EFA is not particularly limited. For example, the eye peripheral area EFA may be a range inside a predetermined number of pixels from the outer periphery of the face area FA, or each of the vertical and horizontal directions of the face area FA. A range calculated by multiplying the number of pixels by a predetermined constant may be used.

FIG. 22B is an explanatory diagram illustrating the eye peripheral area and feature points in the second embodiment. When the initial position of the feature point CP is set (step S330), the initial position setting unit 233 sets the feature point CP as a temporary setting position on the attention image OI using the eye peripheral area EFA instead of the face area FA. . Specifically, a temporary setting position of the feature point CP is set so that an average shape s ₀ corresponding to the eye peripheral area EFA is formed at the center of the eye peripheral area EFA, and various global parameter values are changed. To set multiple temporary setting positions. The feature point CP set at this time and the mesh formed by connecting the feature points CP, as shown in FIG. 22B, feature points CP that form an average shape s ₀ corresponding to the eye peripheral area EFA. And mesh.

That is, in setting the shape model in the AAM setting process, the principal component analysis is performed using only the coordinate vector of the feature point CP corresponding to the feature part included in the eye peripheral area EFA, and the modeling by the equation (1) is performed. I'm doing it. Thereby, the average shape s ₀ formed only by the feature points CP corresponding to the feature parts included in the eye peripheral area EFA is set. Since the feature points CP constituting the average shape s ₀ include the feature points CP set on the contour of the eyes, the eye area EA can be detected. Since the eye region detection processing steps other than those described above are the same as those in the first embodiment, the description thereof will be omitted.

  According to the image processing apparatus in the second example, the shape model and the texture model for the eye peripheral area EFA of the face image are set, so that the eye area included in the image can be efficiently detected. Specifically, in the second embodiment, since the shape model and the texture model for only the eye peripheral area EFA are set in the face image, the facial feature portion using the shape model and the texture model for the entire face is set. Compared to the detection, the number of feature points CP for detecting the coordinate position is small, the processing load necessary for the detection can be reduced, and the eye area can be detected efficiently.

C. Variation:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

C1. Modification 1:
The image processing apparatus according to the first embodiment detects the eye area EA from the attention image OI and determines whether or not the eye is in the eye-meditation state from the eye image included in the eye area. For example, it is also possible to determine the position (orientation) of the black eye. Specifically, a texture vector that correlates with the position of the black eye is set, and the position (orientation) of the black eye is determined based on the value of the texture parameter calculated by the inner product of this texture vector and the image vector eye of the average shape eye area BEA. can do. Further, the characteristic part detected from the target image is not limited to the eyes, and may be another organ such as a mouth. In this case, for example, the opening / closing state of the mouth can be determined.

C2. Modification 2:
The image processing apparatus according to the first embodiment uses the characteristic part reliability of the detected image DI1 as the detection information DCI in the process determination of the eye region detection process for the target image OI. The detection information DCI may be, for example, a difference between the characteristic part reliability levels of the detected image DI1 and the detected image DI2. In this case, the position of the feature part detected in the detected image DI when the difference between the feature part reliability detected in the two detected images DI for which the eye region detection processing has been performed is different from a predetermined value or less. Is likely to be the position of the feature part of the face, as in the first embodiment, both the face area FA detection process, or both the face area FA detection process and the feature point CP initial position setting process The position of the feature part can be detected without performing the process, and the process for detecting the position of the feature part of the face included in the image of interest can be made more efficient and faster.

  Further, the detection information DCI may be other than the characteristic part reliability. For example, the difference between the coordinate positions of the facial feature portions detected in the detected image DI1 and the detected image DI2 may be used as the detection information DCI. The difference in coordinate position may be a distance from a set position in the detected image DI1 to a set position in the detected image DI2 for a predetermined feature point CP, or an average value of the respective distances at all 68 locations. There may be. In this case, if the difference in the coordinate position of the facial feature portion detected in the two detected images DI having different eye region detection times is less than or equal to a predetermined value, the feature detected in the detected image DI Since there is a high possibility that the position of the part is truly the position of the feature part of the face, the face area FA detection process or the face area FA detection process and the feature point CP initial position setting are set as in the first embodiment. The position of the feature part can be detected without performing both processes, and the efficiency and speed of the process for detecting the position of the feature part of the face included in the target image can be improved.

C3. Modification 3:
FIG. 23 is an explanatory diagram for explaining eye region detection processing in a modified example. FIG. 23A is an explanatory diagram illustrating the face area and the eye peripheral area in the modified example. FIG. 23B is an explanatory diagram illustrating the eye peripheral area and feature points in the modification. In the second embodiment, as shown in FIG. 22A, the eye peripheral area EFA is a part of the face area FA and is described as an area including the eyebrows, eyes, and nose among the facial features. However, the eye peripheral area EFA is not limited to this, and may include, for example, eyebrows and noses as long as it includes eyes, and is set for each eye as shown in FIG. It may be. When the eye peripheral area EFA is set for each eye, an average shape s ₀ corresponding to the eye peripheral area EFA is formed for each eye peripheral area EFA, as shown in FIG. Such a temporarily set position of the feature point CP is set, and the feature part is detected. Even in this case, the number of feature points CP to be detected is small compared to the detection of facial feature parts using the shape model and texture model for the entire face, and the detection load can be reduced. Thus, the eye area can be detected. It can also be realized as an image processing device that detects the state of either the right eye or the left eye, and because the state of the left and right eyes can be detected independently, the state of the left and right eyes such as a wink It is also possible to detect images with different values.

C4. Modification 4:
The image processing apparatus according to the present embodiment includes the eye state determination unit 238 and the eye meditation determination unit 239, but does not include the eye state determination unit 238 or the eye state determination unit 238 and the eye meditation determination unit 239. May be. For example, even if the eye state determination unit 238 is not provided, a digital still having a function of displaying on the display unit 114 the value of the texture parameter λ ′ ₁ indicating whether or not the eye is in the meditated state calculated by the eye state detection unit 237. It can also be realized as a camera 100, and can also be realized as a digital still camera 100 having a function of determining only eye meditation.

C5. Modification 5:
The image processing apparatus according to the present embodiment includes the process determination unit 236, but does not include the process determination unit 236, and the face area FA detection process and the feature point CP initial position setting process for all the generated target images OI. It is also possible to implement the above. Even in this case, it is possible to efficiently detect the state of the eye in the target image OI in order to detect whether or not the eye image included in the target image OI is in the eye-meditation state based on the value of the texture parameter λ ′ _1. it can.

C6. Modification 6:
The sample image SI in this embodiment is merely an example, and the number and type of images employed as the sample image SI can be arbitrarily set. In the present embodiment, the predetermined feature portion of the face indicated by the position of the feature point CP is merely an example, and a part of the feature portion set in the embodiment may be omitted or another portion may be used as the feature portion. May be adopted.

  In this embodiment, the texture model is set by principal component analysis with respect to the luminance value vector formed by the luminance values in each of the pixel groups x of the sample image SIw, but the luminance representing the texture (appearance) of the face image. The texture model may be set by principal component analysis for index values other than the values (for example, RGB values).

In this embodiment, the size of the average face image A ₀ (x) and the average eye image A ′ ₀ (x) is not limited to 56 pixels × 56 pixels, and may be other sizes. The average face image A ₀ (x) and the average eye image A ′ ₀ (x) do not need to include the mask area MA, and are configured only by the average shape area BSA or the average shape eye area BEA. Also good. Further, instead of the average face image A ₀ (x) and the average eye image A ′ ₀ (x), another reference face image or reference eye image set based on the statistical analysis of the sample image SI is used. Also good.

  In this embodiment, the shape model and the texture model are set using AAM. However, the shape model and the texture model using other modeling methods (for example, a method called Morphable Model or a method called Active Blob) are used. A texture model may be set.

  In this embodiment, the image represented by the image data generated by the image generation unit 210 is the attention image OI. The attention image OI is, for example, an external device connected via the I / F unit 112. It may be an image acquired from a device. Further, the digital still camera 100 may include a card slot (not shown), and an image acquired from the memory card may be used as the attention image OI.

  In this embodiment, image processing by the digital still camera 100 as an image processing apparatus has been described. However, part or all of the processing is performed by another type of image processing apparatus such as a personal computer, a printer, or a digital video camera. It is good also as what is done. Further, if the image processing according to the present embodiment is used, it can be realized as an eye state detection device that detects a drowsy driving in an automobile, a railway vehicle, a ship or the like and issues an alarm.

  In this embodiment, a part of the configuration realized by hardware may be replaced with software, and conversely, a part of the configuration realized by software may be replaced by hardware.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

DESCRIPTION OF SYMBOLS 100 ... Digital still camera 102 ... Lens 104 ... Lens drive part 106 ... Lens drive control part 108 ... Image pick-up element 110 ... A / D converter 110
112 ... I / F unit 114 ... Display unit 116 ... Operation unit 118 ... CPU
DESCRIPTION OF SYMBOLS 122 ... Bus 200 ... Internal memory 210 ... Image generation part 220 ... Display processing part 230 ... Image processing part 231 ... Face area detection part 232 ... Eye area detection part 233 ... Initial position setting part 234 ... Feature position detection part 235 ... Detection information Calculation unit 236 ... processing determination unit 237 ... eye state detection unit 238 ... eye state determination unit 239 ... dozing determination unit

Claims (11)

An image processing apparatus for detecting a predetermined eye state from an eye image included in a target image,
A face area detector that detects an area including at least a part of a face image from the noted image as a face area;
An eye area detection unit that detects an eye area that is an area including the image of the eye from the face area;
An arbitrary eye state calculated based on an image vector of a converted image obtained by converting the image of the eye region detected by the eye region detection unit and a plurality of sample images including an eye image whose eye state is known An eye state detection unit that detects a predetermined eye state from the eye region using a predetermined eigenvector that is an eigenvector associated with the predetermined eye state among one or more eigenvectors used to represent And an image processing apparatus. The image processing apparatus according to claim 1.
The eye state detection unit is an image processing device that detects the predetermined eye state based on a parameter calculated by an inner product of the image vector and the predetermined eigenvector. The image processing apparatus according to claim 1 or 2,
The image processing device is an image processing apparatus in which the conversion is a conversion in which the detected shape of the eye region becomes a reference shape that is a reference of the shape of the eye region defined based on the plurality of sample images. The image processing apparatus according to claim 3.
The eigenvector is calculated by performing principal component analysis on an image vector of a converted image obtained by performing the conversion on the image of the eye region in the plurality of sample images. The image processing apparatus according to any one of claims 1 to 4,
The predetermined eye state is an image processing apparatus in which the eyes are meditated. The image processing apparatus according to claim 5 further includes:
An image processing apparatus comprising: an eye state determination unit that determines whether an eye image included in the target image is in a state where the eye is meditated using the parameter. The image processing apparatus according to any one of claims 1 to 6,
The eye area detector
An initial position for setting an initial position of a feature point set in the target image in order to detect a position of a feature part of the face including at least a position of an outer periphery of the eye image with respect to the target image based on the face region A setting section;
The feature point set in the image of interest is updated so as to be close to the position of the feature part of the face, and the feature point for detecting the outer peripheral position of the eye image at the updated set position An image processing apparatus comprising: a feature position detection unit that detects an enclosed area as the eye area. The image processing apparatus according to claim 7.
The initial position setting unit is an image that includes the eye area as a part and sets the initial position using an eye peripheral area that is an area defined based on a position of the detected face area with respect to the target image. Processing equipment. A digital camera that detects a predetermined eye state from an eye image included in an attention image,
A shooting unit that shoots an area including a face and generates a notice image;
A face area detector that detects an area including at least a part of a face image from the noted image as a face area;
An eye area detection unit that detects an eye area that is an area including the image of the eye from the face area;
An arbitrary eye state calculated based on an image vector of a converted image obtained by converting the image of the eye region detected by the eye region detection unit and a plurality of sample images including an eye image whose eye state is known An eye state detection unit that detects a predetermined eye state from the eye region using a predetermined eigenvector that is an eigenvector associated with the predetermined eye state among one or more eigenvectors used to represent And a digital camera. An image processing method for detecting a predetermined eye state from an eye image included in a target image,
Detecting an area including at least a part of a face image from the noted image as a face area;
Detecting an eye area which is an area including the eye image from the face area;
Used to represent an arbitrary eye state calculated based on an image vector of a converted image obtained by converting the detected image of the eye region and a plurality of sample images including eye images whose eye states are known Detecting the predetermined eye state from the eye region using a predetermined eigenvector that is an eigenvector associated with the predetermined eye state among one or more eigenvectors. A computer program for image processing for detecting a predetermined eye state from an eye image included in an attention image,
A face area detection function for detecting an area including at least a part of a face image from the attention image as a face area;
An eye area detection function for detecting an eye area that is an area including the image of the eye from the face area;
Used to represent an arbitrary eye state calculated based on an image vector of a converted image obtained by converting the detected image of the eye region and a plurality of sample images including eye images whose eye states are known An eye state detection function for detecting the predetermined eye state from the eye region using a predetermined eigenvector that is an eigenvector associated with the predetermined eye state among one or more eigenvectors is realized in a computer Computer program

Images