JP4315234B2 - Imaging apparatus and facial expression evaluation apparatus - Google Patents

Description

The present invention relates to an imaging equipment for capturing an image using a solid-state image pickup element, and relates to the expression evaluation equipment for evaluating the expression of the captured face.

  In an imaging apparatus, a so-called self-timer function that automatically releases a shutter after a predetermined time has elapsed since an operation of pressing a shutter button is generally mounted not only on a silver salt camera but also on a digital still camera. However, since the timing at which the shutter is released by this self-timer function is determined in advance, there is no guarantee that the subject will always have a good expression when the shutter is released, and an unsatisfactory photograph is taken. There was a problem that was often done.

  On the other hand, in recent years, an image processing technique for performing digital arithmetic processing based on an image signal is rapidly progressing, and as an example, there is a technique for detecting a human face from an image. For example, a luminance difference between two pixels in a face image is learned as a feature amount, and an estimated value indicating whether or not a predetermined region in the input image is a face is calculated based on the feature amount. There has been a face detection technique that finally determines whether or not an image in a region is a face from the above estimated values (see, for example, Patent Document 1).

Such face detection technology has been developed to a level that can be mounted on a digital imaging device that performs imaging with a solid-state imaging device such as a digital still camera, and more recently, a technology for discriminating detected facial expressions. Etc. are also attracting attention. For example, it is considered that a facial expression of a person to be photographed can be evaluated for each captured image from image signals obtained by continuously capturing a plurality of frames, and a suitable image can be selected from the evaluation information. (For example, refer to Patent Document 2).
Japanese Patent Laying-Open No. 2005-157679 (paragraph numbers [0040] to [0052], FIG. 1) JP 2004-46591 A (paragraph numbers [0063] to [0071], FIG. 3)

  Recently, competition between manufacturers of digital imaging devices has intensified, and there is a strong demand to increase the functionality of such imaging devices and increase their commercial value. In addition, as with the problem of the self-timer function described above, the captured image is not always satisfactory for the photographer and the person to be photographed. Therefore, a function for assisting the image capturing operation in order to increase such satisfaction. Can be said to be very important for increasing the product value. In particular, it is desired to realize such a function by using an image processing technique that has been developed, but a function that assists the operation in real time during an imaging operation has not been realized.

The present invention has been made in view of such problems, and an object thereof is to provide an imaging equipment capable of capturing a higher satisfaction image for the photographer or the photographer.
Another object of the present invention is to provide a facial expression evaluation equipment capable of capturing a higher satisfaction image for the photographer or the photographer.

In the present invention, in order to solve the above-described problem, in an imaging apparatus that captures an image using a solid-state imaging device, the image signal is obtained from the image signal in a period until the image signal obtained by imaging is recorded on a recording medium. A face detection unit for detecting a face and the detected facial expression are evaluated, and a facial expression evaluation value indicating the degree of proximity between the specific facial expression and the specific facial expression is calculated. Obtained by imaging when the facial expression evaluation value exceeds a predetermined threshold value, an informing unit for informing the visual information indicating the magnitude of the calculated facial expression evaluation value, and the facial expression evaluation value An image recording control unit that automatically records the image signal on a recording medium, and the notification unit notifies the notification information that changes according to the facial expression evaluation value and the position of the threshold value. Show in response to changes in information And threshold information, the imaging device is provided, characterized in that the notification as a visual information.

In such an imaging apparatus, the face detection unit detects a human face from the image signal during a period until the image signal obtained by imaging is recorded on the recording medium, and the facial expression evaluation unit detects the face detection unit. The facial expression detected by the above is evaluated, and a facial expression evaluation value indicating the degree of closeness to the specific facial expression between the specific facial expression and the other facial expressions is calculated. The image recording control unit automatically records an image signal obtained by imaging on a recording medium when the facial expression evaluation value exceeds a predetermined threshold value. The notification unit notifies the notification information that visually indicates the magnitude of the calculated facial expression evaluation value, and the threshold information that indicates the position of the threshold corresponding to the change in the notification information is also visually Information is reported .

According to the imaging apparatus of the present invention, in the period until the captured image signal is recorded on the recording medium, a human face is detected from the captured image, the facial expression is evaluated, and the facial expression is A facial expression evaluation value indicating how close to a specific facial expression is calculated between a specific facial expression and other facial expressions. When the facial expression evaluation value exceeds a predetermined threshold value, the image signal is automatically Ru is recorded. And the alerting | reporting information which shows the magnitude | size of the facial expression evaluation value is alert | reported with threshold value information . Accordingly, since the photographer or the photographer can recognize whether the facial expression of the photographed person is suitable for imaging, for example, an image that is highly satisfactory for the photographed person or the photographer can be reliably recorded on the recording medium. It becomes like this.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram showing a main configuration of the imaging apparatus according to the first embodiment of the present invention.

  The imaging apparatus shown in FIG. 1 is realized as a digital still camera or a digital video camera. The imaging apparatus includes an optical block 11, a driver 11a, an imaging device 12, a timing generator (TG) 12a, an analog front end (AFE) circuit 13, a camera signal processing circuit 14, a graphic processing circuit 15, a display 16, an image encoder 17, A recording device 18, a microcomputer 19, an input unit 20, an LED (Light Emitting Diode) light emitting unit 21, and an audio output unit 22 are provided.

  The optical block 11 includes a lens for condensing light from the subject on the image sensor 12, a drive mechanism for moving the lens to perform focusing and zooming, a shutter mechanism, an iris mechanism, and the like. The driver 11 a controls driving of each mechanism in the optical block 11 based on a control signal from the microcomputer 19.

  The image pickup device 12 is a solid-state image pickup device such as a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type, and is driven based on a timing signal output from the TG 12a to receive incident light from a subject. Convert to electrical signal. The TG 12 a outputs a timing signal under the control of the microcomputer 19.

  The AFE circuit 13 performs sample hold on the image signal output from the image sensor 12 so as to maintain a good S / N (Signal / Noise) ratio by CDS (Correlated Double Sampling) processing, and further performs AGC (Auto The gain is controlled by gain control), A / D conversion is performed, and digital image data is output.

  The camera signal processing circuit 14 performs detection processing for AF (Auto Focus), AE (Auto Exposure), various image quality correction processing on the image data from the AFE circuit 13, and from the microcomputer 19 based on detection information. Image quality correction processing is performed according to the output signal. As will be described later, in this embodiment, the camera signal processing circuit 14 has a face detection function and a function of extracting data of the face area.

  The graphic processing circuit 15 converts the image data output from the camera signal processing circuit 14 into a signal for display on the display 16 and supplies the signal to the display 16. Further, in response to a request from the microcomputer 19, information such as a facial expression score, which will be described later, is synthesized on the image. The display 16 is composed of an LCD (Liquid Crystal Display), for example, and displays an image based on an image signal from the graphic processing circuit 15.

  The image encoder 17 compresses and encodes the image data output from the camera signal processing circuit 14 and outputs the encoded data to the recording device 18. Specifically, the image data for one frame processed by the camera signal processing circuit 14 is compression-encoded according to an encoding method such as JPEG (Joint Photographic Experts Group), and encoded data of a still image is output. Note that not only still images but also moving image data may be compressed and encoded.

  The recording device 18 is a device that records the encoded data from the image encoder 17 as an image file. For example, the recording device 18 is realized as a drive device of a portable recording medium such as a magnetic tape or an optical disk, or an HDD (Hard Disk Drive). The

  The microcomputer 19 includes a memory such as a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The microcomputer 19 executes the program stored in the memory so that the imaging apparatus is integrated. To control.

  The input unit 20 outputs a control signal corresponding to an operation input by the user to various input switches to the microcomputer 19. As this input switch, for example, a shutter release button, a cross key for selecting various menus, setting an operation mode, and the like are provided.

  The LED light emitting unit 21 lights an LED provided on the exterior surface of the imaging device based on a control signal from the microcomputer 19. As this LED, for example, an LED indicating that the self-timer function is in operation can be considered.

  The sound output unit 22 outputs sound such as an operation confirmation sound based on a control signal from the microcomputer 19. In the case where an encoder / decoder for audio data is provided, the reproduced audio when the audio data is reproduced may be output.

  In this image pickup apparatus, signals received and photoelectrically converted by the image pickup device 12 are sequentially supplied to the AFE circuit 13, subjected to CDS processing and AGC processing, and then converted into digital image data. The camera signal processing circuit 14 performs image quality correction processing on the image data supplied from the AFE circuit 13, and the processed image data is supplied to the graphic processing circuit 15 and converted into an image signal for display. As a result, the currently captured image (camera through image) is displayed on the display 16, and the photographer can view the image and check the angle of view.

  In this state, when the microcomputer 19 is instructed to record an image by pressing the shutter release button of the input unit 20, the image data from the camera signal processing circuit 14 is sent to the image encoder 17. Supplied, subjected to compression encoding processing, and recorded in the recording device 18. When recording a still image, image data for one frame is supplied from the camera signal processing circuit 14 to the image encoder 17, and when recording a moving image, the processed image data is continuously transmitted to the image encoder 17. Supplied.

  Next, an imaging operation mode provided in the imaging apparatus will be described. This imaging device detects a subject's face from the captured image when taking a still image, evaluates the facial expression, and informs the subject of information indicating the degree of evaluation, and its evaluation. A mode in which the shutter is automatically released in accordance with the degree of recording and still image data is recorded in the recording device 18. Hereinafter, the former mode is referred to as “expression evaluation mode” and the latter is referred to as “expression response recording mode”.

  In the facial expression evaluation mode, when a face is detected from a captured image, the facial expression is evaluated, and information corresponding to the evaluation is notified to the photographed person so that the photographed person has a more suitable expression for imaging. Play a role in encouraging. For example, the degree of whether or not the face is a smile is evaluated. Further, in the facial expression response recording mode, when the evaluation value exceeds a predetermined value, it is determined that the face of the subject has become a facial expression suitable for imaging, and still image data is automatically recorded. This helps to record an image that is more satisfactory for the subject. Here, two modes, the facial expression evaluation mode and the facial expression response recording mode, are provided, but only the facial expression response recording mode may be provided.

FIG. 2 is a block diagram illustrating functions of the imaging apparatus for realizing the facial expression evaluation mode and the facial expression response recording mode.
As shown in FIG. 2, the imaging apparatus has a face detection unit 31, a face image generation unit 32, a facial expression evaluation unit 41, a notification control unit 42, and a recording operation control as functions for realizing each imaging operation mode. A portion 43 is provided. In the present embodiment, the face detection unit 31 and the face image generation unit 32 are realized by hardware in the camera signal processing circuit 14, and the facial expression evaluation unit 41, the notification control unit 42, and the recording operation control unit 43 include a microcomputer. 19 is realized as a function of software executed by the computer 19. However, each of these functions may be realized by either hardware or software. Further, the microcomputer 19 holds in advance a discriminant axis information 44 for use in facial expression evaluation calculation by the facial expression evaluation unit 41 in a memory such as a ROM therein. As will be described later, the discriminant axis information 44 is a vector indicating the discriminant axis of facial expressions obtained by performing linear discriminant analysis based on signal components obtained by principal component analysis from sample data of a large number of faces for two facial expressions. Including coefficient information.

Here, the operation of each function shown in FIG. 2 will be described with reference to FIGS. 3 and 4 below. First, FIG. 3 is a diagram showing an outline of the operation in the facial expression evaluation mode.
In the facial expression evaluation mode, first, the face detector 31 detects the face of the subject from the image based on the image data obtained by imaging by the image sensor 12 and transmitted through the camera signal processing circuit 14 ( Step S1). Then, detection information indicating the detected face area is output to the face image generation unit 32. Note that, when notifying by displaying information corresponding to the facial expression evaluation value on the display 16 as in the present embodiment, the face detection information from the face detection unit 31 is sent to the notification control unit 42 of the microcomputer 19. Is also supplied.

  As a face detection method, a known method can be applied. For example, the method described in Patent Document 1 can be used. In this method, first, a luminance difference between two pixels in a face image is learned and stored in advance as a feature amount. Then, as shown in the column of step S1 in FIG. 3, a window W1 having a certain size is sequentially applied to the input image, and a feature amount is determined as to whether or not a face is included in the image in the window W1. Based on the estimation, the estimated value is output. At this time, it is possible to perform estimation using a window W1 having a certain size by sequentially reducing the input image and performing the same processing. Then, the region where the face exists is finally determined from the estimated values obtained by these operations.

  Next, the face image generation unit 32 cuts out data of the detected face area Af from the input image (step S2). Then, the cut-out image data is converted into image data of a certain size, normalized, and supplied to the facial expression evaluation unit 41 (step S3).

  Here, in this embodiment, as an example of face detection information output from the face detection unit 31, the position of a rectangular detection frame surrounding the face (for example, the coordinates of the left end; hereinafter referred to as face position information). )) And the size of the detection frame (for example, the number of pixels in the horizontal and vertical directions, hereinafter referred to as face size information). In this case, the face image generation unit 32 accesses a memory (RAM) in which image data to be detected as a face is temporarily stored, and corresponds to the face position information and size information from the face detection unit 31. Read only area data.

  Further, the cut out image data is normalized by converting the resolution as image data of a certain size (resolution). The normalized image size is a size that becomes a processing unit when the facial expression evaluation unit 41 evaluates facial expressions. In this embodiment, the size is 48 pixels × 48 pixels as an example.

  Note that the image clipping function and resolution conversion function as described above provided in the face image generation unit 32 are the same functions that the conventional camera signal processing circuit 14 generally includes for detection and generation of an output image. Can be diverted.

  Next, the expression evaluation unit 41 performs an operation for evaluating the degree of expression of the face based on the normalized image data of the face from the face image generation unit 32 and the discrimination axis information 44 stored in advance. The facial expression evaluation value is output (step S4). This facial expression evaluation value indicates the degree to which of the two facial expressions is closer. For example, “smile” and “normal facial expression” are applied as two facial expressions, and it is evaluated that the higher the facial expression evaluation value, the stronger the degree of “smiling” than “normal facial expression”. A method for calculating the facial expression evaluation value will be described later.

  Next, the notification control unit 42 notifies the subject of information according to the facial expression evaluation value output from the facial expression evaluation unit 41 (step S5). For example, information corresponding to the facial expression evaluation value is displayed on the display 16 directed toward the subject through the graphic processing circuit 15. In this case, based on the face position and size information supplied from the face detection unit 31, display may be performed in which the face to be evaluated is specified in the display 16. Further, the LED light emitting unit 21 may be used to notify the difference in the facial expression evaluation value due to, for example, a change in luminance, a change in blinking speed, a change in color, or the like. Or you may alert | report by outputting the different audio | voice according to a facial expression evaluation value through the audio | voice output part 22. FIG.

  In the following description, it is assumed that the facial expression evaluation unit 41 evaluates, for example, the degree of whether the facial expression is a smile or no expression. In the present embodiment, in particular, information corresponding to the facial expression evaluation value is displayed on the display 16 with the display surface facing the subject to be notified to the subject. FIG. 3 shows an example in which a bar graph indicating “smile score” which is a value corresponding to the facial expression evaluation value is displayed on the display 16.

FIG. 4 is a diagram for explaining the movement of the bar graph indicating the smile score.
As shown in FIG. 4, the facial expression evaluation value increases as the degree of facial expression is a smile, and decreases as the degree of facial expression is high. In addition, the smile score shown in the bar graph varies continuously or stepwise in proportion to the facial expression evaluation value. This bar graph is displayed on the display 16 directed toward the subject, and the subject sees the bar graph in real time during imaging, so that his / her facial expression is a smile suitable for imaging. Can be recognized. As a result, the bar graph functions to assist the imaging operation so as to prompt the subject to have a facial expression suitable for imaging and to capture a better image. Note that, as will be described later, specific character information or the like that encourages a smile to a subject who has a low facial expression evaluation value may be displayed.

  Here, when the “expression response recording mode” is set, the expression evaluation unit 41 automatically releases the shutter, that is, records a captured image when the expression evaluation value exceeds a predetermined threshold value. To control. In FIG. 2, the recording operation control unit 43 is a block that controls the recording operation of the captured image data. In a normal imaging operation, the imaging operation device 43 detects when the shutter release button of the input unit 20 is pressed. Are controlled so as to perform an imaging operation (for example, an exposure operation and a signal processing operation) suitable for recording, thereby causing the image encoder 17 to encode the captured image data and causing the recording device 18 to record the encoded data. . Then, the facial expression evaluation unit 41 requests the recording operation control unit 43 to execute an image data recording operation when the facial expression evaluation value exceeds a predetermined threshold value.

  As a result, when a face is detected from the captured image and the facial expression is suitable for imaging (in this case, when the degree of smile increases), the captured image at that time is automatically recorded. Become so. Therefore, as compared with the conventional self-timer function (that is, a function of recording a captured image after a certain period of time by pressing the shutter release button), the photographed person can surely capture an image with a good expression. Thus, the satisfaction level of the subject and the photographer can be increased.

Next, a specific example of a smile score display screen on the display 16 will be described.
FIG. 5 is a diagram illustrating a screen display example of a smile score using a bar graph.

  In FIG. 5, a digital video camera 100 is assumed as the imaging apparatus. In the digital video camera 100, a display 16 for checking the angle of view is provided on the side surface of the camera body 101. In the digital video camera 100 having such a configuration, the angle and direction of the display surface of the display 16 are generally variable, and the display surface is in the direction in which the imaging lens 102 is provided as shown in FIG. It is also possible to point in the direction of the person. In the facial expression evaluation mode and the facial expression response recording mode, the display 16 is used with the display surface facing the photographer as described above, and information corresponding to the facial expression evaluation value is displayed together with the image of the imaged subject. .

  In the screen display example of FIG. 5, the smile score display unit 202 is synthesized and displayed on the captured image including the face 201. The smile score display unit 202 displays a smile score corresponding to the facial expression evaluation value as a bar graph 203, and the numerical value display unit 204 displays the smile score as a numerical value. In the facial expression response recording mode, a boundary display icon 205 indicating the boundary of the smile score when the captured image is automatically recorded is displayed. In this example, the threshold value is further displayed numerically on the boundary display icon 205.

  In the example of FIG. 5, together with the smile score display unit 202, a face display frame 206 is displayed around the face 201 corresponding to the smile score, and the smile evaluation target face 201 is displayed in an easy-to-understand manner. Further, in the vicinity of the face display frame 206, a character display unit 207 that displays different characters according to the facial expression evaluation value is provided, and the smile is more intense as the frequency of smiles is reduced. Prompt by letters.

  Also, in the facial expression response recording mode, the user can arbitrarily set a threshold value for facial expression evaluation values when a captured image is recorded, and an image when the subject is smiling is obtained. You may be free to decide what you want. In the example of FIG. 5, for example, when the user presses a left / right direction key (not shown) provided in the input unit 20, the threshold value of the facial expression evaluation value is changed, and the boundary display icon 205 is The smile score corresponding to the threshold value of the facial expression evaluation value can be visually recognized by the user. In this case, when the facial expression response recording mode is set, the left and right direction keys may be automatically associated as keys for setting the facial expression evaluation value threshold value. Can be improved.

  Note that the change in the threshold value of the facial expression evaluation value is not limited to the above method, and may be performed, for example, from a dedicated setting screen selected from the menu screen. Alternatively, a dedicated operation key may be provided. When the display 16 is a touch panel system, the threshold value may be changed by bringing a finger into contact with a key image displayed on the display 16, for example. Further, the threshold value may be changed by moving the finger in contact with the boundary display icon 205 in FIG.

  When a plurality of faces are detected from the imaging screen, facial expression evaluation values may be calculated for each of the faces, and information corresponding to those values may be displayed on the display 16. FIG. 6 is a diagram illustrating a first screen display example of information according to facial expression evaluation values when a plurality of faces are detected.

  FIG. 6 shows an example in which two faces 211 and 212 are detected as an example. In each of the faces 211 and 212, face display frames 213 and 214 are displayed around the face area, and character display portions 215 and 216 are provided in the vicinity thereof. The face display frames 213 and 214 are configured such that their line types change according to the expression evaluation values for the faces 211 and 212, and the character display units 215 and 216 display different characters according to the expression evaluation values. Is done.

  In the example of FIG. 6, the face 211 is evaluated to have a sufficiently strong smile, but the face 212 is evaluated to have an insufficient smile level. For example, the facial expression evaluation value for the face 211 has reached the automatic recording threshold, but the facial expression evaluation value for the face 212 is slightly lower than the threshold. At this time, the face display frame 213 for the face 211 is indicated by a solid line, the face display frame 214 for the face 212 is indicated by a broken line, and the corresponding subject is notified of the difference in the evaluation state. Character information that encourages smiling is displayed. In this example, the difference in facial expression evaluation value is represented by the line type of the face display frames 213 and 214, but other expressions such as differences in brightness, color, thickness, etc. of the face display frames are also possible. You may alert | report the difference in an evaluation value.

FIG. 7 is a diagram illustrating a second screen display example of information corresponding to facial expression evaluation values when a plurality of faces are detected.
In the example of FIG. 7 as well, two faces 211 and 212 are detected as in FIG. 6, the face 211 is evaluated as having a sufficiently strong smile, and the face 212 is evaluated as having insufficient smile. . In the example of FIG. 7, the difference between the facial expression evaluation values is notified to the subject by indicating symbols 217 and 218 corresponding to the facial expression evaluation values in the vicinity of each area of the faces 211 and 212.

  As described above, by displaying information according to the facial expression evaluation value using the display, the smile score according to the facial expression evaluation value is indicated by a bar graph or a numerical value, or the face display frame is displayed according to the facial expression evaluation value. Various information can be used to make the subject's facial expression evaluation value easy to understand, such as changing the line type, color, brightness, etc. Can be notified. In particular, in the case of a digital video camera, since it can be notified using a display surface direction variable display that has been provided conventionally, without significantly increasing the development and manufacturing costs by changing the basic configuration of the camera, An image having a high degree of satisfaction for the user can be reliably recorded.

  In the above, a digital video camera equipped with a display with a variable display screen direction is taken as an example, but the display is also applied to a digital still camera or the like provided with a display for checking the angle of view on the surface opposite to the imaging lens. If the display surface direction is variable and the display surface can be directed to the subject, the display image as described above is displayed to inform the subject of information according to the facial expression evaluation value. Is possible.

Next, a facial expression evaluation method used in this imaging apparatus will be described.
FIG. 8 is a diagram conceptually showing information to be generated in advance for facial expression evaluation and the flow of information generation.

In this embodiment, a so-called “Fisher linear discriminant analysis” technique is used as an expression evaluation technique. In this method, first, a large number of face sample images each having two facial expressions are prepared in advance. Based on the data of these sample images, a two-class problem between the two facial expressions is considered, and linear discriminant analysis is performed. (LDA: Linear discriminant Analysis) are formed in advance discriminant axis a _d to determine well these two expressions by. Then, at the time of facial expression evaluation, by obtaining the inner product of the data in the input face image and the discriminant axis A _d, and calculates a facial expression evaluation value.

  As shown in FIG. 8, in this embodiment, a smiling sample image Ps and a normal facial expression sample image Pn are used. These sample images Ps and Pn are prepared as images normalized to a constant size of 48 pixels × 48 pixels, for example. Then, the data of these sample images is handled as (48 × 48) -dimensional vector data, and LDA processing is performed. However, this vector space is a very large space having (48 × 48) coordinate axes. Therefore, prior to LDA processing, principal component analysis (PCA) is performed on these vector data to convert them into low-dimensional space data that efficiently represents only facial features (dimensional compression). To do.

  In this PCA process, first, M axes are set so that the variation (variance) between input M (for example, M = 300) N-dimensional (N = 48 × 48) sample image groups is maximized. Think about getting. Such an axis is obtained as a solution (eigenvector) of the eigenvalue problem of the covariance matrix of the image group, and a vector suitable for representing facial features by extracting only vector components having relatively large coefficients as principal components. The data can be compressed into N′-dimensional (N >> N ′) data of only components. For example, it has been found that by setting N ′ = 40 or so, sufficient accuracy can be maintained for facial expression discrimination. It should be noted that, by removing about several main components obtained from the PCA processing in descending order of the coefficient, the number of dimensions can be further reduced and the load of the next PCA processing can be reduced while maintaining facial expression discrimination accuracy.

Here, FIG. 9 is a diagram for explaining the masking process of the sample image input during the PCA process.
As shown in FIG. 9A, when the PCA process is performed using the sample image P in which the face 221 is captured in a rectangular area of a predetermined size as it is, an appropriate main component is affected by the influence of the background of the face 221 and the hair. May not necessarily be selected. For this reason, as shown in FIG. 9B, a mask 222 is applied to an area other than the face 221 to convert it into a sample image P1 in which only the face area remains as much as possible, and the PCA process is performed with the face information density increased. By doing so, more accurate dimensional compression can be performed. Furthermore, since the mouth greatly changes depending on the facial expression in the face area and often becomes a disturbance factor, the mouth area is also subjected to PCA processing using the sample image P2 with the mask 223 as shown in FIG. 9C. Thus, the accuracy of dimensional compression can be further increased.

In the following, _referring back to FIG. 8, each sample image of a smile and a normal expression that has been dimensionally compressed by the PCA process is on a partial space (PCA space _Spca ) having only coordinate axes that efficiently represent facial features. To vector data. FIG. 8 schematically shows how the sample images Ps and Pn of smiles and normal expressions are projected onto the PCA space _Spca . As shown in this figure, on the PCA space _Spca , it can be said that sample images having the same expression exist relatively close to each other. Therefore, the sample image group of each expression is regarded as two clusters CL _s and CL _n, and a projection axis (discriminant axis A _d ) that best separates these clusters is formed by LDA processing. Such a discriminant axis _Ad is called a “Fischer projection axis”.

In the LDA process, generally, a discriminant axis that maximizes the variance within a class and between classes projected onto an N′-dimensional eigenvector is obtained. That is, in a class, determine the eigenvector corresponding to the largest eigenvalue of the covariance matrix between classes, to do this the vector on the discriminant axis A _d (Fisher vector). Expressions (1) and (2) show the relationship between each covariance matrix, eigenvalue, and eigenvector.

Here, the inverse matrix, eigenvalue, and eigenvector of the left side of Equation (2) are used for LU (Lower-Upper) decomposition method, QR decomposition method (Q: orthogonal matrix, R: upper triangular matrix), Gaussian elimination, respectively. Can be used. Expression evaluation unit 41 as information of the discriminant axis A _d obtained as above (discriminant axis information 44), information such as coefficients of each component of Fisher vector, previously held like ROM.

FIG. 10 is a diagram conceptually showing the relationship between the discrimination axis and the face input image in the pixel space and PCA space.
The basic procedure for expression determination using the above-described discriminant axis A _d, first, the image data of the detected face from the captured image to the PCA process to extract a principal component. Then, the facial expression of the face image, as shown in the PCA space S _pca in FIG 10, is evaluated as the projection component to the discriminant axis A _d of PCA processed face image vector (input face image vector). That is, the calculation of the facial expression evaluation value E _exp can be calculated by the inner product of the input face image vector and the Fisher vector (see Expression (5-1)).

However, the Fisher vector information can also be converted into information in the pixel space _Spxl (the dimension space of the original image data before PCA processing). Expressions (3) and (4) represent the input face image vector and the Fisher vector as vectors in the pixel space S _pxl , respectively, and FIG. 10 conceptually represents these relationships. As shown in Expression (3), Expression (4), and FIG. 10, vector components other than the main components μ ₁ to μ _{N ′} obtained by PCA processing can be approximated by a constant C as an average value of all input images. Therefore, the inner product operation as shown in Expression (5-1) can be equivalently expressed as an inner product operation of a vector on the pixel space S _pxl as shown in Expression (5-2).

In Expression (5-2), since the subtraction result between the Fisher vector component and the constant C in the pixel space _Spxl can be calculated in advance, the facial expression evaluation unit 41 uses the subtraction result and the constant C to determine the discriminant axis information 44. Hold in advance. Then, when a face image vector detected from the captured image is given, the inner product operation of Expression (5-2) is executed without performing PCA processing on the vector. In the evaluation value calculation for one face according to Expression (5-2), only a maximum of (48 × 48) subtractions, multiplications, and additions are performed, and in practice, about 40 principal components. Only the calculation of the coefficients corresponding to μ _{1 to} μ _{N ′} is executed. Therefore, compared with the case where the inner product calculation of the vector in the PCA space S _pca is performed, the calculation amount can be greatly reduced without reducing the accuracy of the facial expression evaluation, and the angle of view before recording the captured image can be reduced. The facial expression evaluation value E _exp can be easily calculated in real time.

  In such a calculation method, for example, a facial expression evaluation with far lower processing load and high accuracy can be performed compared to a method for evaluating facial expressions by matching a large number of facial image templates with detected facial images. Is possible. When performing matching using a template, it is usually necessary to further extract parts such as eyes and mouth from the detected face image and perform matching processing for each part. On the other hand, in the method of the present embodiment, after normalizing the detected face image data to a certain size, the face image is replaced with vector information and is directly (or only partially masked). As described above, the inner product operation is also a simple one consisting of subtraction, multiplication and addition of about 40 dimensions.

FIG. 11 is a diagram illustrating a calculation example when the facial expression evaluation value is output as a numerical value.
In the present embodiment, as an example, based on the PCA processing result of the sample image, the average of the distribution of the smile face image and the normal expression face image in the PCA space is obtained, and the average of these average discriminant axes _Ad Determine the projection point. Then, the facial expression evaluation value E _exp is converted into a numerical value with the midpoint of each average projection point as a reference. That is, as shown in FIG. 11, the distance between the projection point with respect to the discrimination axis of the input face image and the midpoint of the average projection point is the facial expression evaluation value E _exp, and the side where the smiling sample image is distributed is a positive numerical value. And As a result, it is possible to output whether the detected face image is close to a smile or a normal expression as a continuous numerical value, and a higher expression evaluation value E _exp is evaluated as a stronger smile.

  Next, a processing procedure of the imaging apparatus in the facial expression response recording mode will be described together with a flowchart. FIG. 12 is a flowchart illustrating a process flow of the imaging apparatus in the facial expression response recording mode.

  [Step S11] The face detection unit 31 detects a face from the captured image data, and outputs position information and size information of all detected faces to the face image generation unit 32 and the notification control unit 42.

  [Step S12] The face image generation unit 32 cuts out data of each face area detected from the captured image data based on the position information and size information of the face from the face detection unit 31.

  [Step S13] The face image generation unit 32 normalizes the data of each extracted face area to data of a predetermined number of pixels (here, 48 pixels × 48 pixels), and further masks areas that are not necessary for facial expression detection. Then, the processed image data is output to the facial expression evaluation unit 41.

  [Step S14] The facial expression evaluation unit 41 reads the discriminant axis information 44, and performs an inner product operation of a vector component obtained from one face image among those supplied from the face image generation unit 32 and a vector component of the discriminant axis. Then, the facial expression evaluation value is calculated. The calculated facial expression evaluation value is temporarily stored in, for example, a RAM.

  [Step S15] The facial expression evaluation unit 41 determines whether facial expression evaluation processing has been completed for all detected faces. If not processed, step S14 is executed again for the other face, and if processed, step S16 is executed.

  [Step S16] Based on the facial expression evaluation value calculated in Step S15 and the face position information and size information corresponding to the facial expression evaluation value, the notification control unit 42 displays facial expression information such as a smile score and a display frame. And is synthesized and displayed on the display 16.

  [Step S17] The facial expression evaluation unit 41 determines whether the facial expression evaluation values for all the faces calculated in step S14 exceed a threshold value. If there is an expression evaluation value that does not exceed the threshold value, the process returns to step S11 to instruct the camera signal processing circuit 14 to execute face detection, thereby detecting the next face image and evaluating the expression. Processing begins. If all the facial expression evaluation values exceed the threshold value, step S18 is executed.

  [Step S18] The facial expression evaluation unit 41 requests the recording operation control unit 43 to record captured image data in the recording device 18. As a result, recording signal processing is performed on the captured image, and the encoded image data after processing is recorded in the recording device 18.

  With the above processing, facial expression evaluation values are calculated for all detected faces, and information corresponding to the facial expression evaluation values is notified to the photographed person as display information. You can encourage them to become facial expressions. When all photographed subjects have expressions suitable for imaging, the captured image data is automatically recorded, so that an image that is highly satisfactory for the photographed subject and the photographer can be recorded reliably.

  Note that the determination criterion in step S17 is merely an example, and it is not always necessary to perform control so that image data is recorded when all facial expression evaluation values exceed the threshold value. For example, image data may be recorded when facial expression evaluation values of a certain percentage of detected faces exceed a threshold value. Alternatively, image data is recorded when a certain number of facial expression evaluation values exceed a threshold value, thereby preventing facial expression evaluation from being performed on faces that do not need to be captured accidentally. May be.

  In the facial expression response recording mode described above, a captured image is automatically recorded when the facial expression evaluation value exceeds a predetermined threshold value. In addition to this, for example, the shutter release button is pressed by the photographer. Then, the subject's facial expression may be evaluated after a certain time, and the captured image may be automatically recorded when the facial expression is suitable for imaging. In this case, for example, the microcomputer 19 may start counting the time when detecting the pressing of the shutter release button, and start the processing of FIG. 12 when a certain time has elapsed. By such processing, the photographer who has pressed the shutter release button can move to the imaging range with certainty, and the operability is improved.

  In the above explanation, two facial expressions, “smile” and “normal facial expression”, are defined to determine how close they are to smiles. “Smile” and other facial expressions (non-smile and non-smile) It is also possible to make a determination between This “non-smile” facial expression may include a plurality of facial expressions that are not smiling, such as a straight face, a crying face, and an angry face. In this case, a set of “non-smiles” is obtained from the average of the face sample images corresponding to the plurality of facial expressions, and a discrimination axis for LDA processing is calculated based on this set and the set of “smiles”.

  Furthermore, the facial expression evaluation value does not necessarily indicate the closeness to one facial expression such as “smile”, and for example, a plurality of specific facial expressions such as “smile” and “true face” are considered to be suitable for imaging. It is also possible to show how close to these multiple facial expressions. Also in this case, a set of “facial expressions suitable for imaging” is obtained from the average of the sample face images corresponding to the plurality of facial expressions, and the LDA processing is performed based on this set and a set of “facial expressions unsuitable for imaging”. What is necessary is just to calculate the discriminant axis for this.

[Second Embodiment]
FIG. 13 is a diagram illustrating an appearance of an imaging apparatus according to the second embodiment of the present invention.
In the present embodiment, information corresponding to the facial expression evaluation value is notified to the subject using a part of the LED light emitting unit 21 in the configuration of FIG. The imaging device 110 shown in FIG. 13 is provided with a dedicated LED light emitting unit 21a for notifying information corresponding to the facial expression evaluation value on the surface on which the imaging lens 111, the flash light emitting unit 112, and the like are mounted. The LED light emitting unit 21a is provided with a plurality of LEDs 21b to 21f on one line, and information corresponding to the facial expression evaluation value (here, smile score) is given to the subject by the number of LEDs lit from one of them. Be notified. With such a configuration, even in an imaging apparatus such as a digital still camera that does not have a display screen direction variable display, information corresponding to the facial expression evaluation value is notified to the subject so that an appropriate image is recorded. The imaging operation can be assisted. In addition, by using a small light emitting device such as an LED, it is possible to minimize the increase in size of the imaging apparatus main body.

  Further, in the LED light emitting unit 21a, the LED on the other side (LED 21f in the figure) indicates a smile score when the captured image is automatically recorded. You may make it light-emit by brightness | luminance etc. As a result, the smile score when the automatic recording is performed can be clearly notified to the photographed person, and the photographed person can recognize that the automatic recording has been performed.

[Third Embodiment]
FIG. 14 is a diagram illustrating an appearance of an imaging apparatus according to the third embodiment of the present invention.
In the imaging device 120 shown in FIG. 14, an LED light emitting unit 21g having only one LED is provided on a surface on which the imaging lens 121, the flash light emitting unit 122, and the like are mounted. In such an LED light emitting unit 21g, it is possible to notify the photographer of the smile score by changing the blinking speed of the LED, changing the brightness or color of the LED, or the like according to the facial expression evaluation value. is there. For example, as the facial expression evaluation value increases, it is possible to perform control such as gradually changing the LED color to red, green, and blue, or gradually increasing the LED brightness. In this way, by using only one LED, the enlargement of the imaging device main body can be more significantly prevented.

  Further, when the imaging apparatus 120 has a conventional self-timer function, the LED used when the self-timer is activated can also be used for facial expression evaluation. For example, when the self-timer is activated, the LED blinking speed is gradually increased as time elapses after the shutter release button is pressed and before recording. In the facial expression evaluation mode and the facial expression response recording mode, the higher the facial expression evaluation value, the higher the blinking speed of the LED. With such a configuration, information corresponding to the facial expression evaluation value can be notified to the subject without changing the basic configuration and appearance of the conventional imaging device. Further, the light emitting unit that can be used in combination is not limited to the self-timer, and for example, the light emitting unit for photometry during exposure control can also be used. However, in this case, it is necessary to emit visible light at least during facial expression evaluation.

[Fourth Embodiment]
FIG. 15 is a diagram illustrating an appearance of an imaging apparatus according to the fourth embodiment of the present invention.
In each of the above embodiments, information according to the facial expression evaluation value is visually notified. On the other hand, in this embodiment, information corresponding to the facial expression evaluation value is notified by voice using the voice output unit 22 shown in FIG. In the imaging device 130 shown in FIG. 15, a speaker 22a is provided on the side where the imaging lens 131 is mounted, and different sounds are reproduced and output according to the facial expression evaluation value. As the voice to be output, for example, like the character information shown in FIG. 5 and FIG. 6, the voice that prompts the photographed person to make the smile stronger as the frequency of smile is lower. In this case, the imaging apparatus 130 may hold the audio data to be reproduced in advance in association with the facial expression evaluation value step by step. Further, a method of changing the pitch or output interval according to the facial expression evaluation value, a method of outputting different melody voices according to the facial expression evaluation value, or the like may be employed. Note that voice notification and visual information notification may be used in combination.

[Fifth Embodiment]
FIG. 16 is a diagram showing the appearance of a PC (personal computer) according to the fifth embodiment of the present invention.

  The facial expression evaluation function, the information notification function according to the facial expression evaluation value, and the automatic image recording function according to the facial expression evaluation value in each of the above embodiments are also realized in various computers such as the PC 140 shown in FIG. can do. In FIG. 16, as an example, a display 141 composed of an LCD, a keyboard 142, and a notebook PC 140 in which a main body is integrally formed are illustrated. The PC 140 is provided with an image capturing unit 143 integrally at the upper end of the display 141, for example, so that the user operating the PC 140 can be imaged. The imaging unit 143 may be connected to the outside via a communication interface such as a USB (Universal Serial Bus).

  In such a computer, these functions are realized on the computer by executing the program describing the processing contents of the above functions on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory.

  When this program is distributed, for example, a portable recording medium such as an optical disk on which the program is recorded is sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

1 is a block diagram illustrating a main configuration of an imaging apparatus according to a first embodiment of the present invention. It is a block diagram which shows the function with which an imaging device is provided in order to implement | achieve facial expression evaluation mode and facial expression response recording mode. It is a figure which shows the outline | summary of the operation | movement in facial expression evaluation mode. It is a figure for demonstrating the motion of the bar graph which shows a smile score. It is a figure which shows the example of a screen display of the smile score using a bar graph. It is a figure which shows the 1st screen display example of the information according to the facial expression evaluation value in case a some face is detected. It is a figure which shows the 2nd example of a screen display of the information according to the facial expression evaluation value in case a some face is detected. It is a figure which shows notionally the information which should be produced | generated in advance for facial expression evaluation, and the flow of the information production | generation. It is a figure for demonstrating the masking process of the sample image input at the time of PCA process. It is a figure which shows notionally the relationship between the discrimination | determination axis | shaft in a pixel space and PCA space, and the input image of a face. It is a figure which shows the example of calculation in the case of outputting a facial expression evaluation value as a numerical value. It is a flowchart which shows the flow of a process of the imaging device in facial expression response recording mode. It is a figure which shows the external appearance of the imaging device which concerns on the 2nd Embodiment of this invention. It is a figure which shows the external appearance of the imaging device which concerns on the 3rd Embodiment of this invention. It is a figure which shows the external appearance of the imaging device which concerns on the 4th Embodiment of this invention. It is a figure which shows the external appearance of PC concerning the 5th Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 ... Optical block, 11a ... Driver, 12 ... Image sensor, 12a ... Timing generator (TG), 13 ... Analog front end (AFE) circuit, 14 ... Camera signal processing circuit, 15 ... Graphic processing Circuit: 16: Display, 17: Image encoder, 18: Recording device, 19: Microcomputer, 20: Input unit, 21: LED light emitting unit, 22: Audio output unit, 31: Face detection Part 32... Face image generation part 41. Expression expression evaluation part 42 .. notification control part 43 .. recording operation control part 44 .. discrimination axis information

Claims (13)

In an imaging device that captures an image using a solid-state imaging device,
A face detection unit for detecting a person's face from the image signal in a period until the image signal obtained by imaging is recorded on the recording medium;
A facial expression evaluation unit that evaluates the facial expression detected and calculates a facial expression evaluation value that indicates how close the specific facial expression is between the specific facial expression and other facial expressions;
An informing unit for informing information that visually indicates the size of the calculated facial expression evaluation value;
An image recording control unit that automatically records an image signal obtained by imaging on a recording medium when the facial expression evaluation value exceeds a predetermined threshold;
I have a,
The notification unit notifies, as visual information, the notification information that changes according to the facial expression evaluation value, and threshold information that indicates the position of the threshold corresponding to the change of the notification information. An imaging apparatus characterized by that.   The expression evaluation unit is based on a first face set based on a plurality of face data determined to be included in the specific expression and a plurality of face data determined to be included in an expression other than the specific expression The imaging apparatus according to claim 1, wherein the expression evaluation value indicating a degree of how close to the first face set is calculated with respect to the second face set.   The imaging apparatus according to claim 1, wherein the facial expression evaluation unit applies a smile as the specific facial expression, and calculates a degree of how close the detected face is to the smile as the facial expression evaluation value.   The imaging apparatus according to claim 3, wherein the facial expression evaluation unit applies a normal facial expression as a facial expression other than the specific facial expression.   The imaging apparatus according to claim 3, wherein the facial expression evaluation unit applies facial expressions other than a smile as facial expressions other than the specific facial expression. The imaging apparatus according to claim 1, wherein the threshold value can be changed according to a user input operation. The notification unit displays on the display device a bar display image indicating the size of the facial expression evaluation value by the length of the bar, and displays a position display image indicating the position of the threshold on the bar display image. ,
  The threshold value is changed according to an input operation by a user, and the position of the position display image on the bar display image is changed according to the change of the threshold value. The imaging device described. When the image recording control unit detects an image recording operation by a user, the facial expression evaluation value is acquired from the facial expression evaluation unit after a predetermined time, and when the facial expression evaluation value exceeds a predetermined threshold, imaging is performed. The imaging apparatus according to claim 1, wherein the obtained image signal is automatically recorded on a recording medium. The imaging apparatus according to claim 1, wherein the notification unit notifies the notification information toward a subject person. The imaging device according to claim 9, wherein the notification unit displays the notification information according to the facial expression evaluation value together with an image being captured on a display device. The imaging device according to claim 10, wherein the notification unit displays, as the notification information, a bar display image indicating a size of the facial expression evaluation value by a length of a bar on the display device. The imaging device according to claim 1, wherein the notification unit further outputs a different notification sound corresponding to a size of the facial expression evaluation value as the notification information. In a facial expression evaluation apparatus that evaluates facial expressions imaged using a solid-state imaging device,
  A face detection unit for detecting a person's face from the image signal in a period until the image signal obtained by imaging is recorded on the recording medium;
  A facial expression evaluation unit that evaluates the facial expression detected and calculates a facial expression evaluation value that indicates how close the specific facial expression is between the specific facial expression and other facial expressions;
  An image recording control unit that automatically records an image signal obtained by imaging when the facial expression evaluation value exceeds a predetermined threshold;
  Notification information that changes in accordance with the calculated expression evaluation value and threshold information that indicates the position of the threshold corresponding to the change in the notification information are notified as visual information. A display information generation unit for generating display information for
  A facial expression evaluation apparatus characterized by comprising:

Images