JP2009017215A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image processing method, and an image processing program which are capable of quickly and precisely deducing the expression on the face of a subject. <P>SOLUTION: The image processing apparatus includes a means for acquiring images of the subject, with prescribed resolutions at a plurality of different timings; a means for detecting face areas from the images of the subject; a means for extracting information relating to shapes of components of the subject's face from the face areas; a means for selecting reference face information being a reference for estimating the expression from face information; and an expression estimation means for estimating the expression on the subject's face, on the basis of reference face information. First, a reference facial information is generated from an image photographed by a first resolution; next a second facial information relating to shapes of components of the subject's face is extracted from an image photographed at a second resolution which is lower than the first resolution; and then, the reference face information and the second face information are compared, to deduce the expression on the subject's face by the expression estimation means. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

  In recent years, techniques for detecting facial expressions, particularly human facial expressions, are being developed. In general, when a person is photographed with a camera, it is often desired that the facial expression of the person who is the subject is photographed with a good expression such as a smile. Therefore, the application of facial expression detection technology to digital cameras is being studied.

In the technique disclosed in Patent Document 1, a plurality of first feature points are extracted from an expressionless reference image for use in facial expression recognition processing in advance, and a distance between the first feature points is calculated. Further, after extracting a plurality of second feature points from the determination target image, a distance between the second feature points is calculated. Then, the facial expression is determined by calculating a difference value between the distance between the first feature points and the distance between the second feature points.
JP 2005-56388 A

  To shoot without missing a photo opportunity when it is determined that the facial expression is good, it is necessary to estimate the facial expression of the subject at high speed. In order to estimate the facial expression of the subject at high speed, it is conceivable to reduce the resolution of the acquired image data, reduce the amount of data to be processed, and shorten the processing time.

  Here, by lowering the resolution of the acquired image data, the number of facial feature points used for estimating the facial expression may be reduced, which may reduce the accuracy of estimating the facial expression of the subject.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of estimating the facial expression of a subject at high speed and with high accuracy.

  An image processing apparatus according to a first aspect of the present invention is configured to acquire first image data obtained by imaging a subject at a plurality of different timings at a first resolution, and acquired by the acquisition unit. In each of the plurality of first image data, a face area detection unit that detects a first face area of the subject, and a plurality of the first face areas detected by the face area detection unit, Face information extraction means for extracting first face information related to the shape of the constituent elements of the face of the subject, and the facial expression of the subject from the plurality of first face information extracted by the face information extraction means. Face information selection means for selecting reference face information serving as a reference for estimation, and facial expression estimation means for estimating a facial expression of the subject based on the reference face information, wherein the acquisition means includes the reference Face Is selected, the second image data obtained by imaging the subject is acquired at a second resolution lower than the first resolution, and the face area detecting means is acquired by the acquiring means. A second face area of the subject is detected in the second image data, and the face information extracting means detects the face of the subject from the second face area detected by the face area detecting means. Second facial information related to the shape of the component is extracted, and the facial expression estimating means compares the reference facial information with the second facial information, and compares the reference facial information with the second facial information. It is characterized by estimating facial expressions.

  The image processing method according to the second aspect of the present invention includes a first acquisition step of acquiring, at a first resolution, first image data obtained by imaging a subject at a plurality of different timings; In each of the plurality of first image data acquired in the acquisition step, the first face area detection step for detecting the first face area of the subject and the first face area detection step detected. A first face information extraction step for extracting first face information related to the shape of the constituent elements of the face of the subject from each of the plurality of first face regions, and the first face information extraction step. A first face information selection step of selecting reference face information serving as a reference for estimating a facial expression of the subject from the plurality of first face information; and the first face information selection step; Reference face information is selected After that, the second acquisition step of acquiring the second image data obtained by imaging the subject at a second resolution lower than the first resolution is acquired at the second acquisition step. In the second image data, the second face area detecting step for detecting the second face area of the subject, and the second face area detected in the second face area detecting step, A second face information extracting step for extracting second face information related to the shape of the constituent element of the subject's face is compared with the second face information by comparing the reference face information and the second face information. And a facial expression estimation step for estimating the facial expression of the subject corresponding to.

  An image processing program according to a third aspect of the present invention includes a first acquisition step of acquiring, at a first resolution, first image data obtained by imaging an object at a plurality of different timings in an image processing apparatus. A first face area detecting step for detecting a first face area of the subject in each of the plurality of first image data obtained in the first obtaining step; and the first face area detection. A first face information extracting step of extracting first face information relating to the shape of a constituent element of the face of the subject from each of the plurality of first face regions detected in the step; and the first face information A first face information selection step of selecting reference face information as a reference for estimating the facial expression of the subject from the plurality of first face information extracted in the extraction step; and the first face Information selection step A second acquisition step of acquiring second image data obtained by imaging the subject at a second resolution lower than the first resolution after the reference face information is selected in In the second image data acquired in the second acquisition step, a second face area detection step for detecting a second face area of the subject, and the detection in the second face area detection step. A second face information extracting step for extracting second face information relating to the shape of a constituent element of the subject's face from a second face area; and comparing the reference face information and the second face information. A facial expression estimation step of estimating a facial expression of the subject corresponding to the second facial information.

  According to the present invention, the facial expression of a subject can be estimated at high speed and with high accuracy.

  An image processing apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a configuration diagram of an image processing apparatus 100 according to an embodiment of the present invention.

  The image processing apparatus 100 is a digital camera, for example. The image processing apparatus 100 includes the following components.

  The protection means 102 is, for example, a barrier. The protection unit 102 is disposed between the outside and the photographing lens 10 to protect the photographing lens 10. The shutter 12 is disposed between the photographing lens 10 and a photoelectric conversion unit 14 described later, and has a diaphragm function for reducing the amount of light guided from the photographing lens 10 to the photoelectric conversion unit 14.

  The acquisition unit 17 includes a photoelectric conversion unit 14, a reading unit 15, and an A / D conversion unit 16. The photoelectric conversion means 14 generates an image signal by photoelectrically converting the optical image of the subject. The reading unit 15 reads an image signal from the photoelectric conversion unit 14. The A / D conversion unit 16 A / D converts the image signal read by the reading unit 15 to generate image data.

  Here, the acquisition means 17 images a subject at a plurality of different timings, and acquires a plurality of first image data of the subject at a first resolution. That is, the photoelectric conversion unit 14 photoelectrically converts the optical image of the subject to generate a first image signal. The readout unit 15 reads out the first image signal from the photoelectric conversion unit 14 with the first number of pixels. The A / D conversion unit 16 A / D converts the first image signal read by the reading unit 15 to generate first image data. The resolution of the first image data is the first resolution. These operations are performed at a plurality of different timings, and a plurality of first image data are acquired at the first resolution.

  The acquisition unit 17 captures the subject and acquires second image data of the subject at a second resolution lower than the first resolution. That is, the photoelectric conversion unit 14 photoelectrically converts the optical image of the subject to generate a second image signal. The reading unit 15 reads the second image signal from the photoelectric conversion unit 14 with a second number of pixels smaller than the first number of pixels. The A / D conversion unit 16 A / D converts the second image signal read by the reading unit 15 to generate second image data. The resolution of the second image data is the second resolution.

  The timing generation unit 18 is controlled by the memory control unit 22 and the system control unit 50, and supplies a clock signal and a control signal to the photoelectric conversion unit 14, the reading unit 15, the A / D conversion unit 16, and a D / A converter 26 described later. Supply.

  The image processing unit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D conversion unit 16 or the data from the memory control unit 22.

  The image processing unit 20 performs predetermined calculation processing using the image data received from the A / D conversion unit 16 and the like, and supplies the obtained calculation result to the system control unit 50. As a result, the system control unit 50 controls the exposure control means 40 and the distance measurement control means 42, and performs TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, EF (flash pre-emission) processing is performed.

  Further, the image processing unit 20 performs predetermined calculation processing using the captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained calculation result.

  The image processing unit 20 includes a face area detecting unit 20a and a face information extracting unit 20b. The face area detection unit 20 a detects the face area of the subject in each of the plurality of image data acquired by the acquisition unit 17. For example, a shape corresponding to a facial component such as a mouth or eyes is extracted from image data, and a face region is detected based on the position of the structural component. The face information extraction unit 20b extracts face information related to the shape of the constituent elements of the face of the subject from each of the plurality of face regions detected by the face region detection unit 20a. The face information includes, for example, information on specific shapes and coordinates such as the mouth, eyes, and eyebrows, or includes information on the shape and coordinates of, for example, the nose end point, the nostril center point, and the ear end point. . Here, the face information is extracted from the image of the face to which each end point, center point, eye black point, or the like is input, by using a method such as edge detection using a neural network or a spatial filter, for example. be able to. When the face area detecting means 20a extracts the face area, it is determined whether or not there is a shape that satisfies the conditions as the constituent elements of the face such as the mouth and eyes. In other words, it doesn't matter what the specific shape of the mouth is, as long as it meets the mouth condition, whether it ’s a wide open mouth when you are laughing or a closed mouth when you are silent. On the other hand, when the face information extracting means 20b extracts the face information, even the same mouth, including information on the shape of the mouth corner and the shape of the mouth and the coordinates of the face area are included. Extracted.

  Here, the face area detection unit 20 a detects the first face area of the subject in each of the plurality of first image data acquired by the acquisition unit 17. The face information extraction unit 20b extracts first face information related to the shape of the constituent elements of the face of the subject from each of the plurality of first face regions detected by the face region detection unit 20a.

  Further, the face area detection unit 20 a detects the second face area of the subject in the second image data acquired by the acquisition unit 17. The face information extraction means 20b extracts second face information related to the shape of the constituent elements of the subject's face from the second face area detected by the face area detection means 20a.

  The memory control circuit 22 controls the A / D conversion unit 16, the timing generation unit 18, the image processing unit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / decompression unit 32. The memory control circuit 22 writes the data received from the A / D conversion unit 16 and the image processing unit 20 in the image display memory 24 or the memory 30.

  The image display memory 24 stores image data for display. The D / A converter 26 receives display image data from the image display memory 24 via the memory control unit 22 and D / A converts the display image data into an analog signal.

  The first notification means 28 displays an image corresponding to the analog signal received from the D / A converter 26. The first notification means 28 includes, for example, a TFT-LCD. The first notification means 28 can display images sequentially to realize an electronic finder function.

  Here, the first notification means 28 can arbitrarily turn on / off the display according to an instruction from the system control unit 50, and greatly reduce the power consumption of the digital camera 100 when the display is turned off. I can do it.

  The memory 30 stores captured still images and moving images. The memory 30 has a storage capacity sufficient to store a predetermined number of still images and a moving image for a predetermined time. Thereby, even in the case of continuous shooting or panoramic shooting in which a plurality of still images are continuously shot, it is possible to write a large amount of images to the memory 30 at high speed. The memory 30 can also be used as a work area for the system control unit 50.

  The compression / decompression unit 32 compresses and decompresses image data by adaptive discrete cosine transform (ADCT) or the like. The compression / decompression unit 32 reads an image stored in the memory 30, performs compression processing or decompression processing, and writes the processed data to the memory 30.

  The exposure control means 40 controls the shutter 12 having an aperture function. The exposure control means 40 can realize a flash light control function by cooperating with the flash 48.

  The distance measurement control means 42 controls the focusing of the taking lens 10. The zoom control unit 44 controls zooming of the taking lens 10. The barrier control unit 46 controls the operation of the protection unit 102. The flash 48 has an AF auxiliary light projecting function and a flash light control function.

  The system control unit 50 controls the entire digital camera 100. For example, the system control unit 50 controls the exposure control unit 40 and the distance measurement control unit 42 based on the calculation result calculated by the image processing unit 20 on the image data acquired by the acquisition unit 17.

  The system control unit 50 includes a face information selection unit 50a, a facial expression estimation unit 50b, and a control unit 50c. The face information selection unit 50a selects reference face information that serves as a reference for estimating the facial expression of the subject from the plurality of first face information extracted by the face information extraction unit 20b. The reference face information is, for example, face information for an expression in an expressionless state, that is, a clear expression. Here, the face information selection unit 50a determines that the expression is in an expressionless state, for example, when the amount of change in the shape of the mouth does not exceed a predetermined threshold for a predetermined period, and the face corresponding to the determination result. Information is selected as reference face information.

  The face information selection unit 50a may use the shape of a component other than the mouth when determining the expressionless state. For example, the face information selection means 50a may further use the eye open / closed state as the face information and select the reference face information based on the eye opening state timing using a change in the size of the eyes. In addition, the face information selection unit 50a may learn a pupil image with a neural network and determine the open / closed state of the eye from the number of pupil neurons greater than a fixed threshold.

  The facial expression estimation means 50b compares the reference face information and the second face information, and estimates the facial expression of the subject corresponding to the second face information. That is, the facial expression estimation means 50b receives the reference face information from the face information selection means 50a, and receives the second face information from the face information extraction means 20b of the image processing unit 20. The facial expression estimation means 50b calculates the difference between the reference face information and the second face information, and generates change amount information indicating the change amount of the second face information with respect to the reference face information. The facial expression estimation means 50b receives threshold information indicating a change amount threshold for determining that the facial expression to be detected has been reached from the memory 52 described later. The facial expression estimation means 50b determines whether or not the change amount exceeds the threshold value based on the change amount information and the threshold value information. Thereby, the facial expression estimation means 50b determines whether or not the face of the target subject has reached the target facial expression.

  The control unit 50c controls the acquisition unit 17 so as to acquire the first image data at the first resolution and acquire the second image data at the second resolution. Specifically, the control unit 50 c controls at least one of the photoelectric conversion unit 14 and the reading unit 15 by controlling a signal supplied from the timing generation unit 18 to at least one of the photoelectric conversion unit 14 and the reading unit 15. . That is, the control unit 50c reads at least one of the photoelectric conversion unit 14 and the reading unit 15 so as to read the first image signal with the first number of pixels in order to obtain the first image data with the first resolution. Control. In addition, the control unit 50c reads the second image signal with the second number of pixels smaller than the first number of pixels in order to obtain the second image data with the second resolution. At least one of the reading means 15 is controlled.

  The memory 52 stores constants and variables for the operation of the system control unit 50. Further, the memory 52 stores threshold information received from the input means 75 described later as setting information in advance.

  The second notification unit 54 notifies the operation state, the message, and the like using characters, images, sounds, and the like according to the execution of the program in the system control unit 50. One or a plurality of second informing means 54 are installed at positions in the vicinity of an operation unit 70 (to be described later) of the digital camera 100 that are easily visible. The second notification means 54 is configured by a combination of a liquid crystal display (LCD), an LED, a sounding element (speaker), and the like, for example.

  The second notification means 54 is partially installed in the optical viewfinder 104.

  For example, the second notifying unit 54 includes a single shot / continuous shooting display, a self-timer display, a compression ratio display, a recording pixel number display, a recording number display, a remaining image number display, a shutter speed display, an aperture value display, and An exposure compensation display or the like is displayed on an LCD or the like. Alternatively, the second notification unit 54 may be, for example, a flash display, a red-eye reduction display, a macro shooting display, a buzzer setting display, a clock battery remaining amount display, a battery remaining amount display, an error display, and a multi-digit number information display. Are displayed on the LCD. Or the 2nd alerting | reporting means 54 displays the attachment / detachment state display of the external recording medium 120, communication I / F operation display, date / time display, etc. on LCD etc., for example.

  Further, the second notification means 54 displays in-focus display, camera shake warning display, flash charge display, shutter speed display, aperture value display, exposure correction display, and the like in the optical viewfinder 104.

  The nonvolatile memory 56 is an electrically erasable / recordable memory, and stores a program such as an image processing program. As the nonvolatile memory 56, for example, an EEPROM or the like is used.

  The input means 75 accepts an extraction target instruction that specifies a subject from which face information is to be extracted from a plurality of subjects. Alternatively, the input means 75 accepts a facial expression detection instruction for detecting the facial expression of the subject. Alternatively, the input means 75 accepts a change instruction for changing the change amount threshold value for determining that the facial expression to be detected has been reached.

  Further, the input unit 75 receives various operation instructions from the system control unit 50. A predetermined instruction is received through one or a combination of a switch, a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, and the like. The input means 75 includes a mode dial switch 60, a shutter button 61, a first shutter switch 62, a second shutter switch 64, an image display ON / OFF switch 66, and an operation unit 70.

  The mode dial switch 60 receives instructions for switching and setting each function mode such as power-off, automatic shooting mode, shooting mode, panoramic shooting mode, playback mode, multi-screen playback / erase mode, and PC connection mode.

  The shutter button 61 receives an instruction for taking a still image from the user. For example, when the shutter button 61 is pressed halfway, AF (auto focus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, EF (flash pre-flash) processing, and the like are performed. Accept instructions. For example, when the shutter button 61 is fully pressed, the second instruction for taking a still image is received.

  When the first shutter switch (SW1) 62 receives the first instruction from the shutter button 61, the first shutter switch (SW1) 62 is turned on, and supplies information indicating the on state to the system control unit 50. The system control unit 50 performs AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing in response to the first shutter switch 62 being in the ON state. The operation start is instructed to each part.

  When the second shutter switch (SW2) 64 receives the second instruction from the shutter button 61, the second shutter switch (SW2) 64 is turned on, and supplies information indicating the on state to the system control unit 50. As a result, the system control unit 50 instructs the start of a series of photographing processing operations. In a series of photographing processes, an exposure process in which the signal read from the photoelectric conversion unit 14 is written into the storage unit 30 via the A / D conversion unit 16 and the memory control unit 22, the image processing unit 20 and the memory control unit 22 Development processing using the above calculation is performed. In a series of photographing processes, image data is read from the storage unit 30, compressed by the compression / decompression unit 32, and recording processing for writing the image data to the recording medium 200 or 210 is performed.

  The image display ON / OFF switch 66 receives an instruction for setting ON / OFF of the first notification means 28. By this instruction, the system control unit 50 can cut power supply to the first notifying unit 28 when taking an image using the optical viewfinder 104, thereby saving power.

  The quick review ON / OFF switch 68 accepts an instruction for setting a quick review function for automatically reproducing image data taken immediately after photographing. For example, it is assumed that the setting of the quick review function when the first notification unit 28 is turned off can be received.

  The operation unit 70 includes various buttons and a touch panel. The operation unit 70 includes a menu button, a macro button, a multi-screen playback page break button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, a menu movement + (plus) button, and a menu movement− (minus) button. . The operation unit 70 includes a playback image movement + (plus) button, a playback image-(minus) button, a shooting image quality selection button, an exposure correction button, a date / time setting button, a bracket mode selection button, and the like.

  The power supply control means 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether or not a battery is attached, the type of battery, and the remaining battery level. The power control unit 80 controls the DC-DC converter based on the detection result and an instruction from the system control unit 50, and supplies a necessary voltage to each unit including the external recording medium 120 for a necessary period.

  The connector 82 is connected to the power supply control means 80. The connector 84 is connected to the power source 86. The power source 86 is, for example, a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter.

  The card controller 90 transmits / receives data to / from an external recording medium such as a memory card. The interface 91 functions as an interface between the external recording medium 120 such as a memory card and the card controller 90. The connector 92 is connected to an external recording medium 120 such as a memory card. The recording medium attachment / detachment detection means 98 detects whether or not the external recording medium 120 is attached to the connector 92.

  Note that the number of interfaces and connectors for attaching the recording medium may be two or more. Further, interfaces and connectors having different standards may be combined. The interface and the connector may be configured using a semiconductor memory card or the like that conforms to the standard. In this case, by connecting various communication cards, image data and management information attached to the image data can be transferred to and from other computers and peripheral devices such as a printer. The various communication cards are, for example, LAN cards, modem cards, USB cards, IEEE 1394 cards, P1284 cards, SCSI cards, PHS communication cards, and the like.

  The optical viewfinder 104 is used for confirming a subject when shooting. If the optical viewfinder 104 is used, it is possible to perform shooting without using the electronic viewfinder function of the first notification means 28. The optical viewfinder 104 also includes a part of information displayed by the second notification unit 54, for example, focusing display, camera shake warning display, flash charge display, shutter speed display, aperture value display, exposure correction display, and the like. Information is displayed.

  The external recording medium 120 is detachably connected to the connector 92. The external recording medium 120 is, for example, a memory card.

  Next, the flow of processing when the image processing apparatus 100 estimates the facial expression of the subject (smiling shooting mode processing) will be described with reference to the flowchart shown in FIG. FIG. 2 is a flowchart showing a flow of processing when the image processing apparatus 100 estimates the facial expression of the subject (smiling shooting mode processing).

  In step S1 (first acquisition step), the input means 75 accepts an instruction for selecting a smile shooting mode. This smile shooting mode is a shooting mode in which shooting is automatically performed when a smile on a specific subject is detected. The acquisition unit 17 acquires first image data obtained by imaging a subject at a plurality of different timings at a first resolution.

  That is, the control unit 50c reads at least one of the photoelectric conversion unit 14 and the reading unit 15 so as to read the first image signal with the first number of pixels in order to obtain the first image data with the first resolution. Control. The photoelectric conversion unit 14 photoelectrically converts the optical image of the subject to generate a first image signal. The readout unit 15 reads out the first image signal from the photoelectric conversion unit 14 with the first number of pixels. The A / D conversion unit 16 A / D converts the first image signal read by the reading unit 15 to generate first image data. The resolution of the first image data is the first resolution. These operations are performed at a plurality of different timings, and a plurality of first image data are acquired at the first resolution.

  In step S2 (first face area detection step), the input unit 75 receives an instruction for performing face area detection from the user, and supplies the instruction to the face area detection unit 20a. The face area detection unit 20a detects the first face area of the subject in each of the plurality of first image data acquired by the acquisition unit 17 in response to the instruction. The face area detection unit 20 a supplies information related to the detected first face area to the system control unit 50.

  In step S3, a preparation process is performed. Details of the preparation process will be described later.

  In step S4, the system control unit 50 determines whether or not an expression should be detected.

  For example, the system control unit 50 determines that a facial expression should be detected when a facial expression detection instruction for detecting the facial expression of the subject is received from the input means 75. The system control unit 50 determines that the facial expression should not be detected when the facial expression detection instruction for detecting the facial expression of the subject is not received from the input unit 75.

  Alternatively, for example, the system control unit 50 determines that the facial expression should be detected when the reference face information is selected for all of the target subjects. The system control unit 50 determines that the facial expression should not be detected when there is a subject for which the reference face information is not selected among the subject subjects.

  If the system control unit 50 determines that the facial expression should be detected, the process proceeds to step S5. If the system control unit 50 determines that the facial expression should not be detected, the process proceeds to step S1.

  In step S5 (second acquisition step), in order for the acquisition means 17 to estimate the facial expression of the subject, the second image data obtained by imaging the subject is set to the second resolution lower than the first resolution. Acquire at a resolution of 2. Here, when the acquisition unit 17 determines that the second face information has not been successfully extracted in the detection process (step S7) described later, the resolution is higher than the second resolution used in the previous process. Thus, a resolution lower than the first resolution is newly set as the second resolution.

  That is, in order to obtain the second image data at the second resolution, the control unit 50c reads the second image signal with the second number of pixels smaller than the first number of pixels, and the photoelectric conversion unit 14 and At least one of the reading means 15 is controlled. Here, when it is determined that the second face information is not successfully extracted in the detection process (step S7) described later, the control unit 50c is higher than the second number of pixels used in the previous process. The number of pixels is newly set as the second number of pixels.

  The photoelectric conversion means 14 photoelectrically converts the optical image of the subject to generate a second image signal. The reading unit 15 reads the second image signal from the photoelectric conversion unit 14 with a second number of pixels smaller than the first number of pixels. The A / D conversion unit 16 A / D converts the second image signal read by the reading unit 15 to generate second image data. The resolution of the second image data is the second resolution.

  In step S6 (second face area detecting step), the face area detecting unit 20a detects the face area of the subject in each of the plurality of image data acquired by the acquiring unit 17. The face area detection unit 20 a supplies information related to the detected face area to the system control unit 50.

  In step S7, a detection process is performed. Details of the detection process will be described later.

  In step S8, the system control unit 50 determines whether or not shooting should be performed.

  For example, the system control unit 50 determines that shooting should be performed when the number of subjects with the facial expression flag turned on is equal to or greater than a predetermined number, and the number of subjects with the facial expression flag turned on is less than the predetermined number. , Judge that you should not shoot.

  Alternatively, for example, when the information indicating that the display flag is ON is received from the second shutter switch 64, the system control unit 50 determines that the image should be taken. For example, when the information indicating that the display flag is in the ON state is not received from the second shutter switch 64, the system control unit 50 determines that shooting should not be performed.

  If the system control unit 50 determines that the image should be taken, the process proceeds to step S9. If the system control unit 50 determines that the image should not be taken, the process proceeds to step S10.

  In step S9, the system control unit 50 instructs the start of a series of shooting processing operations. In a series of photographing processes, an exposure process in which a signal read from the image sensor 14 is written into the storage means 30 via the A / D converter 16 and the memory control unit 22, and the image processing unit 20 and the memory control unit 22 Development processing using computation is performed. In a series of photographing processes, image data is read from the storage unit 30, compressed by the compression / decompression unit 32, and recording processing for writing the image data to the recording medium 200 or 210 is performed.

  In step S10, the system control unit 50 determines whether or not the smile shooting mode process should be terminated.

  For example, if the system control unit 50 determines in the detection process (step S7) that the second face information has not been successfully extracted, the system control unit 50 determines that the smile shooting mode process should not be terminated. If the system control unit 50 determines that the second face information has been successfully extracted in the detection process (step S7), the system control unit 50 determines that the smile shooting mode process should be terminated.

  For example, when the system control unit 50 receives an instruction to change the shooting mode to a shooting mode other than the smile shooting mode, the system control unit 50 determines that the process of the smile shooting mode should be terminated. When the system control unit 50 does not receive an instruction to change the shooting mode to a shooting mode other than the smile shooting mode, the system control unit 50 determines that the process of the smile shooting mode should not be terminated.

  Alternatively, for example, when receiving an instruction to end the smile shooting mode, the system control unit 50 determines that the process of the smile shooting mode should be ended. When the system control unit 50 does not receive an instruction to end the smile shooting mode, the system control unit 50 determines that the process of the smile shooting mode should not be ended.

  Next, the flow of the preparation process (step S3) will be described with reference to FIG. FIG. 3 is a flowchart showing the flow of preparation processing for each subject. FIG. 3 shows processing when attention is paid to a specific subject (for example, the subject O1 shown in FIG. 5). That is, the preparation process is performed in parallel independently for each subject.

  In step S11, the face information extraction unit 20b determines whether or not the specific subject is the subject.

  For example, when the face information extraction unit 20b receives an extraction target instruction for a specific subject from the input unit 75, the face information extraction unit 20b determines that the specific subject is a target subject. When the face information extraction unit 20b does not receive an extraction target instruction for a specific subject from the input unit 75, the face information extraction unit 20b determines that the specific subject is not the target subject.

  When it is determined that the specific subject is the target subject, the face information extraction unit 20b advances the process to step S12, and when it is determined that the specific subject is not the target subject, the processing ends.

  In step S12 (first face information extraction step), the face information extraction unit 20b performs first processing on the shape of the face component of the subject from each of the plurality of first face regions detected by the face region detection unit 20a. 1 face information is extracted.

  For example, the face information extraction unit 20b normalizes the size and orientation of the selected face in order to increase the calculation accuracy in the subsequent steps. The face information extraction unit 20b performs normalization processing by affine transformation using the positions of both eyes so that the distance between both eyes is a predetermined distance and the face direction is a predetermined direction. The face information extraction unit 20b detects feature points representing the shape of the end points such as the mouth, eyes, and eyebrows from the normalized face. Then, the face information extraction unit 20b calculates a first feature amount from each detected feature point. That is, the face information extraction unit 20b extracts first face information including the first feature amount of the face of the subject from each of the plurality of first face regions.

  Here, as shown in FIG. 5, for example, the first face information includes information on specific shapes and coordinates of feature points 500 to 517 such as a mouth, eyes, and eyebrows as a first feature amount. The first face information is extracted from a face image to which each end point, center point, eye black point, or the like is input by using, for example, a technique such as edge detection using a neural network or a spatial filter. be able to. That is, the face information extraction unit 20b detects each feature point by setting a predetermined detection range for each feature point and using a neural network for detecting a feature amount within each detection range. As the neural network, each network learned in advance using each feature point image data is used in the same manner as face detection.

  For example, the detection range for each feature point is set as follows. For example, as shown in FIG. 6, the face information extraction unit 20 b uses the interocular distance a and the left eye position 400 to detect the detection ranges 600 and 601 with respect to the feature points 500 and 502 (see FIG. 5). Set. The face information extraction unit 20b also sets a detection range for the feature points 503 and 505 (see FIG. 5) using the interocular distance a and the right eye position 401 in the same manner as the feature points 500 and 502. The face information extraction unit 20b sets a detection range 602 as shown in FIG. 7 for the feature point 501 (see FIG. 5). The face information extraction unit 20b also sets a detection range for the feature point 504 (see FIG. 5) in the same manner as the feature point 501. The face information extraction unit 20b sets detection ranges 603 and 604 for the feature points 506 and 509 (see FIG. 5) as shown in FIG. The face information extraction unit 20b sets a detection range for the feature points 510 and 513 as well as the feature points 506 and 509. The face information extraction unit 20b sets detection ranges 605 and 606 for the feature points 507 and 508 (see FIG. 5) as shown in FIG. The face information extraction unit 20b also sets detection ranges for the feature points 511 and 512 (see FIG. 5) in the same manner as the feature points 507 and 508. The face information extraction unit 20b sets detection ranges 607 and 608 for the feature points 514 and 517 (see FIG. 5) as shown in FIG. The face information extraction unit 20b sets detection ranges 609 and 610 for the feature points 515 and 516 (see FIG. 5) as shown in FIG.

  The feature amount calculated by the face information extraction unit 20b is, for example, the following information. A feature amount of a smile expression will be described as an example of the feature amount.

  The smile expression is proposed in the Facial Action Coding System (FACS) as follows. (P. Ekman and WV Friesen, Facial Action Coding System (FACS) Manual, Palo Alto: Consulting Psychologists Press, 1978). “Defining a smiling facial expression in FACS” includes “lifting the cheek” and “pulling the edge of the lips”.

  In the present embodiment, the face information extraction unit 20b calculates the feature amounts 710 to 712 shown in FIG. 12 according to the FACS definition as the feature amount of the smile expression. That is, the face information extraction unit 20b calculates the distance 710 in the y direction between the feature point 506 and the feature point 514. The face information extraction unit 20b calculates a distance 711 between the feature point 513 and the feature point 517 in the y direction. The face information extraction unit 20b calculates a distance 712 between the feature point 514 and the feature point 517 in the x direction. The distances 710 to 712 are all relative to a reference distance (for example, the interocular distance a).

  In step S13, the face information extraction unit 20b determines whether or not the face information has been successfully extracted. If it is determined that the face information extraction unit 20b has succeeded in extracting the face information, the process proceeds to step S15. If it is determined that the face information extraction has not been successful, the process ends.

  In step S15 (first face information selection step), the face information selection unit 50a uses a plurality of pieces of face information extracted by the face information extraction unit 20b as a reference face for estimating the facial expression of the subject. Select information. The reference face information is, for example, face information for an expression in an expressionless state, that is, a clear expression. Here, the face information selection unit 50a determines that the expression is in an expressionless state, for example, when the amount of change in the shape of the mouth does not exceed a predetermined threshold for a predetermined period, and the face corresponding to the determination result. Information is selected as reference face information.

  The face information selection unit 50a may use the shape of a component other than the mouth when determining the expressionless state. For example, the face information selection means 50a may further use the eye open / closed state as the face information and select the reference face information based on the eye opening state timing using a change in the size of the eyes. In addition, the face information selection unit 50a may learn a pupil image with a neural network and determine the open / closed state of the eye from the number of pupil neurons greater than a fixed threshold.

  Next, the flow of the detection process (step S7) will be described with reference to FIG. FIG. 4 is a flowchart showing the flow of detection processing for each subject. That is, the detection process is performed independently and in parallel for each subject. FIG. 4 shows a process when attention is paid to a specific subject.

  In step S23 (second face information extraction step), the face information extraction unit 20b uses the second face information related to the shape of the face component of the subject from the second face region detected by the face region detection unit 20a. To extract. That is, the face information extraction unit 20b extracts second face information including the second feature amount of the face of the subject from the second face area. The details of step S23 are the same as step S12 described above.

  Note that the face information extraction unit 20b does not detect all the feature points to be detected due to insufficient resolution of the acquired second image data, but if two or more feature points are detected, A second feature amount is calculated using the two or more feature points.

  In step S24, the face information extraction unit 20b determines whether the second face information has been successfully extracted.

  For example, the face information extraction unit 20b may detect, for example, two or more feature points even when not all of the feature points to be detected are detected due to insufficient resolution of the acquired second image data. For example, it is determined that the second face information has been successfully extracted. The number of feature points that are determined to be successful in extracting the second face information is set to be smaller than the number of feature points that are determined to be successful in extracting the first face information.

  If the face information extracting unit 20b determines that the second face information has been successfully extracted, the process proceeds to step S25, and if it is determined that the second face information has not been successfully extracted, The process ends.

  In step S25, the facial expression estimation means 50b receives the reference face information from the face information selection means 50a, and receives the second face information from the face information extraction means 20b of the image processing unit 20. The facial expression estimation means 50b calculates the difference between the reference face information and the second face information, and generates change amount information indicating the change amount of the second face information with respect to the reference face information.

For example, the facial expression estimation means 50b calculates the difference between the first feature amount included in the reference face information and the second feature amount included in the second face information for each component of the face. Then, the facial expression estimation means 50b calculates the smile facial expression level, that is, the smile level, using the following formula 1. When the difference between the reference face information for each component of the subject's face and other face information is v1, v2, v3,..., The facial expression estimation means 50b
SumScore = ΣScore _i
= Σg (w _i , v _i )...
And i = 1 to N are calculated in total. Here, N is a number for identifying each component of the face, w is the weight of each component, and g is a score calculation function. That is, the facial expression estimation means 50b generates this smile level SumScore as change amount information. This smile degree SumScore is represented by a numerical value such as 0 to 100, for example.

  The facial expression estimation means 50b calculates the amount of change in the feature amount using only the calculated feature amount even when not all of the feature points to be detected are detected due to insufficient resolution of the acquired second image data. Calculate the smile level. For example, when the distance 710 cannot be calculated from the distances 710, 711, and 712 that are smile feature amounts, the degree of smile is calculated using only the distances 711 and 712. Further, the facial expression estimation means 50b may calculate a ratio between the reference face information and other face information, and generate change amount information indicating a change amount of the other face information with respect to the reference face information.

  In step S27 (expression estimation step), the expression estimation means compares the reference face information with the second face information, and estimates the expression of the face of the subject corresponding to the second face information. That is, the facial expression estimation means 50b determines whether or not the facial expression to be detected has been reached.

  For example, the facial expression estimation means 50b receives threshold value information indicating the threshold value of the change amount for determining that the facial expression to be detected has been reached from the memory 52 described later. The facial expression estimation means 50b determines whether or not the change amount exceeds the threshold value based on the change amount information and the threshold value information. When it is determined that the amount of change exceeds the threshold value, the facial expression estimation means 50b determines that the facial expression to be detected has been reached for the subject. When it is determined that the amount of change does not exceed the threshold value, the facial expression estimation unit 50b determines that the facial expression to be detected has not reached the target subject.

  When it is determined that the facial expression to be detected has been reached, the facial expression estimating means 50b advances the process to step S29, and when it is determined that the facial expression to be detected has not been reached, the processing is terminated.

  In step S29, the facial expression estimation means 50 accesses the memory 52 and rewrites the facial expression flag of the subject that has reached the target facial expression from the OFF state to the ON state.

  As described above, the second resolution of the second image data for estimating the facial expression of the subject is lower than the first resolution of the first image data for acquiring the reference face information. Yes.

  That is, since the resolution of the second image data for estimating the facial expression of the subject is low, the processing time for estimating the facial expression of the subject can be reduced, so that the facial expression of the subject can be estimated at high speed. Can do.

  In addition, since the resolution of the first image data for acquiring the reference face information is high, it is possible to avoid a decrease in the number of feature points extracted for obtaining the reference face information (first feature amount). it can. Thereby, even if the feature point included in the second face information obtained from the second image data fluctuates, the accuracy of estimating the facial expression of the subject can be maintained with high accuracy. Note that the first image data and the second image data having different resolutions are generated by changing the number of pixels read from the photoelectric conversion unit 14, but other methods may be used. For example, the first image data generated by A / D conversion may be resized to generate first image data and second image data having a different resolution.

  Therefore, the facial expression of the subject can be estimated at high speed and with high accuracy. As a result, it is possible to shoot without missing a photo opportunity when it is determined that the expression is good.

  Moreover, since the control signal for estimating the facial expression of the subject can be reduced, the power for estimating the facial expression of the subject can be reduced.

  Next, a program for realizing the functions of the above-described embodiment will be described.

  Software program codes (programs) that operate various devices so as to realize the functions of the above-described embodiments, and realize the functions of the above-described embodiments for a computer in an apparatus or system connected to the various devices. ).

  A device in which various devices described above are operated in accordance with a program (software or hardware) stored in a computer (CPU or MPU) of the system or apparatus is also included in the scope of the present invention. include.

  In this case, the software program itself realizes the functions of the above-described embodiment.

  The program itself and means for supplying the program code of the program to a computer, for example, a storage medium storing such a program are also included in the scope of the present invention.

  As a storage medium for storing such a program, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the functions of the above-described embodiments are not only realized by the computer executing the supplied program.

  For example, when the program realizes the functions of the above-described embodiment in cooperation with an OS (operating system) running on a computer or another application, the program is also included in the scope of the present invention. .

  Further, the supplied program is stored in a memory provided in a function expansion board of the computer or a function expansion unit connected to the computer.

  Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and when the functions of the above-described embodiments are realized by the processing, Such a program is included in the category of the present invention.

1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention. 5 is a flowchart showing a flow of processing (smiling shooting mode processing) when the image processing apparatus according to the embodiment of the present invention estimates a facial expression of a subject. 6 is a flowchart showing a flow of preparation processing for each subject in the embodiment of the present invention. The flowchart which shows the flow of the detection process for every to-be-photographed object in embodiment of this invention. The figure which shows the feature point to detect in embodiment of this invention. The figure which shows the detection range in embodiment of this invention. The figure which shows the detection range in embodiment of this invention. The figure which shows the detection range in embodiment of this invention. The figure which shows the detection range in embodiment of this invention. The figure which shows the detection range in embodiment of this invention. The figure which shows the detection range in embodiment of this invention. The figure which shows the feature-value in embodiment of this invention.

Explanation of symbols

10: Photography lens
12: Shutter
14: Image sensor
16: A / D converter
18: Timing generation circuit
20: Image processing circuit 22: Memory control circuit
24: Image display memory
26: D / A converter
28: Image display section
30: Memory
32: Image compression / decompression circuit 40: Exposure control means
42: Ranging control means
44: Zoom control means
46: Barrier control means
48: Flash
50: System control circuit
52: Memory
54: Display section
56: Nonvolatile memory 60: Mode dial switch
62: Shutter switch SW1
64: Shutter switch SW2
66: Image display ON / OFF switch
68: Quick review ON / OFF switch
70: Operation unit 80: Power control means
82: Connector
84: Connector
86: Power supply means 90: Card controller
91: Interface
92: Connector
98: Recording medium attachment / detachment detecting means 100: Digital camera
102: Protection means
104: Optical viewfinder 120: External recording medium

Claims (6)

Obtaining means for obtaining first image data obtained by imaging a subject at a plurality of different timings at a first resolution;
Face area detection means for detecting a first face area of the subject in each of the plurality of first image data acquired by the acquisition means;
Face information extracting means for extracting, from each of the plurality of first face areas detected by the face area detecting means, first face information relating to the shape of the constituent elements of the face of the subject;
Face information selection means for selecting reference face information serving as a reference for estimating the facial expression of the subject from the plurality of first face information extracted by the face information extraction means;
A facial expression estimating means for estimating a facial expression of the subject based on the reference face information;
With
The acquisition means acquires second image data obtained by imaging the subject after the reference face information is selected, at a second resolution lower than the first resolution,
The face area detecting unit detects a second face area of the subject in the second image data acquired by the acquiring unit;
The face information extracting means extracts second face information related to the shape of the constituent element of the face of the subject from the second face area detected by the face area detecting means;
The image processing apparatus, wherein the facial expression estimation means estimates the facial expression of the subject corresponding to the second face information by comparing the reference face information and the second face information. When the feature amount included in the second face information is less than the feature amount included in the first face information, the facial expression estimation means includes the feature amount included in the second face information, and the first face information The image according to claim 1, wherein the facial expression of the subject is estimated by comparing a feature amount included in the face information with a feature amount corresponding to the feature amount included in the second face information. Processing equipment. The acquisition means includes
Photoelectric conversion means for photoelectrically converting the optical image of the subject to generate an image signal;
Reading means for reading out an image signal from the photoelectric conversion means;
A / D conversion means for A / D converting the image signal read by the reading means to generate image data;
The image processing apparatus according to claim 1, further comprising: In order to obtain the first image data with the first resolution, the first image signal is read with the first number of pixels, and the second image data is obtained with the second resolution by using the second resolution. 4. The apparatus according to claim 3, further comprising a control unit that controls at least one of the photoelectric conversion unit and the reading unit so that the image signal is read with a second number of pixels smaller than the first number of pixels. An image processing apparatus according to 1. A first acquisition step of acquiring first image data obtained by imaging a subject at a plurality of different timings at a first resolution;
A first face area detecting step of detecting a first face area of the subject in each of the plurality of first image data acquired in the first acquiring step;
A first face information extracting step of extracting first face information relating to the shape of the constituent element of the face of the subject from each of the plurality of first face areas detected in the first face area detecting step; ,
A first face information selection step of selecting reference face information serving as a reference for estimating the facial expression of the subject from the plurality of first face information extracted in the first face information extraction step; ,
After the reference face information is selected in the first face information selection step, second image data obtained by imaging the subject is acquired at a second resolution lower than the first resolution. A second acquisition step;
A second face area detecting step of detecting a second face area of the subject in the second image data acquired in the second acquiring step;
A second face information extracting step of extracting second face information related to the shape of the constituent element of the face of the subject from the second face area detected in the second face area detecting step;
A facial expression estimation step of comparing the reference face information and the second facial information to estimate the facial expression of the subject corresponding to the second facial information;
An image processing method comprising: In the image processing device,
A first acquisition step of acquiring first image data obtained by imaging a subject at a plurality of different timings at a first resolution;
A first face area detecting step of detecting a first face area of the subject in each of the plurality of first image data acquired in the first acquiring step;
A first face information extracting step of extracting first face information relating to the shape of the constituent element of the face of the subject from each of the plurality of first face areas detected in the first face area detecting step; ,
A first face information selection step of selecting reference face information serving as a reference for estimating the facial expression of the subject from the plurality of first face information extracted in the first face information extraction step; ,
After the reference face information is selected in the first face information selection step, second image data obtained by imaging the subject is acquired at a second resolution lower than the first resolution. A second acquisition step;
A second face area detecting step of detecting a second face area of the subject in the second image data acquired in the second acquiring step;
A second face information extracting step of extracting second face information related to the shape of the constituent element of the face of the subject from the second face area detected in the second face area detecting step;
A facial expression estimation step of comparing the reference face information and the second facial information to estimate the facial expression of the subject corresponding to the second facial information;
An image processing program for executing

Images