JP2010050933A - Imaging device and red-eye correction method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform red-eye detection accurately for improving a red-eye correction factor even when an object or a photographer is moving. <P>SOLUTION: The imaging device has a face detection means (38) to perform processing for detecting a human face from the image represented by the image signal, a red-eye detection means (38) to perform processing for detecting a red-eye from the detected face, a development means (50) to perform processing for development including at least luminance and color correction for the image signal, a red-eye correction means (50) to correct the detected red-eye, and a system control section (60). The system control section makes the face detection means perform processing for detecting a face from the image signal before processing for development is performed, makes the development means perform processing for development on the basis of the detected face when the face is detected, makes the face detection means perform processing for detecting a face from the developed image signal, and makes the red-eye detection means perform processing for detecting the red-eye from the face detected from the developed image signal when the face is detected from the developed image signal. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging device and a red-eye correction method, and more particularly to an imaging device that performs red-eye correction and a red-eye correction method used in the imaging device.

  Regardless of the old film camera or the current mainstream digital camera, a red-eye phenomenon is known in which a person's eyes are photographed in red when shooting with flash in the dark. This is a phenomenon that occurs because the blood vessels of the retina are photographed by flash light in a dark place where the human pupil is wide open.

  As one method for alleviating and reducing this red-eye phenomenon, there is a method of projecting light to a person before photographing and closing a wide open pupil. As the light projecting device, a xenon tube used for flash photography or an LED used for distance measurement may be used. In any projector, it is assumed that the subject is gazing at the camera, and even if it is gazing, red eyes may occur unless the pupil is fully closed. Was a way of remaining challenges.

  On the other hand, a digital camera is very different from a film camera, and can perform photo development processing in the camera. By reading out the subject image exposed by the image sensor and performing development processing by the image processing device, the image is displayed on an image display device such as an LCD, and the developed image is stored in an image storage medium. The captured image can be displayed on an external device such as a PC. In this in-camera development process, the human face is detected and whether the red eye phenomenon appears in the detected face, and if it is red eye, the red eye phenomenon is eliminated by performing image processing to paint the red eye black. The way to be done.

  For example, Patent Document 1 proposes to perform red-eye correction using a low-resolution image for thumbnails in order to reduce the time required for face detection performed during red-eye correction. In Patent Document 2, in order to detect at a high speed that a red-eye phenomenon has occurred in a photographic image, detection is performed by changing the face detection conditions for each image area.

Japanese Patent Laying-Open No. 2005-167697 JP 2005-322220 A

  In order to detect whether or not a red-eye phenomenon has occurred on the face of a person during flash photography, luminance information and color information are separated from the captured image to form a so-called YUV format image. A face area is detected from the YUV format image, and red of the red-eye phenomenon is detected from the detected eye area of the face.

  When detecting a face, the brightness and contrast of the face are properly reproduced, and when red-eye is detected, the color is appropriately reproduced with higher detection accuracy. It is preferable that the image is selected.

  However, the brightness, contrast, and color of the image may not be reproduced properly. This depends on the shooting sequence of the camera that performs shooting processing after performing face detection before shooting a still image, determining exposure conditions such as aperture value, shutter speed, ISO sensitivity, and flash light emission intensity. For example, if the subject or the photographer is moving, the situation of the subject at the time of shooting condition determination and the situation of the subject at the time of actual shooting may differ, and the brightness of the image will not be reproduced properly, In conjunction with this, colors may not be appropriate.

  As described above, when face detection or red-eye detection is performed on an image whose luminance or color is not appropriate, detection cannot be performed with high accuracy, which is one of the factors that reduce the red-eye correction rate. It was.

  With the method disclosed in Patent Document 1, it is possible to reduce the time as the number of pixels of an image sensor increases. However, since a low-resolution image is used, it is necessary to accurately determine the position of a person's eyes. There is a method that is not suitable for red-eye correction.

  Further, Patent Document 2 is not suitable as a countermeasure for the case where the brightness and color are not appropriate in the state where the subject and the photographer are moving as described above.

  The present invention has been made in view of the above problems, and it is an object of the present invention to perform red-eye detection with high accuracy and improve the red-eye correction rate even when the subject or the photographer is moving.

  It is a further object to keep the time taken for red-eye detection short.

  In order to achieve the above object, an imaging apparatus according to the present invention performs processing for detecting a human face from an image signal, and processing for detecting red eyes from a face detected by the face detecting unit. A red-eye detecting unit; a developing unit that performs development processing including at least luminance and color correction on the image signal; a red-eye correcting unit that corrects a red-eye detected by the red-eye detecting unit; and a control unit. The control unit controls the face detection unit to perform processing for detecting a face from the image signal. When the face is detected by the face detection unit, the control unit controls the development unit, Based on the detected face, the development process is performed, the face detection unit is controlled to perform a process of detecting a face from the developed image signal, and a face is detected from the developed image signal If It controls the eye detection means, to perform processing of detecting a red-eye from the detected face from developing processed image signal.

  The imaging apparatus further includes an imaging element that converts an optical image of a subject into an image signal, and the control unit performs processing for detecting a face from the image signal before the development processing from the imaging element. The face detection means is controlled so as to be performed in parallel with the process of reading the image signal.

  The red-eye correction method according to the present invention includes a first face detection process for performing processing for detecting a human face from an image signal, and the face detected in the first face detection process. A development step for performing development processing including at least luminance and color correction on the image signal based on a face; and a second face detection step for performing processing for detecting a face on the developed image signal; When a face is detected in the second face detection step, a red-eye detection step for performing processing for detecting red eyes from the detected face, and a red-eye correction step for correcting red eyes detected in the red-eye detection step And have.

  The red-eye correction method further includes a reading step of reading the image signal from the image sensor, and the first face detection step is performed in parallel with the reading step.

  Even when the subject or the photographer is moving, the red-eye detection can be performed with high accuracy and the red-eye correction rate can be improved.

  In addition, the time required for red-eye detection can be kept short.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings. However, the dimensions, shapes, relative arrangements, and the like of the components exemplified in the present embodiment should be changed as appropriate according to the configuration of the apparatus to which the present invention is applied and various conditions. It is not limited to.

  FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention.

  In FIG. 1, reference numeral 60 denotes a system control unit that controls the operation of the entire imaging apparatus.

  Reference numeral 10 denotes a photographing lens that collects external light. Although the lens is shown as one in the figure, it is usually composed of a plurality of lenses. The photographing lens 10 includes, for example, a focus lens for adjusting the focus and a zoom lens for adjusting the angle of view by moving the lens driving circuit 42 back and forth along the optical axis. Further, based on the camera shake amount measured by the camera shake detection circuit 44, the camera shake correction circuit 40 moves the lens to change the optical axis in the direction of canceling camera shake, thereby performing optical camera shake correction. Including an image stabilization lens to perform. Moreover, it is possible to improve the portability by adopting a system in which the lens barrel portion including the photographing lens 10 is extended and retracted, and the main body volume is reduced when the camera is not used.

  Note that the shake amount detection circuit 44 includes a gyro sensor, and in FIG. 1, camera shake correction is realized by driving a lens. However, camera shake correction is similarly performed by driving an image sensor 16 described later. Is also possible.

  Reference numeral 12 denotes a mechanical shutter (hereinafter simply referred to as “shutter”). The system control unit 60 can control the shutter 12 by transmitting mechanical shutter control information to the shutter drive circuit 28. The exposure time during still image shooting is determined by the opening / closing time of the shutter 12, and the system control unit 60 determines this opening / closing time and issues an instruction to the shutter drive circuit 28.

  Reference numeral 14 denotes an aperture for adjusting the amount of light passing through the photographing lens 10. Examples of the diaphragm 14 include an iris diaphragm composed of a plurality of wings and a round diaphragm in which holes are punched out in advance with various diameters. The system control unit 60 can control the diaphragm 14 by transmitting the diaphragm control information to the diaphragm driving circuit 26. Transmission of control information from the system control unit 60 to the aperture driving circuit 26 is performed by serial communication, a pulse signal, or the like, and means suitable for the specifications of the aperture driving circuit 26 is taken. The system control unit 60 uses the diaphragm 14 and the diaphragm drive circuit 26 to control the diaphragm so as to reduce the amount of light when the subject brightness is high, and when the subject brightness is low, the diaphragm is opened to capture more light. It is possible to control as follows.

  Reference numeral 16 denotes an image sensor such as a CCD or CMOS sensor, which converts an optical image of a subject incident through the taking lens 10, the shutter 12, and the aperture 14 into an electrical signal (image signal) and outputs it. The system control unit 60 can control the image sensor 16 by transmitting an image sensor control signal to a TG (Timing Generator) 24. Transmission of control information from the system control unit 60 to the TG 24 includes serial communication, parallel bus communication, and the like, and means suitable for the specifications of the TG 24 is taken.

  The TG 24 drives the image sensor 16 based on the control information received from the system control unit 60. The image pickup device 16 periodically performs charge accumulation in each pixel and readout operation of the accumulated charge, and this operation is performed with reference to a drive signal from the TG 24. It is possible to read out charges from only a specific line or a specific area among the charges accumulated in the image sensor 16. This can be realized by changing the reading method by the read control pulse output from the TG 24. The system control unit 60 determines an optimum reading method according to the situation and instructs the TG 24. For example, since high resolution is required during still image shooting, all data of the image sensor 16 is read out, and when a high frame rate such as 30 fps or 60 fps is required during electronic viewfinder or movie shooting, only specific lines are thinned out. It is used properly such as reading. Also, the exposure time of the image sensor 16 can be controlled via the TG 24. This is made possible by outputting a drive signal from the TG 24 to the image sensor 16 so as to clear the charge charged by the image sensor 16 at an arbitrary timing.

  Reference numeral 18 denotes a CDS (Correlated Double Sampler) circuit that removes noise components of image data from the electrical signal (image signal) output from the image sensor 16 by a correlated double sampling method. Reference numeral 20 denotes a PGA (Programmable Gain Amplifier) circuit that attenuates / amplifies the level of the image data, and the amount of amplification is controlled by a system control unit 60 described later. Control information transmission from the system control unit 60 to the PGA circuit 20 includes serial communication, parallel bus communication, and the like, and takes measures suitable for the specifications of the PGA circuit 20. Usually, in order to obtain image data of an appropriate level, the amount of light incident on the image sensor 16 is appropriately set by the aperture 14 and the exposure time is appropriately set by the shutter 12 so that the exposure of the image sensor 16 is appropriate. In addition, by attenuating / amplifying the image data signal by the PGA circuit 20, it is possible to play a role of changing the exposure of the image data in a pseudo manner. This is a function of providing the user with a concept of sensitivity as one of the exposure conditions at the time of photographing along with the aperture and shutter speed.

  The image data processed by the PGA circuit 20 is converted from an analog signal to a digital signal by an A / D (Analog / Digital) circuit 22. Depending on the device, the bit width of the digital signal may be 10 bits, 12 bits, 14 bits, or the like. The subsequent image processing circuit 50 can cope with a plurality of types of bit widths. In the figure, the CDS circuit 18, the PGA circuit 20, and the A / D circuit 22 are shown as different configurations, but it is also possible to adopt a circuit in which the functions of these circuits are mounted in one IC package.

  The digitized image data output from the A / D circuit 22 is input to the image processing circuit 50. The image processing circuit 50 includes a plurality of blocks and realizes various functions.

  For example, the image sensor 16 generally extracts light of a specific color component for each pixel through a color filter, and photoelectrically converts the extracted light. The image signal output from the A / D circuit 22 has a data format corresponding to the pixel of the image sensor 16 and the color filter arrangement, and automatic exposure control (AE: AutoExposureControl) that performs exposure control by evaluating only the luminance component. Not suitable for use with. Therefore, the image processing circuit 50 has a function of removing color information from the image signal and extracting only luminance information.

  Furthermore, the image processing circuit 50 has a function of extracting only the frequency component of the image signal read from the image sensor 16, and can be used during automatic focusing control (AF: AutoFocus). A function is provided in which the frequency component of which area of the image signal read from the image sensor 16 is extracted or the area can be divided and set. At the time of this AF processing, it is possible to drive the image sensor 16 to be suitable for focus adjustment. In the case of television AF using the image sensor 16, the focus lens is driven to a new focus position each time an image signal is read from the image sensor 16, so that the higher the frame rate of the image sensor 16, the more quickly the focus lens is focused. Can be moved to a position. For this reason, the image sensor 16 can be driven so as to have a fast frame rate only during AF. Conversely, by reducing the frame rate and reading more pixel data from the image sensor, the amount of data that can be analyzed by the image processing circuit 50 increases, and more accurate focus adjustment control is possible. is there. Such use of the image sensor 16 can be determined adaptively according to the shooting mode of the camera and the brightness of the subject.

  Further, the image processing circuit 50 has a function of adjusting the level of the image data digitized by the A / D circuit 22 and operating the color effect of the image, and also plays a role of adjusting the image quality of the photographed image. . Regarding the level of image data, there are a function for increasing and decreasing the level with a uniform amplification factor for the entire image and a gamma function for converting the signal level according to the magnitude of the original signal level. In addition, various level adjustments such as a function of increasing / decreasing the level with an amplification factor corresponding to a frequency component for each region in the screen can be performed.

  The image data digitized by the A / D circuit 22 can be input to the image processing circuit 50 and simultaneously stored in the temporary storage memory 30. The image data once stored in the temporary storage memory 30 can be read again, the image data can be referred to from the system control unit 60, and the read image data can be input to the image processing circuit 50. Furthermore, the image data processed by the image processing circuit 50 can be written back to the temporary storage memory 30 or arbitrary data can be written from the system control unit 60.

  Image data appropriately processed by the image processing circuit 50 can be input to the image recognition circuit 38. The image recognition circuit 38 can recognize the character information of a person's face, its facial expression, and characters in addition to recognizing the brightness, focus, and color of the input image. In addition, a plurality of images can be input to the image recognition circuit 38. For example, it is possible to determine whether the images are the same by inputting two images and comparing the characteristics of the two images.

  An operation unit 70 includes a power ON / OFF switch 102, a shutter switch 104, a mode change switch 110, a parameter selection switch group 151, and the like.

  Reference numeral 108 denotes an image display device constituted by an LCD or the like. When outputting an image to the image display device 108, the image data subjected to the image processing by the image processing circuit 50 is developed on the VRAM 34 and converted into analog data by the digital / analog (D / A) circuit 36. Then display. In addition, the electronic viewfinder function can be realized by sequentially updating and displaying the continuous images read from the image sensor 16 on the image display device 108. When the image data is expanded on the VRAM 34, the VRAM 34 is adapted to correspond to various display forms such that one image data is maximized on the image display device 108, or a plurality of images are displayed on a multi-screen. Can be deployed on top.

  Further, the image display device 108 can display not only an image but also arbitrary information alone or together with the image. It is also possible to display camera status display, character information such as shutter speed, aperture value, sensitivity information selected by the user or determined by the camera, and a graph such as luminance distribution measured by the image processing circuit 50. The display position and display color of information can be arbitrarily selected. By displaying these various types of information, a user interface can be provided. The image display device 108 can also display image data stored in the image storage medium 82. If the image data is compressed, the image data is decompressed by the compression / decompression block 32 and the data is expanded in the VRAM 34. The expanded data is converted into analog data by the D / A circuit 36 and output.

  The image storage medium 82 is non-volatile, and can mainly store captured image data via the image storage medium interface 80. Regarding the storage of image data, for example, a folder hierarchy can be provided, or folders and file names can be named in ascending order in the order of shooting. Shooting information such as aperture value, shutter speed, ISO sensitivity, shooting time, and the like can be added to each image data, and these can be stored together with the image data. Furthermore, the stored image data can be read out, and the image can be copied, moved, and deleted.

  Reference numeral 86 denotes an external device such as a printer, a PC, or a display device, which is connected via an external device interface 84.

  FIG. 2 shows an external view of the imaging apparatus shown in FIG. As shown in FIG. 2A, a photographing lens 10 is disposed on the front surface of the imaging apparatus, and a subject image can be captured. A flash 90 is disposed on the same surface of the photographic lens 10. A sufficient amount of light can be obtained by causing the flash 90 to emit light when the main subject is dark, and a suitable image can be obtained while maintaining a high shutter speed even in the dark. In FIG. 2, the photographing lens 10 and the flash 90 are arranged on the same plane, but the present invention is not limited to this. For example, in order to avoid the flash light directly hitting the main subject, the flash faces the upper part of the camera. You may arrange as follows.

  As described above, a red-eye phenomenon is known in which a person's eyes appear red when a person is flash-photographed in a dark environment. This is a phenomenon in which the pupil of the eye tends to be large in a dark environment, and when the flash is emitted in this state, the flash light is reflected on the retina and appears red as a photograph.

  As shown in FIG. 2B, an image display device 108 is disposed on the back surface of the imaging device. As described above, not only images but also character information and graphs can be displayed on the image display device 108, which is an important member for interfacing with the user. In recent years, an electronic viewfinder (EVF: Electric ViewFinder) has become mainstream in digital cameras, and a subject is captured by referring to a continuous image output to the image display device 108 and used as a viewfinder. At this time, the photometry area information and distance measurement area information in AE and AF can be displayed superimposed on the live image. Furthermore, as the recognition status of the subject, as a result of recognizing the person's face, a frame is superimposed on the person's face, or the background scene situation such as blue sky, sunset, backlight, etc. is recognized and displayed as an icon, etc. Is possible.

  In addition, a conventional optical finder 106 may be provided. The electronic viewfinder has advantages such that it is easy to realize a high field of view, the subject is easy to see depending on the size of the image display device 108, and there is no field angle difference (parallax) between the captured image and the viewfinder image. On the other hand, power is required to operate the image sensor 16 and the image display device 108, and there is a concern that the battery will be consumed quickly. For this reason, when a large number of shots are desired while avoiding battery consumption, the electronic viewfinder function can be turned off and the optical viewfinder 106 can be used.

  The mode switch 110 can switch camera operation modes such as a still image shooting mode, a moving image shooting mode, an image recognition mode, and a playback mode. Here, although expressed as a member capable of switching several modes, it is possible to provide many still image modes such as a landscape shooting mode and a person shooting mode optimized for a specific scene to be shot.

  The parameter selection switch groups 151a to 151e allow the user to select shooting conditions such as a distance measurement area and a photometry mode, shooting page playback, and overall camera operation settings. . Furthermore, On / Off of the above-described electronic viewfinder can be selected. Further, the image display device 108 can display an image and can be configured as an input device as a touch panel.

  A shutter switch 104 is disposed above the image pickup apparatus. The shutter switch 104 is one of the operation members. The shutter switch 104 instructs the preparation and start of photographing by a two-step pressing operation when the button is pressed lightly (SW1) and when the button is pressed deeply (SW2). In the case of a camera that performs automatic exposure control and automatic focus control with a camera, automatic exposure control and focus control are performed as a preparation for shooting by pressing the shutter switch 104 shallowly, and instructions for still image shooting and image recognition are performed by pressing the shutter switch 104 deeply. Operation is feasible.

  The automatic exposure control operates so as to obtain a suitable exposure in the shooting mode selected by the mode switch 110. The shooting mode includes, for example, a mode specialized for a specific subject such as a portrait mode, a landscape mode, and a night view mode, and a general mode such as an auto mode. There is also a mode in which the user designates the shutter speed and aperture value at the time of shooting, such as the shutter speed priority mode and the aperture priority mode. In these modes, the photographing sensitivity set by the PGA circuit 20 can be automatically selected and set appropriately, or the user can designate the sensitivity in advance. When the user designates the sensitivity in advance, the S / N ratio of the image signal decreases as the shooting sensitivity is increased. Therefore, it is assumed that the user who wants to give priority to the image quality selects the low sensitivity. The AF control can be switched so that a suitable pin can be aligned in each shooting mode. For example, in the landscape mode, it is assumed that the main subject is on the far side, and therefore it is possible to measure the distance only in the vicinity thereof.

  When a person is photographed with the image pickup apparatus having such a configuration, since the face of the person is detected and the camera side adjusts the focus, exposure, and color appropriately, there is a great merit for the user.

  This is shown in FIG. 3A shows the situation when AE and AF are performed when SW1 of the shutter switch 104 is turned on, and FIG. 3B shows the state when the flash emission amount is determined and the photographing is performed when SW2 is turned on. Indicates the situation. The example shown in FIG. 3 shows a case where the position of the subject 301 when SW1 is ON and the position of the subject 301 when SW2 is ON are substantially the same. In this case, there is no difference in the position of the subject between focusing and exposure determination and actual shooting, and there is no deviation between the human face detection result 302 at the time of exposure determination and the human face detection result 302 at the time of shooting. A suitable image can be obtained.

  However, when the state of the subject has changed between when SW1 is turned on and when SW2 is turned on, a suitable image may not be obtained. This is shown in FIG. When SW1 shown in FIG. 4A is ON, the subject 301 is located on the left side of the screen, and the human face is correctly detected at that position. Thereafter, it is assumed that the subject 301 has moved to the right side of the screen when SW2 shown in FIG. In this case, flash shooting is performed by determining the flash emission amount by performing preliminary light emission in a state in which the focus detection and exposure when SW1 is turned on are far away from the face detection position 302 when SW2 is turned on. become. For this reason, the shooting operation is performed by determining the flash emission amount with the intention that the face is located at the position 302 where the face of the subject 301 does not exist. Therefore, the main subject as shown in FIG. In some cases, an image out of the range may be taken. Even if the shooting operation is performed without flash emission, if the brightness of the subject 301 changes between the time when SW1 that determines the exposure is turned on and the time when the shooting operation is performed, the main subject deviates from the proper exposure. May be taken.

  Such a phenomenon occurs in a camera with a specification that performs focus adjustment and exposure control based on a face detection result when SW1 is ON, and is likely to cause harmful effects. The adverse effect can be reduced in the case of a camera that performs face detection even when SW1 is held ON, and that follows exposure and focus according to the detection result. However, even with that, it takes a certain amount of processing time to detect the face, so it cannot follow the actual movement of the subject and cannot fundamentally solve the problem.

  FIG. 5 shows a state for reproducing suitable exposure and color corresponding to the movement of the subject in the embodiment of the present invention. Here, as shown in FIG. 5 (a), the position of the subject 301 when SW1 is turned on and the face detection position 302 at that time are located on the left side of the screen, and then SW2 is turned on as shown in FIG. 5 (b). It is assumed that the image is taken with the subject 301 sometimes moved to the right side of the screen. In this case, as shown in FIG. 5C, even if the brightness of the subject 301 is not photographed with proper exposure, the subject 301 is on the right side of the screen by performing face detection from the photographed image data. Detect. Then, using this detection result, the image processing circuit 50 adjusts the brightness so that the brightness of the face detection position 310 approaches the appropriate exposure, thereby obtaining a suitable image as shown in FIG.

  The image recognition circuit 38 detects a person's face from the image data based on the position, size, reliability (information about whether the person is really a person's face), and the like. Detects red-eye phenomenon that appears red. When the red-eye phenomenon is detected, red-eye correction can be performed by performing image processing in the image processing circuit 50 so that the red eye is painted black.

  In performing the red-eye correction, how the red-eye correction is affected when face detection after shooting and brightness correction after shooting are not performed as described above with reference to FIG. This will be described with reference to FIGS.

  FIG. 6A shows a case where the person who is the subject 301 is darker than the appropriate exposure without performing face detection after shooting and brightness correction after shooting. As shown in FIG. 6B, in order to detect red eyes, first, the face of the subject 301 is detected. In the example shown in FIG. 6A, the subject 301 is photographed darker than the appropriate exposure, and the accuracy of face detection may not be sufficient. In FIG. 6B, the face detection position 310 is described as a face detected, but if this is not recognized as a face, the red eye detection itself in the next step cannot be performed. If face detection is possible, an eye area 320 is set based on the face detection position 310 of the subject 301, and the image recognition circuit 38 detects whether or not the human eye is red-eye. This red-eye detection process is also performed when the subject 301 is dark, and the accuracy of red-eye detection may not be sufficient in a situation where appropriate color reproduction is not possible along with the darkness.

  FIG. 7 shows a case where face detection after shooting and brightness correction after shooting are performed, and the person who is the main subject has proper exposure. In this case, the face detection for red-eye detection can be detected with high accuracy as shown in FIG. 7B because the brightness of the subject 301 is appropriate as shown in FIG. 7A. Further, as shown in FIG. 7C, when detecting red eyes from the eye region 320 set based on the face detection position 310, the brightness and color are appropriately reproduced, so that the red eyes are detected with high accuracy. It can be performed. Even when red-eye correction is performed by the image processing circuit 50 based on the detection result, appropriate red-eye correction can be performed as compared with the case where the person is dark as shown in FIG.

  In this way, after actually taking a picture in response to SW2 being turned on, the face of a person is detected from the taken image data, and the brightness and color of the face are suitably image-processed based on the face detection result. As a result, the red-eye correction rate is improved. However, simply adding processing increases the time required for shooting, which is not preferable as a camera that cuts out the moment with good response.

  Therefore, FIG. 8 shows a timing chart for performing face detection suitably after photographing and performing face image processing appropriately without extending the time taken for the whole photographing and image processing.

  In FIG. 8, “VD” is a vertical synchronization signal that is one of the drive signals of the image sensor 16. When performing exposure at a constant cycle and reading out image signals accumulated in each pixel by exposure, such as during EVF or moving image shooting, VD is also basically at a fixed cycle. Note that the periodicity may be lost when the driving method of the image sensor 16 is switched or the VD period is appropriately controlled just before the exposure data is read out in the still image shooting sequence as shown in FIG. This is schematically shown at a constant period. “Shutter” indicates a shutter closing timing for shielding light so that the image sensor 16 is no longer exposed after the image sensor 16 is exposed for a predetermined time. The “exposure time” in “processing” is a period during which the image sensor 16 is exposed.

  After the shutter 12 is closed, an operation for reading an image signal is performed. FIG. 8 shows a state in which all data is divided into 6 fields and read by an image sensor having a specification of reading every 5 rows. R1 to R6 correspond to reading of each field, and reading is performed in synchronization with VD. The read data is stored in the temporary storage memory 30 and read from the temporary storage memory 30 for image processing by the image processing circuit 50. This process is expressed as “development” in the figure.

  In the case of shooting without flash emission, image data after development is compressed and recorded, but in the case of shooting with flash emission, processing for red-eye correction is performed. First, the developed image data is read and face detection processing is performed. At this time, if the brightness of the developed image is appropriate, highly accurate face detection can be performed, but if the face of a person is dark, face detection may not be possible with sufficient accuracy. Accordingly, in the present invention, face detection is performed before development, and development processing is performed so that the brightness of a person's face is appropriately processed based on the face detection result. As a result, face detection for red-eye correction can be performed with high accuracy.

  The face detection before development may not be appropriate for the brightness of the person's face, so the face detection conditions before the development are set loosely with respect to the face detection after development for red-eye correction. Do. Note that at the stage where image signals are read from some fields, the field images are thinned out in the vertical direction. For this reason, image data is generated by converting the resolution of the field image so that the thinning ratios of the aspect ratio are uniform, and face detection before development is performed on the image data. Thereby, even if the brightness of a person's face is not appropriate, what appears to be a face is detected as a face, and image processing is performed by development. In FIG. 8, face detection before development is performed on the first and second field images at the time when reading of the second field is completed, but the present invention is not limited to this. . For example, face detection before development may be performed when image data sufficient for face detection can be read. Since face detection before development is performed at the stage of reading out an image signal from a part of the field, it is performed on image data having a smaller resolution than face detection after development.

  In this way, face detection for red-eye correction is performed in a state where the brightness of the person's face is appropriate, and then it is inspected from the face detection result whether a red-eye phenomenon has occurred for the position of the person's eyes. To do. Since the image data after development is appropriately processed for brightness and color reproduction is also performed appropriately, red is easily processed as red. For this reason, in the red-eye detection process for searching for red, the detection accuracy is improved, and a good red-eye detection result can be obtained. With this red-eye detection result, a red-eye correction process is performed so that the color information of the red-eye position of the image data stored in the temporary storage memory 30 is replaced with black.

  FIG. 9 shows the timing when the image sensor 16 is driven so that the exposure data is read out once without dividing the field. The basic sequence is the same as that shown in FIG. Rather than storing the read image signal in the temporary storage memory 30 and performing face detection, the image signal is input to the image recognition circuit 38 as it is while reading the image signal, and face detection is performed. The face detection is started simultaneously with the reading of the image signal. As described above, since the read image signal is thinned out in the vertical direction, a part of the image signal is further selected from the read image signal and the thinning ratio is adjusted. Data is input to the image recognition circuit 38. When the sequence for reading the image data stored in the temporary storage memory 30 again after the completion of the reading of the image signal is made, the time required for data reading and face detection is added as it is as the time required for photographing. For this reason, it is possible to suppress a delay in the required shooting time by using a sequence in which the read image signal is directly subjected to face detection.

  FIG. 10 is a flowchart showing the procedures of photographing and image processing in the present embodiment.

  When a shooting mode is selected after the camera is activated, shooting processing is started (step S201), and reading and outputting of an image used for live view is started in order to cause the image display device 108 to function as an electronic viewfinder (EVF). To do. At the start, exposure control (AE) is performed in step S203, white balance control (AWB) is performed in step S205, and focus control (AF) is performed in step S207 in order to make the image quality of the live view image appropriate. Note that the order of these processes is not limited to the above-described process order, and may be reversed, or some or all of them may be performed in parallel.

  In AE in step S203, the aperture diameter of the diaphragm 14, the charge sweeping timing (electronic shutter) of the image sensor 16 by the control of the TG 24, and the gain of the PGA circuit 20 are controlled following the change in the subject brightness. By these controls, the image signal obtained by the image sensor 16 continues to operate periodically so that the signal level is within a predetermined range. In AWB in step S205, the operation continues periodically so that the color of the subject becomes appropriate according to the light source. The AF in step S207 performs focusing control by periodically driving the photographing lens 10 so as to keep focusing on the main subject.

  In parallel with the operation of AE, AWB, and AF, a face detection process for detecting the presence of a human face in the image data is operated (step S209). If a face is detected, AE, AWB, and AF are detected. However, using the face detection result, the operation is performed so that the image quality of the person's face is appropriate.

  When the user of the imaging apparatus attempts to shoot and detects that SW1 is turned on by pressing the shutter switch 104 (step S211), an AE process for determining an exposure condition for still image shooting is performed. This is performed (step S213). Here, the aperture value, shutter speed, and sensitivity (gain) used at the time of still image shooting are determined based on the state of the subject brightness before SW1 is turned on and the face detection result. At this time, if it is determined that there is a high possibility that the image will be blurred because the subject is dark and the shutter speed is slow, it is also determined whether to emit flash light in still image shooting. Depending on the specifications of the imaging device, there are those in which the user can specify in advance whether the flash light is emitted or not, and in this case, the specified setting is followed. Next, focus adjustment control on the subject

This is performed (step S215). In step S207, the subject is substantially focused by AF before SW1 is turned on. Here, after SW1 is turned on, high-precision focus adjustment control for still image shooting is performed again.
If the exposure condition for still image shooting is determined and SW1 is turned off after the focus state is reached (YES in step S217), shooting is stopped, AE for EVF (step S203), AWB (step S205), AF (step S207) is resumed. On the other hand, if SW2 is turned on (YES in step S219), still image shooting is started.

  First, when performing flash photography according to the presence / absence of flash emission determined when the exposure for the still image is determined in step S213, a light control process for determining the light emission amount at the time of photography is performed. (Step S223). Here, the flash is pre-flashed with a predetermined light emission amount, and the intensity of the reflected light is measured, so that the subject condition including the distance and reflectance of the subject is grasped, and the light emission amount during still image shooting is determined. Is done. Since the conditions necessary for photographing have been prepared so far, next actual photographing and image processing of the photographed image are performed (step S300). When shooting and image processing are completed, the process returns to step S217 to repeat the above processing.

  FIG. 11 is a flowchart showing details of the photographing and image processing operations performed in step S300.

  Before the exposure to the image sensor 16 is started, the aperture diameter of the diaphragm 14 is controlled to the aperture diameter for still image shooting obtained in step S213 (step S303). Next, exposure to the image sensor 16 is started in step S305. In the case of an imaging apparatus that performs both opening and closing of exposure by opening and closing the mechanical shutter, such as a focal plane shutter, the mechanical shutter is opened here. On the other hand, when the mechanical shutter is opened before the start of exposure and the exposure is started by an electronic shutter that electronically sweeps away the charge of the image sensor 16, exposure is performed by performing charge sweep control of the image sensor 16 here. To start.

  When shooting is performed with the flash fired (YES in step S307), light is emitted during exposure (step S309). The light emission amount is determined in advance in step S223 in FIG. 10. However, depending on the specifications of the imaging apparatus, there is a light emission amount and light emission timing that can be arbitrarily specified by the user. Perform light emission control. When the exposure time determined in step S213 in FIG. 10 elapses and the shutter 12 is closed, the exposure ends (step S311). Then, before reading the image signal from the image sensor 16, the gain used in the PGA circuit 20 for adjusting the signal level of the image signal is set. This gain is provided to the user as a sensitivity as a digital camera.

  In FIG. 11, as in FIG. 8, the image signal is divided into six fields and read out from the entire image sensor 16 and is sequentially read out for each field (steps S <b> 315, S <b> 317, S <b> 321, S <b> 323, S <b> 325, S <b> 327). . Here, when the reading of the first field and the second field is completed (steps S315 and S317) and the data is stored in the temporary storage memory 30, the face detection for development (first face detection step) is started. (Step S319). As described above, which field of image data is used for face detection is not limited to this. In addition, since the brightness of the face of the person in the image data at this time may not be appropriate, the detection conditions for the face detection are set loosely.

  When the reading of the image signal from the entire image sensor 16 and the development face detection are completed, the image processing circuit 50 determines what kind of development processing is performed. If a face is detected, the image color (AWB) adjustment amount (step S329) and the image brightness adjustment amount (step S331) are determined so that the detected face is appropriate. Image development processing is performed using the adjustment amount (step S333). By this development processing, the brightness and color of a person's face can be appropriately reproduced. After that, as described above, using the developed image data, highly accurate face detection (second face detection step) for red-eye correction (step S335) and red-eye detection (step S337) are performed. If red-eye is detected, it is determined in step S339 whether flash photography has been performed. If flash photography has not been performed, the processing is terminated as is, and if it has been performed, red-eye correction processing is executed (step S341).

  FIG. 12 is a flowchart showing details of the photographing and image processing operations performed in step S300. As described with reference to FIG. 9, the processing when the image signal is read from the entire image sensor 16 without field division is shown. ing. The difference from FIG. 11 is the following two points. That is, instead of steps S315, S317, S321, S323, S325, and S317, reading from the previous pixel of the image sensor 16 is performed in step S417, and simultaneously with the start of the reading, development face detection is performed in step S415. It is. Since the other processes are the same as those in FIG. 11, the description thereof is omitted here.

  As described above, according to the present embodiment, a human face is detected from captured image data, and the brightness and color of the human face are appropriately image-processed. Thereby, even if the subject moves from the shooting preparation time before shooting to the actual shooting, a good shot image can be obtained, and the face detection and red-eye detection accuracy for red-eye correction are improved.

  Furthermore, in order to obtain this effect, it can be realized without extending the processing time for photographing, and it is possible to provide high performance as a camera that cuts out the moment with good response.

<Other embodiments>
Note that the present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a camera head, etc.), or a device (for example, a digital video camera, a digital still camera, etc.) including a single device. You may apply to.

  The object of the present invention can also be achieved as follows. First, a storage medium (or recording medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following can be achieved. That is, when the operating system (OS) running on the computer performs part or all of the actual processing based on the instruction of the read program code, the functions of the above-described embodiments are realized by the processing. It is. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered. Also, a computer network such as a LAN (Local Area Network) or a WAN (Wide Area Network) can be used to supply the program code.

It is a block diagram which shows the structure of the imaging device in embodiment of this invention. It is a figure which shows the external appearance of the imaging device shown in FIG. It is a figure which shows an example of a to-be-photographed object position, face recognition result, and a photography result before imaging | photography. It is a figure which shows an example of the face recognition result and imaging | photography result when a to-be-photographed object moves before imaging | photography and at the time of imaging | photography. It is a figure which shows an example of the face recognition result when a to-be-photographed object moved before and during imaging, an imaging result, and a correction processing result after imaging. It is a figure for demonstrating the red eye correction process with respect to the person who was photographed darkly. It is a figure for demonstrating the red-eye correction process with respect to the person who image | photographed with appropriate brightness. It is a timing chart which shows the timing of imaging | photography and correction | amendment processing in embodiment of this invention. It is a timing chart which shows another timing of photography and correction processing in an embodiment of the invention. It is a flowchart which shows the imaging | photography procedure in embodiment of this invention. It is a flowchart which shows the imaging | photography process in embodiment of this invention. It is a flowchart which shows the imaging | photography process in embodiment of this invention.

Explanation of symbols

16 Image sensor 38 Image recognition circuit 50 Image processing circuit 60 System control unit

Claims (10)

Face detection means for performing processing for detecting a human face from an image signal;
Red-eye detection means for performing processing for detecting red eyes from the face detected by the face detection means;
Developing means for performing development processing including at least luminance and color correction on the image signal;
Red-eye correction means for correcting red eyes detected by the red-eye detection means;
Control means,
The control means includes
Controlling the face detection means to perform a process of detecting a face from the image signal;
When a face is detected by the face detecting means, the developing means is controlled to perform the developing process based on the detected face;
Controlling the face detection means to perform a process of detecting a face from the developed image signal;
When a face is detected from the developed image signal, the red-eye detection unit is controlled to perform a process of detecting red eyes from the face detected from the developed image signal. . An image sensor that converts an optical image of the subject into an image signal;
The control means controls the face detection means so that a process of detecting a face from the image signal before the development process is performed in parallel with a process of reading the image signal from the image sensor. The imaging apparatus according to claim 1.   3. The process for detecting a face from the image signal before the development process uses an image having a lower resolution than the process for detecting a face from the image signal after the development process. Imaging device.   If the image signal is an image signal obtained by flash photography, the control means further performs red-eye correction on the red eye detected by the red-eye detection means, and is obtained by flash photography. The imaging apparatus according to any one of claims 1 to 3, wherein the red-eye correction unit is controlled so that the red-eye correction is not performed if the image signal is not an image signal.   5. The control unit according to claim 1, wherein a detection condition used for face detection performed after the development processing is made stricter than a detection condition used for face detection performed before the development processing. The imaging apparatus of Claim 1.   6. The control unit according to claim 1, wherein the control unit controls the development unit to perform a development process that is not based on a face when a face is not detected by the face detection unit. The imaging device described in 1. A first face detection step for performing processing for detecting a human face from an image signal;
A development step of performing development processing including at least luminance and color correction on the image signal based on the detected face when a face is detected in the first face detection step;
A second face detection step of performing a process of detecting a face on the developed image signal;
A red-eye detection step of performing a process of detecting red eyes from the detected face when a face is detected in the second face detection step;
A red-eye correction step of correcting the red-eye detected in the red-eye detection step. A reading step of reading the image signal from the image sensor;
The red-eye correction method according to claim 7, wherein the first face detection step is performed in parallel with the reading step.   The program for making a computer perform each process of the red-eye correction method of Claim 7 or 8.   A computer-readable storage medium storing the program according to claim 9.

Images