JP2021057825A - Signal processing apparatus and method, and imaging apparatus - Google Patents

Abstract

To detect a specific subject even in a state of a large defocus quantity of an imaging optical system while performing focus detection in an imaging plane phase difference manner.SOLUTION: A signal processing apparatus processes a pair of focus detection signals with a parallax and an image signal which can be acquired from an imaging element comprising a plurality of photoelectric conversion parts for a plurality of microlenses respectively, and also comprises subject detection means which uses at least one of the pair of focus detection signals to detect a predetermined subject, and focus detection means which uses the pair of focus detection signals to detect a focus state. When the subject detection means detects the subject, the focus detection means uses a pair of focus detection signals corresponding to a region of the subject among pairs of focus detection signals to detect the focus state.SELECTED DRAWING: Figure 1

Description

The present invention relates to a signal processing device and method and an imaging device, and more particularly to a signal processing device and method having a function of detecting a specific subject such as a face and an imaging device.

In recent years, some imaging devices have a so-called face detection function of detecting a face existing in a generated image by an image recognition technique. The image pickup apparatus automatically focuses on the area including the face detected by the face detection function in the image pickup range, and superimposes and displays a frame showing the area including the face on the image on the monitor.

On the other hand, AF (conventional AF) in which a pupil division signal is acquired by an image sensor provided with a plurality of photoelectric conversion units for one microlens and focus detection is performed based on the phase difference of the obtained plurality of pupil division signals. Autofocus) technology is known.

In Patent Document 1, not only focus detection is performed based on the phase difference of the pupil-divided image, but also all signals from a plurality of photoelectric conversion units sharing the same microlens are added to obtain a plurality of photoelectric conversion units. Is disclosed as one photoelectric conversion unit. This makes it possible to handle signals in the same manner as when one photoelectric conversion unit is provided for one microlens, and it is possible to create an image for recording by a conventional signal processing technique.

Patent Document 2 discloses the following techniques as a signal readout method for an image sensor. First, the first focus detection signal (hereinafter, referred to as “A image”) from one of the two pupil-divided pixels is read out via the input node of the amplification unit. Then, the A image and the B image are read out from the other pixel through the input node of the amplification unit without resetting the input node. Is added to the signal (hereinafter referred to as "A + B image") is read out. The B image is obtained by subtracting the A image from the A + B image.

Japanese Unexamined Patent Publication No. 2008-134389 Japanese Unexamined Patent Publication No. 2013-106194

In an image pickup device equipped with an image sensor that outputs an A image and an A + B image obtained by adding an A image and a B image as an image pickup signal as described above, face detection can be performed by a conventional method using the A + B image. it can. However, since the depth of focus of the A + B image is shallow, it is difficult to detect the face when the defocus amount of the imaging optical system is large. Therefore, in order to perform face detection accurately, it is necessary to narrow down the aperture in the imaging optical system to some extent to deepen the depth of focus.

Further, conventionally, in the image processing for display / recording and the image processing for detecting a specific subject, different image processings have been performed. This is mainly because image processing such as white balance, shading correction, and gamma correction differs depending on the scene mode of the camera between display / recording and detection of a specific subject. For the detection of a specific subject, the optimum image processing for detection is required, and it is not necessary to change it according to the scene mode of the camera. Therefore, separate image processing circuits have been implemented for display / recording image processing and subject detection image processing.

Further, the focus detection signal (A image / B image) also has a problem that the circuit scale is increased and the power consumption is increased by performing another image processing suitable for the focus detection.

The present invention has been made in view of the above problems, and an object of the present invention is to enable detection of a specific subject even when the defocus amount of the imaging optical system is large while performing focus detection by the imaging surface phase difference method. And.

In order to achieve the above object, the signal processing device of the present invention has a pair of focal detection signals and image signals having a difference in range that can be acquired from an image pickup device having a plurality of photoelectric conversion units for each of the plurality of microlenses. A signal processing device that processes the above, and detects a focal state by using a subject detecting means for detecting a predetermined subject by using at least one of the pair of focal detection signals and the pair of focal detection signals. When the subject is detected by the subject detection means, the focus detection means has a pair of focus detection signals corresponding to the region of the subject in the pair of focus detection signals. It is characterized by detecting a focal state using a signal.

According to the present invention, it is possible to detect a specific subject even when the defocus amount of the imaging optical system is large, while performing focus detection by the imaging surface phase difference method.

The block diagram which shows the schematic structure of the image pickup apparatus which concerns on 1st Embodiment of this invention. The figure explaining the pixel arrangement of the image pickup device which concerns on embodiment. The figure which shows the concept that the light flux emitted from the exit pupil of an image pickup lens is incident on a unit pixel. The figure for demonstrating the phase difference AF. The block diagram which shows the structure of the image pickup signal image processing part in embodiment. The block diagram which shows the structure of the focus detection signal image processing part in embodiment. The flowchart which shows the imaging operation of the imaging apparatus in embodiment. The flowchart which shows the face detection process in 1st Embodiment. The block diagram which shows the schematic structure of the image pickup apparatus which concerns on 2nd Embodiment. The figure which shows the position (coordinates) of a face in 2nd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

<First Embodiment>
FIG. 1 is a block diagram showing a schematic configuration of an image pickup apparatus according to a first embodiment of the present invention. The image pickup device of this embodiment includes an optical system 101, an image pickup device 102, a signal processing unit 103, and an external storage device 104.

The optical system 101 is a portion that guides incident light to the image pickup element 102, has at least one or more lenses such as a zoom lens, an aperture, and a focus lens, and is driven by the optical system drive unit 106.

The image sensor 102 converts the subject image incident through the optical system 101 into an electric signal by photoelectric conversion and outputs the image. Here, the pixel configuration of the image pickup device 102 will be described.

FIG. 2 is a diagram illustrating a pixel array constituting the image pickup device 102. In the present embodiment, a plurality of photoelectric conversion units 201A and 201B are arranged so as to correspond to each microlens 201 forming the microlens array. Each region indicated by (0,0), (1,0), (0,1), etc. in the figure indicates a unit pixel 202 in the image sensor 102, and each unit pixel 202 includes one microlens 201 and one microlens 201. It is composed of a plurality of photoelectric conversion units 201A and 201B. In the present embodiment, the unit pixel 202 shows a case where two photoelectric conversion units 201A and 201B are arranged in the X-axis direction, but even if they are arranged in the Y-axis direction, three or more photoelectric conversion units can be arranged. May be configured to correspond to each microlens 201.

FIG. 3 is a diagram observing how the light emitted from the optical system 101 passes through one microlens 201 and is received by the image sensor 102 from the direction perpendicular to the optical axis Z (Y-axis direction). is there. The light that has passed through the exit pupils 301 and 302 of the optical system 101 is incident on the unit pixel 202 about the optical axis Z.

As shown in FIG. 3A, the luminous flux passing through the pupil region 301 is received by the photoelectric conversion unit 201A through the microlens 201, and the luminous flux passing through the pupil region 302 is received by the photoelectric conversion unit 201B through the microlens 201. Therefore, the photoelectric conversion units 201A and 201B receive light that has passed through different regions of the exit pupil of the optical system 101, respectively.

Of the plurality of unit pixels 202 arranged in the X-axis direction configured in this way, the subject image composed of the focus detection signal group of the pupil-divided photoelectric conversion unit 201A is the A image, and the pupil-divided photoelectric conversion unit The subject image composed of the focus detection signal group of 201B is defined as the B image. Then, the amount of image shift (phase difference) is detected by performing a correlation calculation on the A image and the B image. Further, by multiplying the amount of image shift by a conversion coefficient determined according to the focal position and the characteristics of the optical system, the focal position corresponding to an arbitrary subject position on the screen can be calculated. By controlling a focus lens (not shown) included in the optical system 101 based on the focus position information calculated here, imaging surface phase-difference AF for detecting the focal state becomes possible.

Further, an image signal (A + B image) can be obtained by adding and outputting the signal of the photoelectric conversion unit 201A and the signal of the photoelectric conversion unit 201B for each unit pixel 202.

The image signal (A + B image) and the focus detection signal (A image) thus obtained are output to the sensor correction unit 105.

FIG. 3B is a diagram observing from the vertical direction (Y-axis direction) how the light emitted from the optical system 101 passes through one microlens and is received by one photoelectric conversion unit 201A, 201B. is there. Compared with FIG. 3A, the aperture area is doubled because the pupil is not divided. This is the same as the area where the photoelectric conversion unit 201A and the photoelectric conversion unit 201B in FIG. 3A are added together, and the aperture is smaller than that in the photoelectric conversion unit 201A or the photoelectric conversion unit 201B in FIG. 3A. Corresponds to the open state. That is, the focus detection signals (A image, B image) obtained from the photoelectric conversion unit 201A or the photoelectric conversion unit 201B have a deeper depth of focus than the image signal.

Returning to the description of FIG. 1, the signal output from the image sensor 102 having the above-described configuration is input to the signal processing unit 103. The signal processing unit 103 includes a sensor correction unit 105, an optical system drive unit 106, an addition signal separation unit 107, an image sensor drive unit 108, an image pickup signal image processing unit 109, a focus detection signal image processing unit 110, an image display unit 111, and a subject. It has a detection unit 112 and an image shift amount calculation unit 113. Further, although the system control unit 114 is described as being included in the signal processing unit 103, it may be outside the signal processing unit 103.

The sensor correction unit 105 performs correction processing caused by known sensors such as defect pixel correction, fixed pattern noise removal, and smear correction on the A + B image and the A image output from the image sensor 102.

The addition signal separation unit 107 takes the A + B image and the A image output from the sensor correction unit 105 as inputs, and subtracts the A image from the A + B image to obtain the B image. The A + B image is output to the image pickup signal image processing unit 109 as an image signal, and the A image and the B image are output to the focus detection signal image processing unit 110 as a pair of focus detection signals having parallax.

The image pickup signal image processing unit 109 performs a plurality of processes such as shading correction, white balance correction, gamma correction, development processing, and coding processing on the A + B image output from the addition signal separation unit 107, and performs an external storage device. Output to 104 and the image display unit 111.

The external storage device 104 records image data such as JPEG or multi-picture format encoded by a coding block on the output image from the image pickup signal image processing unit 109 on a media card or the like.

The image display unit 111 displays the display image processed by the image pickup signal image processing unit 109 as a live view image. When a face is detected by the subject detection unit 112 as described later, a face detection frame (subject detection frame) is added to the coordinates specified by the system control unit 114 to generate and display a display image.

On the other hand, the focus detection signal image processing unit 110 performs a plurality of processes such as shading correction, white balance correction, gamma correction, and luminance signal conversion processing on the A image and the B image output from the addition signal separation unit 107. , Is output to the image shift amount calculation unit 113. Further, an A image or a B image (A image is described in FIG. 1) is output to the subject detection unit 112.

The subject detection unit 112 is a subject detection unit that detects a face based on the luminance signal (Y data) of the A image or the B image output from the focus detection signal image processing unit 110. When a face is detected, the position of the face (subject position) is output to the system control unit 114. The face detection process is performed using a known image recognition technique such as pattern recognition.

The image shift amount calculation unit 113 performs a correlation calculation for performing a phase difference type focus detection using the A image and the B image output from the focus detection signal image processing unit 110, and calculates the image shift amount.

The system control unit 114 controls the entire imaging device, and outputs optical system drive information such as zoom and aperture to the optical system drive unit 106 based on shooting information obtained from a shooting scene, a shooting mode, and the like. Further, the image pickup system drive information such as pupil division instruction information and exposure time is output to the image sensor drive unit 108. Further, the face position information detected by the subject detection unit 112 is input, and an area for calculating the image shift amount is specified to the image shift amount calculation unit 113 based on the information. Further, the defocus amount is calculated by multiplying the image shift amount obtained from the image shift amount calculation unit 113 by a predetermined sensitivity.

The optical system drive unit 106 controls the optical system 101 according to the focus information and the optical system drive information output from the system control unit 114.

The image sensor drive unit 108 controls the image sensor 102 according to the image pickup system drive information from the system control unit 114. The image sensor 102 may have an electronic shutter function. In that case, the image sensor 102 executes the electronic shutter function so as to have a desired exposure time according to a control signal output from the image sensor drive unit 108. You may.

FIG. 4 is a graph illustrating the phase difference AF used in the present embodiment. FIG. 4 shows a focus detection signal 401 (A image) and a focus detection signal 402 (B image) in a front focus state (a state in which the focus is in front of the in-focus position of the subject). The vertical axis represents the signal strength, and the horizontal axis represents the horizontal position of the pixel of the image sensor 102. The distance information to the subject by the phase difference detection method is the distance 403 (image shift amount) between the images by the focus detection signal 401 and the focus detection signal 402, and the distance from the imaging plane to the ejection pupil at the focus lens position at that time. It can be calculated from. The image shift amount calculation unit 113 outputs the image shift amount to the system control unit 114. The system control unit 114 determines a target focus lens position based on the obtained image shift amount, and outputs the movement direction and the movement amount from the current focus lens position to the optical system drive unit 106 as focus information. The optical system drive unit 106 drives the optical system 101 based on the focus information to adjust the focus.

FIG. 5 is a block diagram showing the configuration of the image pickup signal image processing unit 109. As shown in FIG. 5, the image pickup signal image processing unit 109 includes a memory 501, a development processing unit 502, a patrol NR (noise removal) unit 503, and a geometric deformation unit 504. In addition to these, although not shown, it has a plurality of processing units that perform processing such as shading correction, white balance (WB) correction, gamma correction, chromatic aberration of magnification correction, and scaling. Further, the development processing unit 502, the cyclic NR unit 503, and the geometric deformation unit 504 are each connected to the memory 501, and the respective processing results are temporarily stored in the memory 501 as a work memory.

The development processing unit 502 inputs an A + B image from the memory 501, performs processing such as demosaic processing, gamma correction, and brightness / color generation, and outputs the processed image data to the memory 501, the direct patrol NR unit 503, or both. To do. At this time, the A + B image, which is the input data of the development processing unit 502, is RAW image data, and YUV image data representing the YUV color space is generated by performing development processing (including demosaic processing).

The patrol NR unit 503 is connected to the memory 501, the development processing unit 502, and the geometric deformation unit 504. Then, the patrol NR unit 503 inputs the image data from the development processing unit 502 or the geometric deformation unit 504 and the image data one frame before processed by the patrol NR unit 503 stored in the memory 501 to form a frame. NR processing (noise removal processing) between them is performed. The patrol NR unit 503 outputs the image data after the NR processing to the memory 501.

The geometric deformation unit 504 inputs image data from the memory 501, performs geometric deformation processing, for example, distortion correction for removing distortion of the optical lens, and outputs the processed image data to the memory 501.

Then, the image pickup signal image processing unit 109 processes the image data as described above and outputs the image data stored in the memory 501 to the external storage device 104 and the image display unit 111.

FIG. 6 is a block diagram showing the configuration of the focus detection signal image processing unit 110. As shown in FIG. 6, the focus detection signal image processing unit 110 performs processing such as memory 601, shading correction unit 602, gamma correction unit 603, luminance signal conversion unit 604, and other resizing such as scaling not shown. It is composed of multiple processing units. The shading correction unit 602, the gamma correction unit 603, and the luminance signal conversion unit 604 are each connected to the memory 601 and temporarily store the correction data required for each processing and the signal in the middle of calculation in the memory 601 as a work memory. To do.

The shading correction unit 602 inputs the A image and the B image, reads the correction data stored in advance from the memory 601, performs the shading correction, and outputs the shading correction unit 603 to the gamma correction unit 603.

The gamma correction unit 603 inputs the shading-corrected A image and the B image from the shading correction unit 602, reads out the gamma correction correction data stored in advance from the memory 601 and performs the gamma correction processing to convert the luminance signal. Output to unit 604.

The luminance signal conversion unit 604 inputs the gamma-corrected A image and the B image from the gamma correction unit 603, and uses the memory 601 to form two columns in the horizontal direction and two rows in the vertical direction for each of the A image and the B image. Is added to generate a pseudo luminance signal (Y data) from the RAW image data.

Then, the focus detection signal image processing unit 110 outputs the luminance signals of the A image and the B image processed as described above to the subject detection unit 112 and the image shift amount calculation unit 113.

FIG. 7 is a flowchart showing an imaging operation of the imaging apparatus of the present invention. The photographing operation shown in the flowchart of FIG. 7 is started by turning on a power switch (not shown).

First, in S701, the system control unit 114 performs a preliminary operation. That is, the system control unit 114 detects known parameter evaluation values related to live view such as WB and AE, and sets each related block. Subsequently, in S702, the system control unit 114 sets parameters for the block such as the image sensor 102 based on the evaluation value detection result obtained in S701, and starts the live view. As a result, the image displayed on the image sensor 102 is displayed on the image display unit 111 via the optical system 101.

Next, in S703, the subject detection unit 112 detects the face and notifies the system control unit 114 of the detected face position. The details of S703 will be described later. Subsequently, in S704, the system control unit 114 determines whether or not the switch SW1 is turned on by pressing a shutter button (not shown), for example, halfway, and the preparation for the main shooting is instructed. When the switch SW1 is OFF, the process returns to S702, and when the switch SW1 is ON, the process proceeds to S705.

In S705, the system control unit 114 acquires the image shift amount at the face position (face detection position) from the image shift amount calculation unit 113, and calculates the defocus amount. A method for calculating the defocus amount def is known, and for example, from the image shift amount d, the distance p between each pixel of the image sensor 102, and the value K uniquely determined according to the aperture value of the lens. It is calculated by the following formula (1).
def = d × p × K… (1)

Here, the system control unit 114 outputs a control signal to the optical system drive unit 106 so as to drive the optical system 101 (focus lens) based on the defocus amount calculated using the equation (1). As a result, the focusing position can be moved to the position of the face.

Next, in S706, the system control unit 114 performs a photometric operation based on the signal output from the image sensor 102, and acquires exposure information of the subject. Subsequently, in S707, the system control unit 114 sets shooting conditions such as shutter seconds during shooting and ISO sensitivity based on the exposure information of the subject acquired in S706.

Next, in S708, the system control unit 114 determines whether or not the switch SW2 is turned on by, for example, fully pressing a shutter button (not shown), and recording of a still image is instructed. If the switch SW2 is OFF, the process proceeds to S712. On the other hand, when the switch SW2 is ON, the process proceeds to S709.

In S709, the system control unit 114 controls the image sensor drive unit 108 so as to output drive pulses of the image sensor 102 such as the horizontal synchronization signal HD and the vertical synchronization signal VD. The image sensor 102 performs exposure (accumulation of electric charge) based on this drive pulse. At the same time, known shutter control is performed according to the imaging conditions set in S707. Then, the system control unit 114 controls so as to end the exposure and start reading out the electric charge when the predetermined shutter time elapses.

Next, in S710, various image processing is performed by the image pickup signal image processing unit 109. Subsequently, in S711, the image output to the memory 501 in S710 is output to the external storage device 104, and the stored image is displayed on the image display unit 111.

In S712, the system control unit 114 determines whether or not the switch SW1 has been turned off. When the switch SW1 is maintained in the ON state, the process returns to S708 assuming that the shooting is continued. On the other hand, when the switch SW1 is turned off, the shooting operation is ended and the process proceeds to S713. In S713, the system control unit 114 determines whether or not the power switch (power switch) (not shown) is OFF. When the power switch is turned on, the process returns to S702 and the shooting preparation state is continued in the live view. On the other hand, when the power switch is turned off, the power of the image pickup apparatus of the present invention is turned off.

Next, with reference to FIG. 8, the face detection process performed in S703 of FIG. 7 will be described. FIG. 8 is a flowchart showing the face detection process. First, in S801, the focus detection signal image processing unit 110 performs various signal processing on the A image and the B image, which are the focus detection signals output from the image sensor 102 and separated by the addition signal separation unit 107. The subject detection unit 112 inputs an A image or a B image converted into a luminance signal (Y data) by the luminance signal conversion unit 604 of the focus detection signal image processing unit 110, and a pattern known as to whether or not a face exists. Detect with recognition technology. When a face is detected, the system control unit 114 is notified of the coordinates of the position of the face.

Next, in S802, the system control unit 114 determines whether or not the face has been detected, that is, whether or not the subject detection unit 112 has notified the coordinates of the position of the face. If no face is detected, the face detection process is terminated. On the other hand, if a face is detected, the process proceeds to S803. Subsequently, in S803, the system control unit 114 notifies the image shift amount calculation unit 113 of the coordinate range in which the face is detected. The image shift amount calculation unit 113 calculates the image shift amount (image shift amount at the position of the face) within the coordinate range notified from the system control unit 114, and notifies the system control unit 114 of the image shift amount.

Next, in S804, the system control unit 114 corrects the face position (subject position) detected from the A image or the B image based on the image shift amount calculated by the image shift amount calculation unit 113, and the captured image. Estimate the subject position in. At this time, the system control unit 114 estimates the face position in the captured image according to a predetermined ratio with respect to the image shift amount notified from the image shift amount calculation unit 113, and the image display unit 111 as a correction amount. Notify to. The predetermined ratio is a value caused by the aperture positions of the photoelectric conversion units 201A and 201B that output the A image and the B image used for calculating the image shift amount. The position of the face is corrected by using the correction amount corresponding to this value. When the openings of the photoelectric conversion units 201A and 201B are arranged symmetrically with respect to the center of the optical axis (left-right symmetry), the value d / 2 with respect to the image shift amount d is the center of the face position in the captured image. Become. At this time, the correction amount is half of the image shift amount d.

Next, in S805, the image display unit 111 reads the image data processed by the image pickup signal image processing unit 109 from the memory 501. Then, the image display unit 111 draws a frame (subject detection frame) for notifying the user of the position of the face based on the position of the face designated by the system control unit 114, and outputs this frame. In this way, the image display unit 111 displays the subject detection frame in the area including the position of the face in the captured image estimated by the system control unit 114.

The face detection process is completed by the above process.

If the captured image is not in focus at the face position, the subject is blurred and spreads over a wide range. Therefore, when expanding the frame, the frame may be enlarged and displayed in the image deviation amount direction from the coordinates where the subject is detected. Further, when enlarging in the image shift amount direction, the process may be performed so as to surround the blurred and widened portion by enlarging by the image shift amount d. At this time, the image display unit 111 may display the subject detection frame by enlarging the area including the position of the face in the image data estimated by the system control unit 114.

In the present embodiment, the subject detection process is performed with the subject as a face, but the present invention is not limited to this, and a subject other than the face may be used. Further, although the A + B image and the A image are read out from the image sensor 102, the A + B image and the B image may be read out from the image sensor 102, and the A + B image may be generated later by the addition process. As long as the A + B images can be obtained, the reading method is not particular.

As described above, according to the first embodiment, face detection is performed using the focus detection signal (A image or B image). Since the focus detection signal (A image or B image) has a deeper depth of focus than the image signal (A + B image), even if the defocus amount of the optical system 101 is large, the face is displayed using the focus detection signal with less blur. Can be detected.

<Second embodiment>
Next, a second embodiment of the present invention will be described. FIG. 9 is a block diagram showing a schematic configuration of the image pickup apparatus according to the second embodiment. The difference between the image pickup apparatus shown in FIG. 9 and the image pickup apparatus shown in FIG. 1 is that the focus detection signal image processing unit 110 outputs the A image and the B image to the subject detection unit 112. Since the other parts are the same as those in FIG. 1, the same reference numbers are given and the description thereof will be omitted.

In the first embodiment, a method of correcting the position of the face detected by using the focus detection signal (A image or B image) by the amount of image shift has been described. On the other hand, in the second embodiment, the position of the subject existing in the captured image is estimated and corrected by the calculation without performing the correlation calculation using the A image and the B image.

In the present embodiment, each of the A image and the B image is output from the focus detection signal image processing unit 110 to the subject detection unit 112. The subject detection unit 112 performs face detection on each of the input A image and B image, and outputs the respective coordinates to the system control unit 114. That is, the subject detection unit 112 detects the first subject position from the A image and the second subject position from the B image.

The system control unit 114 outputs to the image display unit 111 the intermediate position between the coordinates of the first subject position and the coordinates of the second subject position detected from the A image and the B image, respectively, as the face position. That is, the system control unit 114 estimates the subject position based on the first subject position and the second subject position. In the present embodiment, the photoelectric conversion units 201A and 201B that output the A image and the B image are arranged symmetrically with respect to the center of the optical axis. Therefore, the system control unit 114 estimates the intermediate position between the first subject position and the second subject position as the subject position (face position).

FIG. 10 is calculated, and a method of calculating the intermediate position will be described. FIG. 10 is a diagram showing the position (coordinates) of the face in the present embodiment. FIG. 10A shows an example of the A image. The coordinates of the rectangular frame of the detected face position are (lx1, ly1) in the upper left part, (lx2, ly1) in the upper right part, (1x1, 1y2) in the lower left part, and (lx2, ly2) in the lower right part. In this case, the range of the face region can be represented by two points in the x direction (lx1, lp2) and two points in the y direction (ly1, ly2).

FIG. 10 (b) shows a B image corresponding to the A image shown in FIG. 10 (a). Similarly, the coordinates of the rectangular frame of the detected face position are (rx1, ry1) in the upper left part, (rx2, ry1) in the upper right part, (rx1, ry2) in the lower left part, and (rx2, ry2) in the lower right part. And. In this case, the range of the face region can be represented by two points in the x direction (rx1, rx2) and two points in the y direction (ry1, ry2).

FIG. 10 (c) is a diagram showing a state in which the A image and the B image are superimposed. The solid line shows the first subject position obtained from the A image, and the dotted line shows the second subject position obtained from the B image. In this case, the system control unit 114 sets the range of the face region (coordinates) existing in the captured image as two points (1x1 + rx1) / 2 and (lx2 + rx2) / 2 in the x direction as shown by the thick line frame. It can be estimated that there are two points in the y direction (ly1 + ry1) / 2, (ly2 + ry2) / 2). In this way, the system control unit 114 takes the average of the first subject position and the second subject position, and notifies the image display unit 111 of the frame indicated by the above-mentioned coordinates.

As described above, according to the second embodiment, the same effect as that of the first embodiment can be obtained without calculating the image shift amount.

It is also conceivable that the face of the captured image is blurred and spreads only at the coordinates detected for each of the A image and the B image. Therefore, the image is enlarged so as to surround the blurred face existing in the captured image by enlarging it into a rectangular frame represented by two points in the x direction (lx1, rx2) and two points in the y direction (ly1, ry2). The unit 111 may be notified.

Further, the captured image is an image obtained by adding the A image and the B image. Therefore, when the amount of light of the A image or the B image is attenuated due to the influence of shading or the like, the position of the center of gravity of the captured image is greatly affected by the image having a large amount of light. Therefore, the subject detection unit 112 may correct the position of the face in the captured image according to the light amount ratio of the A image and the B image by using the formula for obtaining the actual face detection position.

<Other Embodiments>
Even if the present invention is applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a scanner, a video camera, etc.), the present invention is a device composed of one device (for example, a copier, a facsimile machine, etc.). May be applied to.

The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device implement the program. It can also be realized by the process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

101: Optical system, 102: Image sensor, 103: Signal processing unit, 104: External storage device, 105: Sensor correction unit, 106: Optical system drive unit, 107: Addition signal separation unit, 108: Image sensor drive unit, 109 : Image sensor image processing unit, 110: Focus detection signal image processing unit, 111: Image display unit, 112: Subject detection unit, 113: Image shift amount calculation unit, 114: System control unit

Claims (12)

A signal processing device that processes a pair of parallax focus detection signals and image signals that can be acquired from an image sensor equipped with a plurality of photoelectric conversion units for each of the plurality of microlenses.
A subject detection means for detecting a predetermined subject using at least one of the pair of focus detection signals, and a subject detection means.
It has a focus detection means for detecting a focus state using the pair of focus detection signals.
When the subject is detected by the subject detection means, the focus detection means detects the focus state by using the pair of focus detection signals corresponding to the region of the subject among the pair of focus detection signals. A signal processing device characterized by. Further, a first processing means for converting the pair of focus detection signals into luminance signals is further provided.
The signal processing device according to claim 1, wherein the subject detection means and the focus detection means perform detection using the luminance signals of the pair of focus detection signals. The subject detection means detects the subject by using one of the pair of focus detection signals.
When the subject is detected, the focus detecting means detects the amount of deviation of the pair of focus detection signals corresponding to the region of the subject as the focal state.
The first or second aspect of the present invention, further comprising an estimation means for estimating the position of the subject in the image based on the image signal based on the deviation amount and the opening positions of the plurality of photoelectric conversion units. Signal processing device. The signal processing device according to claim 3, wherein the estimation means further expands the area of the subject by the magnitude of the deviation amount in the direction of the deviation amount. The subject detection means detects the subject by using the pair of focus detection signals, respectively.
It is characterized by further having an estimation means for estimating the position of the subject in the image based on the image signal from the first position and the second position of the subject detected from the pair of focus detection signals, respectively. The signal processing apparatus according to claim 1 or 2. The signal processing apparatus according to claim 5, wherein the estimation means estimates a position obtained by averaging the first position and the second position as the position of the subject in an image based on the image signal. .. 5. The estimation means is further characterized in that the region including the regions of the subject detected from the pair of focus detection signals is estimated as the region of the subject in the image based on the image signal. The signal processing apparatus according to. A second processing means for processing the image signal obtained by adding signals to each of the plurality of photoelectric conversion units, and
Further, it has a display means for displaying an image based on the image signal output from the second processing means.
Any one of claims 1 to 7, wherein when the subject is detected by the subject detecting means, the display means superimposes a display indicating a region of the subject on the image signal. The signal processing apparatus according to. The signal processing device according to any one of claims 1 to 8.
An image pickup apparatus characterized in that a plurality of photoelectric conversion units are provided for each of the plurality of microlenses, and a pair of focus detection signals having parallax and an image pickup element that can acquire an image signal are provided. A signal processing method for processing a pair of focus detection signals having parallax, which can be acquired from an image sensor provided with a plurality of photoelectric conversion units for each of a plurality of microlenses.
A subject detection step in which the subject detection means detects a predetermined subject by using at least one of the pair of focus detection signals.
The focus detection means includes a focus detection step of detecting a focus state using the pair of focus detection signals.
When the subject is detected in the subject detection step, in the focus detection step, the focus state is detected by using the pair of focus detection signals corresponding to the region of the subject among the pair of focus detection signals. A signal processing method characterized by. A program for causing a computer to function as each means of the signal processing device according to any one of claims 1 to 8. A computer-readable storage medium that stores the program according to claim 11.

Images