JP2015122677A - Imaging apparatus and control method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus which comprises plural lenses with different focal distances, accurately determined a main subject intended by a user, and outputs an image along a user's intention.SOLUTION: The imaging apparatus comprises: the plural lenses with different focal distances; and plural imaging devices 102 for photoelectric conversion of incident light having passed through the lenses. The imaging apparatus successively executes, for each focal distance of the lenses, detection of a subject from images corresponding to the lenses based on image signals outputted by the imaging device 102. Then, when a subject is detected from the image, the imaging apparatus outputs an image focused on the detected subject.

Description

  The present invention relates to an imaging apparatus and a control method thereof.

  An imaging apparatus equipped with a technique called Computational Photography has been studied. In addition to image brightness and color data, computational photography captures new information that has not been recorded by conventional imaging devices with an image sensor, records it with image data, and uses the recorded information to create an image. This is a technique of performing various processes with a processing LSI. For example, the imaging apparatus records light information (light field) such as information on the incident angle of light from the subject and depth. By using information that could not be recorded by these conventional imaging devices, focus, depth of field by iris, focal length, and the like can be accurately reproduced by subsequent image processing. There has been proposed a camera that can change the in-focus position (focus position) of an actually captured image later using a recorded light field.

  As a typical light field acquisition method, a method for acquiring images from multiple viewpoints will be described. For example, a plurality of images can be acquired by changing the focal length with a single imaging device, and the acquired plurality of images can be left as a single image so that the focal length of the image can be freely changed later by image processing. Can be edited. Further, by moving the camera horizontally or vertically and acquiring a plurality of images, it is also possible to acquire subject distance information from the parallax of the images. These methods use a single imaging device (imaging device). Patent Document 1 discloses a system that acquires and processes light fields from a plurality of cameras.

JP 2008-257686A

  In a system in which a plurality of cameras are provided to acquire a light field, power consumption increases and the apparatus also increases in size. Therefore, an imaging device that obtains information on the light field, that is, the direction of incident light, from an image signal that is output by photoelectrically converting incident light through a plurality of lenses having different focal lengths with an imaging device is conceivable.

  On the other hand, in an imaging apparatus in which the user of the imaging apparatus can edit the zoom position, the focus position, and the depth of field after acquiring the image, it is important to output an image that matches the user's shooting intention. However, when the user wants to follow a specific person as the main subject, what kind of image is output by the imaging apparatus that allows the user to edit the zoom position, focus position, and depth of field after the image is acquired. For example, it is difficult to determine whether the image is intended by the user.

  An object of the present invention is to provide an imaging apparatus that includes a plurality of lenses having different focal lengths, and that accurately determines a main subject intended by a user and outputs an image in which the main subject is in focus. And

  An imaging apparatus according to an embodiment of the present invention includes a plurality of lenses having different focal lengths, a plurality of imaging units that photoelectrically convert incident light through the lenses, and an image signal that is output from the imaging unit. A detection unit that sequentially executes detection processing of the subject from the corresponding image for each focal length of the lens, and when the subject is detected from the image by the detection unit, the in-focus position is aligned with the detected subject. Output means for outputting an image.

  According to the imaging apparatus of the present invention, it is possible to accurately determine the main subject intended by the user and output an image in which the main subject is in focus.

It is a figure which shows the structural example of the imaging device of this embodiment. It is a figure explaining the lens and imaging device which were provided in the imaging device. 3 is a flowchart for explaining an operation process of the imaging apparatus according to the first embodiment. 6 is a flowchart illustrating an operation process of the imaging apparatus according to the second embodiment.

Example 1
FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus according to the present embodiment.
The imaging apparatus shown in FIG. 1 is a digital video camera. Of course, the application range of the present invention is not limited to a digital video camera. The present invention is also applicable to a digital still camera, a digital single-lens reflex camera, and the like.

  The imaging apparatus according to this embodiment includes an imaging optical system 101, an imaging element 102, an image processing LSI 103, a signal recording unit 104, a light field image processing unit 105, a focus control unit 106, an imaging element control unit 107, a system control unit 108, a display. Part 109 is provided.

  The system control unit 108 controls the entire image sensor. The imaging optical system 101 has a mechanism for capturing subject light and changing at least the focus position. The image sensor 102 photoelectrically converts the captured light into an analog electrical signal (image signal) and outputs the analog electrical signal.

  In order to be able to acquire an arbitrary light field based on an image signal output from the image sensor 102, the image pickup apparatus includes a plurality of image sensors 102, and each image sensor has a lens ( In this example, a microlens) is provided. That is, the light field (information regarding the direction of incident light) can be obtained from the signal output from the image sensor 102 by the structure of the image pickup optical system 101 and the image sensor 102. As another form, the imaging device may include one imaging device, and a microlens may be provided in each of a plurality of imaging units included in the imaging device.

  The image processing LSI 103 converts the image signal (video signal) output from the image sensor 102 into a digital signal, and performs image processing such as digital gain, gamma, and knee on the digitized video signal. The image processing LSI 103 measures the black level output from the image sensor 102 and clamps the video signal.

  The signal recording unit 104 records a video signal digitally processed by the image processing LSI. The image processing LSI 105 executes arbitrary image processing on the video signal using light field information obtained from the video signal output from the image sensor 102. The focus control unit 106 drives a focus lens included in the imaging optical system 101 to execute a focus adjustment process for changing the focus position.

  The image sensor control unit 107 executes control of the image sensor 102. The display unit 109 displays a signal output from the image processing LSI 103 or a signal output from the light field image processing unit 105 as an image. Each of the processing units illustrated in FIG. 1 is not necessarily a divided LSI. The imaging apparatus may include an LSI that integrates a plurality of processing units into one.

Details of each processing unit will be described below.
The imaging optical system 101 passes light from the subject to the imaging element 102. In this example, the imaging optical system 101 includes a focus lens. Depending on the form of the imaging device, the imaging optical system 101 may include an iris unit that controls incident light, a zoom lens unit that changes the focal length, and the like.

  The imaging element 102 receives incident light from the imaging optical system, converts the incident light into an electrical signal, and outputs the electrical signal. The imaging element 102 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor.

  Video signals output from the image sensor 102 include those that are directly output as analog video signals and those that are A / D-converted inside the image sensor 102 and output as digital data such as LVDS. is there. LVDS is an abbreviation for Low Voltage Differential Signaling.

  The image processing LSI 103 includes an analog front end unit that converts an analog electrical signal output from the image sensor 102 into a digital signal, and a processing unit that performs each process on the digitized video signal. When AD conversion processing is performed inside the image sensor 102, the analog front end unit is omitted.

  For example, when the image sensor 102 is a CMOS image sensor, the image processing LSI 103 performs removal of fixed pattern noise unique to the CMOS image sensor, black level clamp processing, and the like. The image processing LSI 103 performs various image processing such as a pixel addition function, noise reduction, gamma correction, knee, digital gain, and flaw correction, which are typical image processing functions of the imaging apparatus. The image processing LSI 103 also has a storage circuit that stores setting values necessary for each correction and image processing.

  The signal recording unit 104 records the video signal output from the image processing LSI 103 and reflecting each image processing, and the light field information in an arbitrary recording unit. The recording means is, for example, an HDD (hard disk drive), an SD memory card, a compact flash (registered trademark), or the like.

  The light field image processing unit 105 performs arbitrary image processing such as focus position, zoom position, and depth of field adjustment of the imaging optical system based on a light field obtained based on a signal output from the image sensor. The processing unit that acquires the light field may be either the image processing LSI 103 or the light field image processing unit 105.

  The focus control unit 106 drives and controls a focus lens included in the imaging optical system 101. Note that the imaging apparatus may be configured not to include the focus control unit 106. The image sensor control unit 107 controls the image sensor 101. Specifically, the image sensor control unit 107 determines a method such as an exposure process via the system control unit based on the image processing result by the image processing LSI, and controls the image sensor.

  Based on the exposure information received from the image processing LSI 103, the system control unit 108 determines control for setting the image capturing apparatus to a target exposure, and instructs the focus control unit 106 and the image sensor control unit 107.

  The display unit 109 displays a video (image). This indicates to the user what kind of video the imaging apparatus is currently recording. The display unit 109 is, for example, an external monitor, a liquid crystal monitor attached to the imaging device, a viewfinder, or the like.

FIG. 2 is a diagram illustrating a lens and an imaging element provided in the imaging apparatus of the present embodiment.
As shown in FIG. 2A, a plurality of lenses (microlenses) and imaging elements (sensors) having different focal lengths are arranged in a 4 × 4 matrix on the front surface of the imaging apparatus. FIG. 1B shows the focal length of each lens. The first lens is a lens with a focal length A, the second lens is a lens with a focal length B, the third lens is a lens with a focal length C, and the fourth lens is a lens with a focal length D.

  The lens with the focal length A has the shortest focal length. That is, the lens with the focal length A is the lens on the widest side. The focal length increases as the lens moves to the focal length B lens, the focal length C lens, and the focal length D lens. The lens with the focal length D has the longest focal length. That is, the lens with the focal length D is the lens on the most Tele side.

  The imaging apparatus can change the focal length arbitrarily by the light field image processing unit 105 by preparing an imaging device with a changed focal length and recording images with a plurality of angles of view. In addition, the imaging apparatus can obtain the distance information of the subject by using the parallax information of the light by arranging the imaging elements having the same focal length at different positions. The imaging apparatus can control the depth of field of the image by using the distance information of the subject to adjust the focus position or to calculate the background blur by image processing.

FIG. 3 is a flowchart illustrating an operation process of the imaging apparatus according to the first embodiment.
First, the system control unit 108 determines whether the operation mode of the imaging device is the face detection priority mode (step S1). The face detection priority mode is an operation mode for executing face detection processing for detecting a face as a subject from an image output from the image sensor 102. The user can set the operation mode to the face detection priority mode by operating a predetermined operation unit (not shown) included in the imaging apparatus. The reason for executing the face detection process is to determine what kind of subject the user wants to capture (want) and to perform processing such as preferential focusing and exposure adjustment. The detection target is not limited to the face. The imaging device may detect an object such as a human body or a pet as a subject.

  If the operation mode of the imaging apparatus is not the face detection priority mode, the process proceeds to step S5. Then, the system control unit 108 outputs an arbitrary video related to the video signal output from the image sensor 102 to the display unit 109.

  If the operation mode of the imaging apparatus is the face detection priority mode, the process proceeds to step S2. Then, the system control unit 108 analyzes the video signal obtained through the lens and the image sensor having the longest focal length among the plurality of lenses and the image sensor 102, and performs face detection for the video related to the video signal. Processing is executed (step S2).

  Next, the system control unit 108 determines whether a face has been detected (step S3). If a face is detected, the process proceeds to step S4. The system control unit 108 uses the light field obtained from the video signal output from the image sensor 102 to generate a video in which the detected face is in focus (step S4). Then, the system control unit 108 outputs the generated video to the display unit 109.

  If no face is detected, the process proceeds to step S6. The system control unit 108 determines whether the image sensor corresponding to the image currently being processed is the image sensor (the widest angle sensor) corresponding to the lens with the shortest focal length (step S6). If the image sensor corresponding to the image currently being processed is the widest angle sensor, the process proceeds to step S5. Then, the system control unit 108 outputs an arbitrary video related to the video signal output from the image sensor 102 to the display unit 109.

  If the image sensor corresponding to the image currently being processed is not the widest angle sensor, the process proceeds to step S7. Then, the image corresponding to the image sensor corresponding to the lens with the next long focal distance is set as the target of face detection processing (step S7), and the processing proceeds to step S2.

  The imaging apparatus according to the first exemplary embodiment sequentially performs a process for detecting a subject from an image corresponding to a lens based on an image signal output from the imaging element 102 for each focal length of the lens. Specifically, the imaging device analyzes the face by sequentially shifting from the image of the image sensor corresponding to the lens with a long focal length to the image of the image sensor with a short focal length, and detects when the face is detected. An image focused on the face is generated and displayed using a light field. Therefore, the user can confirm an image in which the face in the center is focused, and shooting is easy. Note that, depending on the subject to be detected, the face detection process may be sequentially performed from the image corresponding to the lens having a short focal length.

(Example 2)
FIG. 4 is a flowchart illustrating an operation process of the imaging apparatus according to the second embodiment.
First, the system control unit 108 determines whether the operation mode of the imaging device is the face detection priority mode (step S11). If the operation mode of the imaging apparatus is the face detection priority mode, the process proceeds to step S12. Then, the system control unit 108 sets a video signal obtained through the lens and the image sensor having the longest focal length among the plurality of lenses and the image sensor as an analysis target (step S12).

  Steps S13, S14, S17, and S18 are the same as steps S2, S3, S6, and S7 in FIG. 3, respectively. If it is determined in step S17 that the image sensor corresponding to the image currently being processed is not the widest-angle sensor, the process proceeds to step S19.

  If a face is detected in the determination process of step S14, the process proceeds to step S15. Then, the system control unit 108 controls the focus control unit 106 so that the detected face is in focus (step S15). Then, the system control unit 108 outputs to the display unit 109 an image in which the in-focus position is matched with the face in step S15 (step S16).

  If it is determined in step S11 that the operation mode of the imaging apparatus is not the face detection priority mode, the process proceeds to step S19. Then, the system control unit 108 calculates a control amount for adjusting the in-focus position to the subject included in the video related to the video signal output from the widest angle sensor (step S19). This is because the video from the widest angle sensor has the largest amount of information on the subject. Subsequently, the system control unit 108 controls the focus control unit 106 using the control amount calculated in step S19 to move the focus position on the subject (step S20). Then, the system control unit 108 outputs an image with the in-focus position on the subject to the display unit 109 (step S21).

  The imaging apparatus according to the second embodiment sequentially switches the face detection process from the image corresponding to the telephoto lens to the image corresponding to the wide-angle lens, and the image in focus on the detected face is driven by the focus lens. Acquire and output to the display unit. Therefore, the user can confirm an image in which the face in the center is in focus, and shooting is easier.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

101 imaging optical system 102 imaging element 103 image processing LSI
106 Focus control unit 107 Image sensor control unit 108 System control unit 109 Display unit

Claims (8)

Multiple lenses with different focal lengths;
A plurality of imaging units that photoelectrically convert incident light through the lens;
Detection means for sequentially executing detection processing of a subject from an image corresponding to the lens based on an image signal output from the imaging unit for each focal length of the lens;
An image pickup apparatus comprising: an output unit configured to output an image in which an in-focus position is aligned with the detected subject when the subject is detected from the image by the detection unit. The imaging apparatus according to claim 1, wherein the detection unit is a target for subject detection processing from an image corresponding to the lens in order of increasing focal length. When the subject cannot be detected from the image corresponding to the lens, the detection means sets the image corresponding to the lens having the second focal length next to the focal length of the lens as a subject of the subject detection processing. The imaging device according to claim 1 or 2. The detection unit determines whether the lens is a lens having the shortest focal length when a subject cannot be detected from an image corresponding to the lens, and if the lens is not the lens having the shortest focal length, the lens The imaging apparatus according to claim 3, wherein an image corresponding to a lens having a long focal length next to the focal length is set as a subject of the subject detection process. The imaging apparatus according to claim 1, wherein the detection unit is a target for subject detection processing from an image corresponding to the lens in order of decreasing focal length. When the object is detected from the image by the detection means, the in-focus position is in alignment with the detected object based on the information on the direction of the incident light obtained based on the image signal output from the imaging unit. Comprising generating means for generating an image;
The imaging apparatus according to any one of claims 1 to 5, wherein the output unit outputs an image generated by the generation unit. A control means for performing focus adjustment processing by driving and controlling the focus lens;
The control means performs the focus adjustment process so that the in-focus position is aligned with the detected subject when the subject is detected from the image by the detection means,
The imaging apparatus according to any one of claims 1 to 5, wherein the output unit outputs an image in which a focus position is aligned with the subject by the focus adjustment process. A control method of an imaging apparatus comprising a plurality of lenses having different focal lengths and a plurality of imaging units that photoelectrically convert incident light through the lenses,
A detection step of sequentially executing subject detection processing from an image corresponding to the lens based on an image signal output by the imaging unit for each focal length of the lens;
A control method comprising: an output step of outputting an image in which the detected subject is in focus when the subject is detected from the image by the detection step.

Images