JP6479149B2 - IMAGING DEVICE, IMAGING DEVICE CONTROL METHOD, AND IMAGE PROCESSING DEVICE - Google Patents

Description

The present invention relates to an imaging apparatus , a method for controlling the imaging apparatus, and an image processing apparatus .

  An imaging apparatus equipped with a technique called Computational Photography has been studied. In addition to image brightness and color data, computational photography captures new information that has not been recorded by conventional imaging devices with an image sensor, records it with image data, and uses the recorded information to create an image. This is a technique of performing various processes with a processing LSI. For example, the imaging apparatus records light information (light field) such as information on the incident angle of light from the subject and depth. By using information that could not be recorded by these conventional imaging devices, focus, depth of field by iris, focal length, and the like can be accurately reproduced by subsequent image processing. There has been proposed a camera that can change the in-focus position (focus position) of an actually captured image later using a recorded light field.

  As a typical light field acquisition method, a method for acquiring images from multiple viewpoints will be described. For example, a plurality of images can be acquired by changing the focal length with a single imaging device, and the acquired plurality of images can be left as a single image so that the focal length of the image can be freely changed later by image processing. Can be edited. Further, by moving the camera horizontally or vertically and acquiring a plurality of images, it is also possible to acquire subject distance information from the parallax of the images. These methods use a single imaging device (imaging device). Patent Document 1 discloses a system that acquires and processes light fields from a plurality of cameras.

JP 2008-257686A

  In a system in which a plurality of cameras are provided to acquire a light field, power consumption increases and the apparatus also increases in size. Therefore, an imaging device that obtains information on the light field, that is, the direction of incident light, from an image signal that is output by photoelectrically converting incident light through a plurality of lenses having different focal lengths with an imaging device is conceivable.

  On the other hand, in an imaging apparatus in which the user of the imaging apparatus can edit the zoom position, the focus position, and the depth of field after acquiring the image, it is important to output an image that matches the user's shooting intention. However, when the user wants to follow a specific person as the main subject, what kind of image is output by the imaging apparatus that allows the user to edit the zoom position, focus position, and depth of field after the image is acquired. For example, it is difficult to determine whether the image is intended by the user.

  According to the present invention, in an imaging apparatus having imaging elements with different focal lengths in one imaging apparatus, the imaging device used for detection in the imaging apparatus is sequentially changed according to a predetermined subject detection mode, and the light field is changed while changing the focal length. It is an object of the present invention to provide a user with an easy-to-understand image indicating where the focus is in focus by performing image processing.

Imaging device according to an embodiment of the present invention, the incident light is the focal length through different lenses, acquired a plurality of image pickup unit for photoelectrically converting each of the at least one imaging unit of said plurality of image pickup unit a detection means performs the detection process of detecting an object from an image signal, and output means for outputting an image to focus on the detected object by said detecting means, wherein the detecting means comprises a first focal length When the subject is detected based on the image signal corresponding to, the subject detection process from the image signal corresponding to the second focal length shorter than the first focal length is not performed. The detection process ends.

  According to the imaging apparatus of the present invention, in an imaging apparatus having imaging elements with different focal lengths in one imaging apparatus, the imaging elements used for detection in the imaging apparatus are sequentially changed according to a predetermined subject detection mode, and the focal length is changed. By acquiring the light field while changing the image and performing image processing, it is possible to provide the user with an easy-to-understand video indicating where the focus is in focus.

It is a figure which shows the structural example of the imaging device of this embodiment. It is a figure explaining the lens and imaging device which were provided in the imaging device. 3 is a flowchart for explaining an operation process of the imaging apparatus according to the first embodiment. 6 is a flowchart illustrating an operation process of the imaging apparatus according to the second embodiment.

Example 1
FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus according to the present embodiment.
The imaging apparatus shown in FIG. 1 is a digital video camera. Of course, the application range of the present invention is not limited to a digital video camera. The present invention is also applicable to a digital still camera, a digital single-lens reflex camera, and the like.

  The imaging apparatus according to this embodiment includes an imaging optical system 101, an imaging element 102, an image processing LSI 103, a signal recording unit 104, a light field image processing unit 105, a focus control unit 106, an imaging element control unit 107, a system control unit 108, a display. Part 109 is provided.

  The system control unit 108 controls the entire image sensor. The imaging optical system 101 has a mechanism for capturing subject light and changing at least the focus position. The image sensor 102 photoelectrically converts the captured light into an analog electrical signal (image signal) and outputs the analog electrical signal.

  In order to be able to acquire an arbitrary light field based on an image signal output from the image sensor 102, the image pickup apparatus includes a plurality of image sensors 102, and each image sensor has a lens ( In this example, a microlens) is provided. That is, the light field (information regarding the direction of incident light) can be obtained from the signal output from the image sensor 102 by the structure of the image pickup optical system 101 and the image sensor 102. As another form, the imaging device may include one imaging device, and a microlens may be provided in each of a plurality of imaging units included in the imaging device.

  The image processing LSI 103 converts the image signal (video signal) output from the image sensor 102 into a digital signal, and performs image processing such as digital gain, gamma, and knee on the digitized video signal. The image processing LSI 103 measures the black level output from the image sensor 102 and clamps the video signal.

  The signal recording unit 104 records a video signal digitally processed by the image processing LSI. The image processing LSI 105 executes arbitrary image processing on the video signal using light field information obtained from the video signal output from the image sensor 102. The focus control unit 106 drives a focus lens included in the imaging optical system 101 to execute a focus adjustment process for changing the focus position.

  The image sensor control unit 107 executes control of the image sensor 102. The display unit 109 displays a signal output from the image processing LSI 103 or a signal output from the light field image processing unit 105 as an image. Each of the processing units illustrated in FIG. 1 is not necessarily a divided LSI. The imaging apparatus may include an LSI that integrates a plurality of processing units into one.

Details of each processing unit will be described below.
The imaging optical system 101 passes light from the subject to the imaging element 102. In this example, the imaging optical system 101 includes a focus lens. Depending on the form of the imaging device, the imaging optical system 101 may include an iris unit that controls incident light, a zoom lens unit that changes the focal length, and the like.

  The imaging element 102 receives incident light from the imaging optical system, converts the incident light into an electrical signal, and outputs the electrical signal. The imaging element 102 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor.

  Video signals output from the image sensor 102 include those that are directly output as analog video signals and those that are A / D-converted inside the image sensor 102 and output as digital data such as LVDS. is there. LVDS is an abbreviation for Low Voltage Differential Signaling.

  The image processing LSI 103 includes an analog front end unit that converts an analog electrical signal output from the image sensor 102 into a digital signal, and a processing unit that performs each process on the digitized video signal. When AD conversion processing is performed inside the image sensor 102, the analog front end unit is omitted.

  For example, when the image sensor 102 is a CMOS image sensor, the image processing LSI 103 performs removal of fixed pattern noise unique to the CMOS image sensor, black level clamp processing, and the like. The image processing LSI 103 performs various image processing such as a pixel addition function, noise reduction, gamma correction, knee, digital gain, and flaw correction, which are typical image processing functions of the imaging apparatus. The image processing LSI 103 also has a storage circuit that stores setting values necessary for each correction and image processing.

  The signal recording unit 104 records the video signal output from the image processing LSI 103 and reflecting each image processing, and the light field information in an arbitrary recording unit. The recording means is, for example, an HDD (hard disk drive), an SD memory card, a compact flash (registered trademark), or the like.

  The light field image processing unit 105 performs arbitrary image processing such as focus position, zoom position, and depth of field adjustment of the imaging optical system based on a light field obtained based on a signal output from the image sensor. The processing unit that acquires the light field may be either the image processing LSI 103 or the light field image processing unit 105.

  The focus control unit 106 drives and controls a focus lens included in the imaging optical system 101. Note that the imaging apparatus may be configured not to include the focus control unit 106. The image sensor control unit 107 controls the image sensor 101. Specifically, the image sensor control unit 107 determines a method such as an exposure process via the system control unit based on the image processing result by the image processing LSI, and controls the image sensor.

  Based on the exposure information received from the image processing LSI 103, the system control unit 108 determines control for setting the image capturing apparatus to a target exposure, and instructs the focus control unit 106 and the image sensor control unit 107.

  The display unit 109 displays a video (image). This indicates to the user what kind of video the imaging apparatus is currently recording. The display unit 109 is, for example, an external monitor, a liquid crystal monitor attached to the imaging device, a viewfinder, or the like.

FIG. 2 is a diagram illustrating a lens and an imaging element provided in the imaging apparatus of the present embodiment.
As shown in FIG. 2A, a plurality of lenses (microlenses) and imaging elements (sensors) having different focal lengths are arranged in a 4 × 4 matrix on the front surface of the imaging apparatus. FIG. 1B shows the focal length of each lens. The first lens is a lens with a focal length A, the second lens is a lens with a focal length B, the third lens is a lens with a focal length C, and the fourth lens is a lens with a focal length D.

  The lens with the focal length A has the shortest focal length. That is, the lens with the focal length A is the lens on the widest side. The focal length increases as the lens moves to the focal length B lens, the focal length C lens, and the focal length D lens. The lens with the focal length D has the longest focal length. That is, the lens with the focal length D is the lens on the most Tele side.

  The imaging apparatus can change the focal length arbitrarily by the light field image processing unit 105 by preparing an imaging device with a changed focal length and recording images with a plurality of angles of view. In addition, the imaging apparatus can obtain the distance information of the subject by using the parallax information of the light by arranging the imaging elements having the same focal length at different positions. The imaging apparatus can control the depth of field of the image by using the distance information of the subject to adjust the focus position or to calculate the background blur by image processing.

FIG. 3 is a flowchart illustrating an operation process of the imaging apparatus according to the first embodiment.
First, the system control unit 108 determines whether the operation mode of the imaging device is the face detection priority mode (step S1). The face detection priority mode is an operation mode for executing face detection processing for detecting a face as a subject from an image output from the image sensor 102. The user can set the operation mode to the face detection priority mode by operating a predetermined operation unit (not shown) included in the imaging apparatus. The reason for executing the face detection process is to determine what kind of subject the user wants to capture (want) and to perform processing such as preferential focusing and exposure adjustment. The detection target is not limited to the face. The imaging device may detect an object such as a human body or a pet as a subject.

  If the operation mode of the imaging apparatus is not the face detection priority mode, the process proceeds to step S5. Then, the system control unit 108 outputs an arbitrary video related to the video signal output from the image sensor 102 to the display unit 109.

  If the operation mode of the imaging apparatus is the face detection priority mode, the process proceeds to step S2. Then, the system control unit 108 analyzes the video signal obtained through the lens and the image sensor having the longest focal length among the plurality of lenses and the image sensor 102, and performs face detection for the video related to the video signal. Processing is executed (step S2).

  Next, the system control unit 108 determines whether a face has been detected (step S3). If a face is detected, the process proceeds to step S4. The system control unit 108 uses the light field obtained from the video signal output from the image sensor 102 to generate a video in which the detected face is in focus (step S4). Then, the system control unit 108 outputs the generated video to the display unit 109.

  If no face is detected, the process proceeds to step S6. The system control unit 108 determines whether the image sensor corresponding to the image currently being processed is the image sensor (the widest angle sensor) corresponding to the lens with the shortest focal length (step S6). If the image sensor corresponding to the image currently being processed is the widest angle sensor, the process proceeds to step S5. Then, the system control unit 108 outputs an arbitrary video related to the video signal output from the image sensor 102 to the display unit 109.

If the image sensor corresponding to the image currently being processed is not the widest angle sensor, the process proceeds to step S7. Then, the image corresponding to the image sensor corresponding to the lens with the next shortest focal length is set as the target of face detection processing (step S7), and the processing proceeds to step S2.

  The imaging apparatus according to the first exemplary embodiment sequentially performs a process for detecting a subject from an image corresponding to a lens based on an image signal output from the imaging element 102 for each focal length of the lens. Specifically, the imaging device analyzes the face by sequentially shifting from the image of the image sensor corresponding to the lens with a long focal length to the image of the image sensor with a short focal length, and detects when the face is detected. An image focused on the face is generated and displayed using a light field. Therefore, the user can confirm an image in which the face in the center is focused, and shooting is easy. Note that, depending on the subject to be detected, the face detection process may be sequentially performed from the image corresponding to the lens having a short focal length.

(Example 2)
FIG. 4 is a flowchart illustrating an operation process of the imaging apparatus according to the second embodiment.
First, the system control unit 108 determines whether the operation mode of the imaging device is the face detection priority mode (step S11). If the operation mode of the imaging apparatus is the face detection priority mode, the process proceeds to step S12. Then, the system control unit 108 sets a video signal obtained through the lens and the image sensor having the longest focal length among the plurality of lenses and the image sensor as an analysis target (step S12).

  Steps S13, S14, S17, and S18 are the same as steps S2, S3, S6, and S7 in FIG. 3, respectively. If it is determined in step S17 that the image sensor corresponding to the image currently being processed is not the widest-angle sensor, the process proceeds to step S19.

  If a face is detected in the determination process of step S14, the process proceeds to step S15. Then, the system control unit 108 controls the focus control unit 106 so that the detected face is in focus (step S15). Then, the system control unit 108 outputs to the display unit 109 an image in which the in-focus position is matched with the face in step S15 (step S16).

  If it is determined in step S11 that the operation mode of the imaging apparatus is not the face detection priority mode, the process proceeds to step S19. Then, the system control unit 108 calculates a control amount for adjusting the in-focus position to the subject included in the video related to the video signal output from the widest angle sensor (step S19). This is because the video from the widest angle sensor has the largest amount of information on the subject. Subsequently, the system control unit 108 controls the focus control unit 106 using the control amount calculated in step S19 to move the focus position on the subject (step S20). Then, the system control unit 108 outputs an image with the in-focus position on the subject to the display unit 109 (step S21).

  The imaging apparatus according to the second embodiment sequentially switches the face detection process from the image corresponding to the telephoto lens to the image corresponding to the wide-angle lens, and the image in focus on the detected face is driven by the focus lens. Acquire and output to the display unit. Therefore, the user can confirm an image in which the face in the center is in focus, and shooting is easier.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

101 imaging optical system 102 imaging element 103 image processing LSI
106 Focus control unit 107 Image sensor control unit 108 System control unit 109 Display unit

Claims (13)

The incident light which the focal length through different lenses, and a plurality of the imaging unit for photoelectrically converting each
Detecting means for executing detection processing for detecting a subject from an image signal acquired from at least one of the plurality of imaging units;
Has an output means for outputting an image to focus on the detected object by said detecting means,
When the subject is detected based on the image signal corresponding to the first focal length, the detection means uses the image signal corresponding to the second focal length that is shorter than the first focal length. The imaging apparatus is characterized in that the detection process is terminated without performing the subject detection process . The imaging apparatus according to claim 1, wherein the detection unit sequentially executes the detection process for each focal length of a corresponding lens until an object is detected .   3. The detection unit according to claim 1, wherein among the plurality of image signals, the detection unit sequentially performs the detection processing from an image signal corresponding to incident light through a lens having the longest focal length. Imaging device.   3. The detection unit according to claim 1, wherein among the plurality of image signals, the detection unit sequentially performs the detection processing from an image signal corresponding to incident light through a lens having the shortest focal length. Imaging device. If the subject cannot be detected from the image signal to be detected, the detection means detects the image signal corresponding to the lens having the shortest focal length next to the focal length of the lens corresponding to the image signal. The imaging apparatus according to claim 1, wherein the imaging apparatus is a processing target. When the object is detected from the image by the detection means, the in-focus position is in alignment with the detected object based on the information on the direction of the incident light obtained based on the image signal output from the imaging unit. Comprising generating means for generating an image;
The imaging apparatus according to claim 1, wherein the output unit outputs an image generated by the generation unit. A control means for performing focus adjustment processing by driving and controlling the focus lens;
The control means performs the focus adjustment process so that the in-focus position is aligned with the detected subject when the subject is detected from the image by the detection means,
The imaging apparatus according to claim 1, wherein the output unit outputs an image in which an in-focus position is aligned with the subject by the focus adjustment process.   The imaging apparatus according to claim 1, wherein the plurality of adjacent imaging units have different focal lengths.   The imaging apparatus according to claim 1, wherein the plurality of imaging units are arranged horizontally or vertically.   The plurality of imaging units are arranged so that a lens having a relatively long focal length is provided on the right side of a lens having a relatively short focal length in front of the imaging device. The imaging device according to any one of 1 to 9.   The plurality of imaging units are arranged so that a lens having a relatively long focal length is provided below a lens having a relatively short focal length in front of the imaging device. Item 10. The imaging device according to any one of Items 1 to 9. The incident light which the focal length through a plurality of different lenses, each method of controlling an imaging apparatus including a plurality of imaging units for photoelectrically converting,
A detection step of executing a detection process of detecting a subject from an image signal acquired from at least one of the plurality of imaging units;
An output step of outputting an image focused on the subject detected in the detection step ;
Have
In the detection step, when a subject is detected based on an image signal corresponding to the first focal length, an image signal corresponding to a second focal length that is shorter than the first focal length is used. A method for controlling an imaging apparatus, wherein the detection process is terminated without performing the subject detection process . The incident light which the focal length through different lenses, the detection means performs the detection process of detecting an object from an image signal obtained from at least one imaging unit of the plurality of the imaging unit for photoelectrically converting each
Has an output means for outputting an image to focus on the detected object by said detecting means,
When the subject is detected based on the image signal corresponding to the first focal length, the detection means uses the image signal corresponding to the second focal length that is shorter than the first focal length. The image processing apparatus is characterized in that the detection process is terminated without performing the subject detection process .

Images