JP2015080168A - Image-capturing device, control method therefor, and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user to recognize a subject with high accuracy in an image-capturing device for acquiring a light field.SOLUTION: An image-capturing device comprises a plurality of image-capturing lenses 101 and a plurality of image-capturing elements 102 in each of which an optical image is formed via the plurality of image-capturing elements. The optical image is formed in the plurality of image-capturing elements by the image-capturing lenses while differentiating a focal distance from each other, and an image corresponding to the optical image is obtained from each of the image-capturing elements and light-field data is obtained also. A video analysis unit 111 detects, under control of a system control unit 108, a subject in the image obtained from the image-capturing elements. When the subject is detected, a video selection unit 112 outputs an image that is largest in subject size among the images in which the subjects are detected under control of a system control unit, to a display unit via a light-field image processing unit 105.

Description

  The present invention relates to an imaging apparatus such as a digital single-lens reflex camera, a digital still camera, or a digital video camera, a control method thereof, and a control program.

  2. Description of the Related Art In recent years, imaging devices equipped with computational photography (Computational Photography) have been developed. In this computational photography, in addition to data relating to brightness and color of an image (that is, image data), other information (other data) is taken from the image sensor, and the other data is stored together with the image data. Various processes are performed by the image processing LSI.

  As other data described above, for example, there are data related to the incident angle of light from the subject and data related to depth. Then, using these other data, the focus, the depth of field by the iris, the focal distance, and the like that are uniquely determined when the subject is imaged are accurately reproduced by image processing.

  The other data is light ray information generally called a light field of a subject. Using this light ray information, an imaging device having a function that cannot be realized by a conventional imaging device can be obtained. Using this ray information, the focus position can be changed by image processing for image data obtained as a result of imaging.

  Here, an outline of obtaining image data from multiple viewpoints as an example of acquiring a light field (light ray information) will be described.

  Now, a plurality of image data are acquired by changing the focal length using a single imaging device. When these image data are combined into one image data, the focal length in the image data can be changed by image processing using the light field included in the image data.

  Furthermore, if a plurality of pieces of image data are acquired while moving the imaging apparatus horizontally or vertically, distance information indicating the distance of the subject can be obtained according to the parallax in these image data.

  The above-described method has a demerit that a light field cannot be effectively acquired for a dynamic subject because a single imaging device (that is, an imaging device) is used.

  In order to acquire images from multiple viewpoints while compensating for such disadvantages, a plurality of imaging elements or imaging devices themselves are used (see Patent Document 1). Here, a plurality of imaging elements or imaging devices are arranged in an array, and a plurality of pieces of image data obtained as a result of imaging are given simultaneity to increase the amount of information related to the light field. The light field is acquired effectively.

JP 2008-257686A

  However, the technique described in Patent Document 1 requires a plurality of imaging elements or imaging devices. As a result, power consumption inevitably increases, and the imaging device or imaging system itself increases in size.

  In addition, the imaging device that acquires the light field has a problem of what kind of image should be displayed to the user. For example, in a conventional imaging apparatus, when a person is desired to be tracked in order to record the movement of a specific person, the person is detected (a face or the like is detected) and zoomed to a position where the user can easily track the subject. It is carried out.

  On the other hand, in an imaging device that acquires a light field, the zoom position, focus position, and depth of field are changed by image processing, so what kind of image is output (that is, displayed) There is a problem of not knowing.

  As described above, in an imaging apparatus that acquires a light field, it is necessary to determine what image the user wants to obtain, that is, determine the main subject recognized by the user and output the image.

  Accordingly, an object of the present invention is to provide an imaging apparatus that allows a user to accurately recognize a subject, a control method thereof, and a control program in an imaging apparatus that acquires a light field at the time of shooting.

  In order to achieve the above object, an imaging apparatus according to the present invention includes a plurality of imaging lenses and a plurality of imaging elements on which optical images are formed through the plurality of imaging lenses, An image pickup apparatus that forms optical images on a plurality of image pickup devices at different focal lengths, obtains an image corresponding to the optical image from each of the image pickup devices, and obtains light field data. Detection means for detecting a subject in an image obtained from the element, and display control means for displaying, on the display unit, an image having the largest subject among the images in which the subject is detected when the detection means detects the subject. It is characterized by having.

  The control method according to the present invention includes a plurality of imaging lenses and a plurality of imaging elements on which optical images are formed via the plurality of imaging lenses, respectively, and the plurality of imaging elements are focal lengths to each other by the imaging lens. The optical image is formed with different values to obtain an image corresponding to the optical image from each of the imaging elements and to obtain light field data. And a display control step for displaying an image having the largest subject among the images in which the subject is detected when the subject is detected in the detection step. It is characterized by.

  A control program according to the present invention includes a plurality of imaging lenses and a plurality of imaging elements on which optical images are formed through the plurality of imaging lenses, respectively, and the plurality of imaging elements are focal lengths to each other by the imaging lens. A control program used in an imaging apparatus that forms the optical image by differentiating to obtain an image corresponding to the optical image from each of the imaging elements and obtains light field data. A detection step for detecting a subject in an image obtained from the image sensor on a computer, and when the subject is detected in the detection step, an image having the largest subject among the images in which the subject is detected is displayed on a display unit. And a display control step for displaying.

  According to the present invention, when the subject is detected in the image, the image having the largest subject among the images in which the subject is detected is displayed on the display unit, so that the user can recognize the subject satisfactorily. . As a result, the user can easily photograph the subject.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure for demonstrating the external appearance about an example of the imaging device by the 1st Embodiment of this invention, (a) is a perspective view which shows an external appearance, (b) is a figure which shows a photographic lens part from the front. FIG. 2 is a block diagram for explaining the camera shown in FIG. 1, (a) is a block diagram showing the configuration thereof, and (b) is a block diagram showing a configuration of the image processing LSI shown in (a). 3 is a flowchart for explaining display processing in the camera shown in FIG. 2. It is a flowchart for demonstrating the display process in the camera by the 2nd Embodiment of this invention.

  Hereinafter, an example of an imaging apparatus according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram for explaining the appearance of an example of an imaging apparatus according to the first embodiment of the present invention. FIG. 1A is a perspective view showing the appearance, and FIG. 1B is a view showing the photographing lens unit from the front.

  The illustrated imaging apparatus 100 is a so-called video camera (hereinafter simply referred to as a camera), and the camera 100 has an imaging lens unit 202 disposed on the front surface of a camera housing 201. For example, sixteen openings are formed in the photographing lens unit 202, and a photographing lens unit (hereinafter simply referred to as a photographing lens) 203 is disposed in each of the openings. An imaging sensor (not shown) is disposed at the rear stage of the photographic lens, and this imaging sensor has an imaging element corresponding to the photographic lens.

  As shown in the figure, a display unit 109 is disposed on the side surface of the camera housing 201 so as to be opened and closed. Further, in the illustrated example, a recording button (that is, a shutter button, a power switch, etc.) arranged on the camera housing 201 is omitted.

  Although not shown, in each of the imaging elements, a microlens array including a plurality of microlenses is disposed on a light receiving surface (also referred to as an imaging surface). In addition to the light intensity distribution on the light receiving surface of the image sensor, information related to the traveling direction of the light (that is, light field data) is obtained by the microlens array. In the illustrated camera, after imaging, an image set at an arbitrary viewpoint or an arbitrary focus is reconstructed by light field data.

  The plurality of image sensors described above are arranged in an array (that is, a two-dimensional matrix) on the same plane, and the number thereof is a number necessary for obtaining light field data. Sixteen image sensors are arranged on the same plane.

  As shown in FIG. 1B, the 16 imaging elements are divided into four groups “1” to “4”, and the focal length by the imaging lens 203 is controlled for each group. In the illustrated example, the focal length of the group “1” is “A”, the focal length of the group “2” is “B”, the focal length of the group “2” is “C”, and the focal length of the group “4” is “ D ". The focal length A is on the wide side, and the focal length D is on the tele side.

  In this way, if the focal length is changed for each group and image data is obtained at a plurality of angles of view, image data with an arbitrary focal length can be obtained by a light field image processing unit described later. Furthermore, since there are a plurality of imaging elements having the same focal length (four blades in the illustrated example), distance information indicating the distance of the subject can be obtained using the parallax of light. If the distance information is used, the depth of field in the image data can be controlled by adjusting the focus position and background blur.

  FIG. 2 is a block diagram for explaining the camera shown in FIG. FIG. 2A is a block diagram showing the configuration, and FIG. 2B is a block diagram showing the configuration of the image processing LSI shown in FIG.

  In FIG. 2A, the imaging lens unit 202 shown in FIG. 2A is shown by an optical system 101, and the optical system 101 receives light from an object (that is, an optical image) and a light receiving surface of the imaging element 102. To form an image. The optical system 101 (that is, each imaging lens) includes, for example, a focus lens for adjusting the focus position, an iris unit for controlling incident light, and a zoom lens unit for changing the focal length. .

  As described above, the plurality of image sensors 102 are arranged in an array on a plane to acquire so-called light field data, and each of the image sensors 102 outputs an electrical signal (analog signal) corresponding to the optical image. . An analog signal that is an output of the image sensor 102 is given to the image processing LSI 103.

  As the image sensor 102, for example, a CCD image sensor (Charge Coupled Device) or a CMOS image sensor is used.

  In the above example, the image sensor 102 outputs an analog signal as an image signal. However, the image sensor 102 performs an A / D conversion process and outputs a digital signal such as LVDS (Low voltage differential signaling). It may be.

  Although not shown, the image processing LSI 103 includes an analog front window unit and an image processing block. The analog front end unit converts an analog signal output from the image sensor 102 into a digital signal. Then, the image processing block performs various kinds of image processing on the digital signal to generate image data.

  Note that when the A / D conversion process is performed in the image sensor 102, the analog front end unit is omitted.

  The image processing block performs so-called pixel addition processing, noise reduction processing, gamma correction processing, knee processing, digital gain processing, and the like as image processing, and also performs scratch correction processing. If the image sensor 102 is a CMOS image sensor, the image processing block performs removal of fixed pattern noise unique to the CMOS image sensor, black level clamping processing, and the like. In addition, the image processing LSI 103 includes a storage circuit (not shown) that stores setting values necessary for image processing and the like.

  All of the image data that is the output of the image processing LSI is recorded in the signal recording unit 104, and one of the image data that is the output of the image processing LSI is given to the light field image processing unit 105 as will be described later. The signal recording unit 104 also records the above-described light field data.

  As the signal recording unit 104, for example, an HDD (Hard Disk Drive), an SD memory card, or a Compact Flash (registered trademark) memory card is used.

  Under the control of the system control unit 108, the light field image processing unit 105 performs image processing, which will be described later, on the image data in accordance with the light field data added to the image data, and outputs it as light field image data. The light feed image data is sent to the display unit 109.

  For example, the light field image processing unit 105 performs image processing such as adjustment of the focus position, zoom position, and depth of field of the optical system on the image data in accordance with the light field data.

  The focus control unit 106 adjusts the focus position by controlling a focus mechanism (that is, a focus lens) provided in the optical system 101 under the control of the system control unit 108.

  The image sensor control unit 107 controls each image sensor 102 under the control of the system control unit 107. For example, the image sensor control unit 107 determines the exposure time of the image sensor 102 under the control of the system control unit 108 and drives and controls the image sensor 102.

  The system control unit 108 controls the entire camera, for example, receives exposure information from the image processing LSI 103 and controls the exposure of the camera according to the exposure information. At this time, the system control unit 108 controls the focus control unit 106 and the image sensor control unit 107 in accordance with the exposure information to drive and control the focus lens and drive the image sensor 102 as described above.

  The display unit 109 displays an image corresponding to the image data obtained by the camera. The display unit 109 is, for example, an external monitor, a liquid crystal monitor attached to the camera, or a viewfinder.

  As shown in FIG. 2B, the image processing LSI 103 includes a plurality of video analysis units 111 and a video selection unit 112, and the output of the video analysis unit 111 is connected to the signal recording unit 104 and the video selection unit. Connected to the unit 112. The output of the video selection unit 112 is given to the light field image processing unit 105. Note that each of the video analysis units 111 corresponds to the image sensor 102.

  The image processing LSI 103 receives image signals that are outputs of the plurality of image sensors 102. All of the image data output from the video analysis unit 103 is recorded in the signal recording unit 104.

  Each of the video analysis units 111 analyzes an image indicated by an image signal (that is, a digital signal) to detect a main subject (here, for example, a face). Then, each of the video analysis units 111 sends the analysis result (that is, the main subject detection result) to the system control unit 108.

  The system control unit 108 controls the video selection unit 112 according to the analysis result, selects one of the image data obtained as a result of the image processing by the video analysis unit 111, and sends it to the light field image processing unit 105. That is, the system control unit 108 determines which image sensor 102 obtains an image corresponding to the image data on the display unit 109 according to the analysis result. Then, the system control unit 108 controls the video selection unit 112 to select image data most suitable for display, and outputs it to the light field image processing unit 105.

  FIG. 3 is a flowchart for explaining display processing in the camera shown in FIG. Note that the processing according to the flowchart shown in FIG. 3 is performed under the control of the system control unit 108.

  When the display process is started, the system control unit 108 determines whether or not the face detection mode is set by the user (step S1).

  In step S1, it is determined whether a human body detection mode for detecting a human body or a pet detection motor for detecting a pet is set in addition to the face detection mode for detecting a human face. Good. The system control unit 108 determines what kind of subject the user will shoot according to which of these modes is set, and performs focus control and exposure control.

  When the face detection mode is set (YES in step S1), the system control unit 108 controls the video analysis unit 111 to analyze the output of the imaging element having the shortest focal length and the widest angle (step). S2). Then, the system control unit 108 determines whether or not the main subject exists in the image according to the analysis result (step S3).

  When the main subject exists in the image (YES in step S3), the system control unit 108 determines the image sensor 102 having the longest focal length (telephoto side) among the image sensors 102 that output the image in which the main subject exists. Then, the system control unit 108 controls the video selection unit 112 according to the determination result, selects the image data that is the output of the telephoto image sensor 102, and supplies it to the light field image processing unit 105 (step S4). .

  As a result, the telephoto image data processed by the light field image processing unit 105 is sent to the display unit 109 for display (step S6). When an image is displayed on the display unit 109, the system control unit 108 ends the display process.

  If the face detection mode is not set (NO in step S1), the system control unit 108 controls the video selection unit 112 to output image data corresponding to the output of the image sensor having the widest angle to the light field image processing unit 105. (Step S6). Then, the system control unit 108 proceeds to the process of step S6.

  If there is no main subject in the image (NO in step S3), system control unit 108 proceeds to the process in step S6.

  As described above, in the first embodiment of the present invention, the main subject is detected by using the output of the image sensor having the widest focal length. Then, when the main subject is detected, an image obtained by the image pickup element having the focal length at which the main subject is the largest is output to the display unit. As a result, the main subject can be displayed large and clear.

  On the other hand, if the main subject cannot be detected, an image obtained by the widest-angle image sensor is displayed on the display unit. As a result, the user can display an image with the widest angle of view that can be edited later.

  Note that the mode for detecting the subject such as the face detection mode in step S1 is set using a touch panel provided in the display unit 109, for example. In step S6, an image other than the image obtained by the widest-angle image sensor may be output.

[Second Embodiment]
Next, an example of a camera according to the second embodiment of the present invention will be described. The configuration of the camera according to the second embodiment is the same as that of the camera shown in FIGS.

  FIG. 4 is a flowchart for explaining display processing in the camera according to the second embodiment of the present invention. Note that the processing according to the flowchart shown in FIG. 4 is performed under the control of the system control unit 108.

  Now, it is assumed that the display process is started and the main subject is detected in the output (that is, the image) of the image sensor 102. The system control unit 108 controls the video analysis unit 111 to analyze the image obtained by the image sensor 102 (step S11). Then, the system control unit 108 determines whether or not a main subject (for example, a face) is still detected according to the analysis result (step S12).

  When the main subject is detected (YES in step S12), the system control unit 108 performs digital zoom for enlarging the main subject by driving the image sensor 102 by the image sensor control unit 107 (step S13). Then, the system control unit 108 controls the video selection unit 112 to give the digital zoomed image to the light field image processing unit 105.

  As a result, the image data processed by the light field image processing unit 105 is sent to the display unit 109 for display (step S14). When an image is displayed on the display unit 109, the system control unit 108 ends the display process.

  For example, if the main subject moves and no main subject exists in the image obtained by the imaging element 102 having the current focal length, that is, the main subject is not detected (NO in step S12), the system control unit 108 determines whether the image sensor is the widest-angle image sensor (step S15).

  If the image sensor from which the main subject has been captured is not the widest-angle image sensor 102 (NO in step S15), the system control unit 108 can obtain the next image sensor 102 with the shortest focal length. The selected image is selected (step S16). Then, the system control unit 108 returns to the process of step S11 to control the video analysis unit 101, and then analyzes an image obtained by the imaging element 102 having a short focal length.

  If the image sensor 102 from which the image in which the main subject has been present has been obtained is the widest-angle image sensor 102 (YES in step S15), the system control unit 108 determines that there is no image in which the main subject exists. Then, the video selection unit 112 is controlled to continue outputting the image obtained by the widest-angle image sensor 102 (step S17). Then, the system control unit 108 proceeds to the process of step S14.

  Thus, in the second embodiment of the present invention, the image is switched from the telephoto image sensor 102 to the wide angle image sensor 102 in accordance with the detection result of the main subject. When the main subject is detected, the main subject is enlarged and displayed. As a result, the user can observe the enlarged main subject, so that the main subject can be easily captured.

  As is clear from the above description, in the example shown in FIG. 2, the system control unit 108 and the video analysis unit 111 function as detection means, and the system control unit 108, the video selection unit 112, and the light field image processing unit 105 It functions as a display control means. The video analysis unit 111 and the system control unit 108 function as a determination unit.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the imaging apparatus. Further, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the imaging apparatus. The control program is recorded on a computer-readable recording medium, for example.

  Each of the above control method and control program has at least a detection step and a display control step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

DESCRIPTION OF SYMBOLS 101 Optical system 102 Image pick-up element 103 Image processing LSI
104 signal recording unit 105 light field image processing unit 106 focus control unit 107 image sensor control unit 108 system control unit 109 display unit 111 video analysis unit 112 video selection unit

Claims (7)

A plurality of image pickup lenses and a plurality of image pickup elements on which optical images are formed through the plurality of image pickup lenses, respectively, and the plurality of image pickup elements are made to have different focal lengths by the image pickup lens, and the optical image An imaging device that obtains light field data while obtaining an image corresponding to the optical image from each of the imaging elements,
Detecting means for detecting a subject in an image obtained from the image sensor;
Display control means for displaying, on the display unit, an image having the largest subject among the images in which the subject is detected when the subject is detected by the detection means;
An imaging device comprising: The detection means detects a subject in an image obtained at the widest angle with the shortest focal length among images obtained from the image sensor,
When the subject is detected in the image obtained at the widest angle, the display control means displays on the display unit an image obtained with a telephoto having the longest focal length among images obtained from the image sensor. The imaging apparatus according to claim 1.   The imaging apparatus according to claim 2, wherein if the subject is not detected in the image obtained at the widest angle, the display control unit displays the image obtained at the widest angle on the display unit. . Determination means for determining whether the subject is still present in the image in a state where the subject is detected in the image by the detection unit;
When the determination unit determines that the subject does not exist in the image, the display control unit displays the widest-angle image on the display unit when the image is an image obtained at the widest angle. The imaging apparatus according to claim 1.   The imaging apparatus according to claim 1, wherein when the determination unit determines that the subject exists in the image, the display control unit displays an enlarged image of the subject on the display unit. . A plurality of image pickup lenses and a plurality of image pickup elements on which optical images are formed through the plurality of image pickup lenses, respectively, and the plurality of image pickup elements are made to have different focal lengths by the image pickup lens, and the optical image An imaging device control method for obtaining light field data while obtaining an image corresponding to the optical image from each of the imaging elements,
A detection step of detecting a subject in an image obtained from the image sensor;
A display control step of displaying, on the display unit, an image having the largest subject among the images in which the subject is detected when the subject is detected in the detection step;
A control method characterized by comprising: A plurality of image pickup lenses and a plurality of image pickup elements on which optical images are formed through the plurality of image pickup lenses, respectively, and the plurality of image pickup elements are made to have different focal lengths by the image pickup lens, and the optical image A control program used in an imaging device that obtains light field data while obtaining an image corresponding to the optical image from each of the imaging elements,
In the computer provided in the imaging device,
A detection step of detecting a subject in an image obtained from the image sensor;
A display control step of displaying, on the display unit, an image having the largest subject among the images in which the subject is detected when the subject is detected in the detection step;
A control program characterized by causing

Images