JP6202982B2 - Image processing apparatus, control method thereof, and control program - Google Patents

Description

The present invention relates to an image processing apparatus, its control method, and a control program, in particular, relates a digital single-lens reflex camera, the image processing equipment such as a digital still camera or digital video camera.

  2. Description of the Related Art In recent years, imaging devices equipped with computational photography (Computational Photography) have been developed. In this computational photography, in addition to data relating to brightness and color of an image (that is, image data), other information (other data) is taken from the image sensor, and the other data is stored together with the image data. Various processes are performed by the image processing LSI.

  As other data described above, for example, there are data related to the incident angle of light from the subject and data related to depth. Then, using these other data, the focus, the depth of field by the iris, the focal distance, and the like that are uniquely determined when the subject is imaged are accurately reproduced by image processing.

  The other data is light ray information generally called a light field of a subject. Using this light ray information, an imaging device having a function that cannot be realized by a conventional imaging device can be obtained. Using this ray information, the focus position can be changed by image processing for image data obtained as a result of imaging.

  Here, an outline of obtaining image data from multiple viewpoints as an example of acquiring a light field (light ray information) will be described.

  Now, a plurality of image data are acquired by changing the focal length using a single imaging device. When these image data are combined into one image data, the focal length in the image data can be changed by image processing using the light field included in the image data.

  Furthermore, if a plurality of pieces of image data are acquired while moving the imaging apparatus horizontally or vertically, distance information indicating the distance of the subject can be obtained according to the parallax in these image data.

  The above-described method has a demerit that a light field cannot be effectively acquired for a dynamic subject because a single imaging device (that is, an imaging device) is used.

  In order to acquire images from multiple viewpoints while compensating for such disadvantages, a plurality of imaging elements or imaging devices themselves are used (see Patent Document 1). Here, a plurality of imaging elements or imaging devices are arranged in an array, and a plurality of pieces of image data obtained as a result of imaging are given simultaneity to increase the amount of information related to the light field. The light field is acquired effectively.

JP 2008-257686A

  However, the technique described in Patent Document 1 requires a plurality of imaging elements or imaging devices. As a result, power consumption inevitably increases, and the imaging device or imaging system itself increases in size.

  In addition, the imaging device that acquires the light field has a problem of what kind of image should be displayed to the user. For example, in a conventional imaging apparatus, when a person is desired to be tracked in order to record the movement of a specific person, the person is detected (a face or the like is detected) and zoomed to a position where the user can easily track the subject. It is carried out.

  On the other hand, in an imaging device that acquires a light field, the zoom position, focus position, and depth of field are changed by image processing, so what kind of image is output (that is, displayed) There is a problem of not knowing.

  As described above, in an imaging apparatus that acquires a light field, it is necessary to determine what image the user wants to obtain, that is, determine the main subject recognized by the user and output the image.

Accordingly, it is an object of the present invention to provide an image processing apparatus, a control method thereof, and a control program that allow a user to accurately recognize a subject when acquiring a light field during shooting.

In order to achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus that processes a plurality of images obtained from a plurality of imaging elements respectively corresponding to a plurality of imaging lenses including optical systems having different focal lengths. Te, detection means for detecting a specific subject from the plurality of images, if the specific subject from the plurality of images by the detecting means is detected, the most focal length of the image that the specific subject is detected Display control means for displaying on the display unit an image obtained with a long maximum telephoto .

The control method according to the present invention is a control method for an image processing apparatus that processes a plurality of images obtained from a plurality of imaging elements respectively corresponding to a plurality of imaging lenses including optical systems having different focal lengths. A detection step of detecting a specific subject from the plurality of images, and when the specific subject is detected from the plurality of images in the detection step, the image obtained with the longest focal length among the images in which the specific subject is detected is obtained. And a display control step for displaying the received image on the display unit.

A control program according to the present invention is a control program used in an image processing apparatus that processes a plurality of images obtained from a plurality of imaging elements respectively corresponding to a plurality of imaging lenses including optical systems having different focal lengths. a computer processing apparatus comprising: a detection step of detecting a specific subject from the plurality of images, if the specific subject from the plurality of images by the detecting step is detected, the specific subject is detected image And a display control step for displaying an image obtained at the maximum telephoto distance with the longest focal length on the display unit.

  According to the present invention, when the subject is detected in the image, the image having the largest subject among the images in which the subject is detected is displayed on the display unit, so that the user can recognize the subject satisfactorily. . As a result, the user can easily photograph the subject.

Are diagrams for about the imaging equipment, which is an example illustrating the appearance of an image processing apparatus according to a first embodiment of the present invention, (a) is a perspective view showing an appearance, (b) the photographic lens unit It is a figure shown from the front. FIG. 2 is a block diagram for explaining the camera shown in FIG. 1, (a) is a block diagram showing the configuration thereof, and (b) is a block diagram showing a configuration of the image processing LSI shown in (a). 3 is a flowchart for explaining display processing in the camera shown in FIG. 2. It is a flowchart for demonstrating the display process in the camera which concerns on the 2nd Embodiment of this invention.

Hereinafter, an example of an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram for explaining the external appearance with the imaging equipment, which is an example of an image processing apparatus according to a first embodiment of the present invention. FIG. 1A is a perspective view showing the appearance, and FIG. 1B is a view showing the photographing lens unit from the front.

  The illustrated imaging apparatus 100 is a so-called video camera (hereinafter simply referred to as a camera), and the camera 100 has an imaging lens unit 202 disposed on the front surface of a camera housing 201. For example, sixteen openings are formed in the photographing lens unit 202, and a photographing lens unit (hereinafter simply referred to as a photographing lens) 203 is disposed in each of the openings. An imaging sensor (not shown) is disposed at the rear stage of the photographic lens, and this imaging sensor has an imaging element corresponding to the photographic lens.

  As shown in the figure, a display unit 109 is disposed on the side surface of the camera housing 201 so as to be opened and closed. Further, in the illustrated example, a recording button (that is, a shutter button, a power switch, etc.) arranged on the camera housing 201 is omitted.

  Although not shown, in each of the imaging elements, a microlens array including a plurality of microlenses is disposed on a light receiving surface (also referred to as an imaging surface). In addition to the light intensity distribution on the light receiving surface of the image sensor, information related to the light traveling direction (that is, light field data) is obtained by the microlens array. In the illustrated camera, after imaging, an image set at an arbitrary viewpoint or an arbitrary focus is reconstructed by light field data.

  The plurality of image sensors described above are arranged in an array (that is, a two-dimensional matrix) on the same plane, and the number thereof is a number necessary for obtaining light field data. Sixteen image sensors are arranged on the same plane.

  As shown in FIG. 1B, the 16 imaging elements are divided into four groups “1” to “4”, and the focal length by the imaging lens 203 is controlled for each group. In the illustrated example, the focal length of the group “1” is “A”, the focal length of the group “2” is “B”, the focal length of the group “2” is “C”, and the focal length of the group “4” is “ D ". The focal length A is on the wide side, and the focal length D is on the tele side.

  In this way, if the focal length is changed for each group and image data is obtained at a plurality of angles of view, image data with an arbitrary focal length can be obtained by a light field image processing unit described later. Furthermore, since there are a plurality of imaging elements having the same focal length (four blades in the illustrated example), distance information indicating the distance of the subject can be obtained using the parallax of light. If the distance information is used, the depth of field in the image data can be controlled by adjusting the focus position and background blur.

  FIG. 2 is a block diagram for explaining the camera shown in FIG. FIG. 2A is a block diagram showing the configuration, and FIG. 2B is a block diagram showing the configuration of the image processing LSI shown in FIG.

  In FIG. 2A, the imaging lens unit 202 shown in FIG. 2A is shown by an optical system 101, and the optical system 101 receives light from an object (that is, an optical image) and a light receiving surface of the imaging element 102. To form an image. The optical system 101 (that is, each imaging lens) includes, for example, a focus lens for adjusting the focus position, an iris unit for controlling incident light, and a zoom lens unit for changing the focal length. .

  As described above, the plurality of image sensors 102 are arranged in an array on a plane to acquire so-called light field data, and each of the image sensors 102 outputs an electrical signal (analog signal) corresponding to the optical image. . An analog signal that is an output of the image sensor 102 is given to the image processing LSI 103.

  As the image sensor 102, for example, a CCD image sensor (Charge Coupled Device) or a CMOS image sensor is used.

  In the above example, the image sensor 102 outputs an analog signal as an image signal. However, the image sensor 102 performs an A / D conversion process and outputs a digital signal such as LVDS (Low voltage differential signaling). It may be.

  Although not shown, the image processing LSI 103 includes an analog front window unit and an image processing block. The analog front end unit converts an analog signal output from the image sensor 102 into a digital signal. Then, the image processing block performs various kinds of image processing on the digital signal to generate image data.

  Note that when the A / D conversion process is performed in the image sensor 102, the analog front end unit is omitted.

  The image processing block performs so-called pixel addition processing, noise reduction processing, gamma correction processing, knee processing, digital gain processing, and the like as image processing, and also performs scratch correction processing. If the image sensor 102 is a CMOS image sensor, the image processing block performs removal of fixed pattern noise unique to the CMOS image sensor, black level clamping processing, and the like. In addition, the image processing LSI 103 includes a storage circuit (not shown) that stores setting values necessary for image processing and the like.

  All of the image data that is the output of the image processing LSI is recorded in the signal recording unit 104, and one of the image data that is the output of the image processing LSI is given to the light field image processing unit 105 as will be described later. The signal recording unit 104 also records the above-described light field data.

  As the signal recording unit 104, for example, an HDD (Hard Disk Drive), an SD memory card, or a Compact Flash (registered trademark) memory card is used.

  Under the control of the system control unit 108, the light field image processing unit 105 performs image processing, which will be described later, on the image data in accordance with the light field data added to the image data, and outputs it as light field image data. The light feed image data is sent to the display unit 109.

  For example, the light field image processing unit 105 performs image processing such as adjustment of the focus position, zoom position, and depth of field of the optical system on the image data in accordance with the light field data.

  The focus control unit 106 adjusts the focus position by controlling a focus mechanism (that is, a focus lens) provided in the optical system 101 under the control of the system control unit 108.

  The image sensor control unit 107 controls each image sensor 102 under the control of the system control unit 107. For example, the image sensor control unit 107 determines the exposure time of the image sensor 102 under the control of the system control unit 108 and drives and controls the image sensor 102.

  The system control unit 108 controls the entire camera, for example, receives exposure information from the image processing LSI 103 and controls the exposure of the camera according to the exposure information. At this time, the system control unit 108 controls the focus control unit 106 and the image sensor control unit 107 in accordance with the exposure information to drive and control the focus lens and drive the image sensor 102 as described above.

  The display unit 109 displays an image corresponding to the image data obtained by the camera. The display unit 109 is, for example, an external monitor, a liquid crystal monitor attached to the camera, or a viewfinder.

  As shown in FIG. 2B, the image processing LSI 103 includes a plurality of video analysis units 111 and a video selection unit 112, and the output of the video analysis unit 111 is connected to the signal recording unit 104 and the video selection unit. Connected to the unit 112. The output of the video selection unit 112 is given to the light field image processing unit 105. Note that each of the video analysis units 111 corresponds to the image sensor 102.

  The image processing LSI 103 receives image signals that are outputs of the plurality of image sensors 102. All of the image data output from the video analysis unit 103 is recorded in the signal recording unit 104.

  Each of the video analysis units 111 analyzes an image indicated by an image signal (that is, a digital signal) to detect a main subject (here, for example, a face). Then, each of the video analysis units 111 sends the analysis result (that is, the main subject detection result) to the system control unit 108.

  The system control unit 108 controls the video selection unit 112 according to the analysis result, selects one of the image data obtained as a result of the image processing by the video analysis unit 111, and sends it to the light field image processing unit 105. That is, the system control unit 108 determines which image sensor 102 obtains an image corresponding to the image data on the display unit 109 according to the analysis result. Then, the system control unit 108 controls the video selection unit 112 to select image data most suitable for display, and outputs it to the light field image processing unit 105.

  FIG. 3 is a flowchart for explaining display processing in the camera shown in FIG. Note that the processing according to the flowchart shown in FIG. 3 is performed under the control of the system control unit 108.

  When the display process is started, the system control unit 108 determines whether or not the face detection mode is set by the user (step S1).

  In step S1, it is determined whether a human body detection mode for detecting a human body or a pet detection motor for detecting a pet is set in addition to the face detection mode for detecting a human face. Good. The system control unit 108 determines what kind of subject the user will shoot according to which of these modes is set, and performs focus control and exposure control.

  When the face detection mode is set (YES in step S1), the system control unit 108 controls the video analysis unit 111 to analyze the output of the imaging element having the shortest focal length and the widest angle (step). S2). Then, the system control unit 108 determines whether or not the main subject exists in the image according to the analysis result (step S3).

  When the main subject exists in the image (YES in step S3), the system control unit 108 determines the image sensor 102 having the longest focal length (telephoto side) among the image sensors 102 that output the image in which the main subject exists. Then, the system control unit 108 controls the video selection unit 112 according to the determination result, selects the image data that is the output of the telephoto image sensor 102, and supplies it to the light field image processing unit 105 (step S4). .

  As a result, the telephoto image data processed by the light field image processing unit 105 is sent to the display unit 109 for display (step S6). When an image is displayed on the display unit 109, the system control unit 108 ends the display process.

  If the face detection mode is not set (NO in step S1), the system control unit 108 controls the video selection unit 112 to output image data corresponding to the output of the image sensor having the widest angle to the light field image processing unit 105. (Step S6). Then, the system control unit 108 proceeds to the process of step S6.

  If there is no main subject in the image (NO in step S3), system control unit 108 proceeds to the process in step S6.

  As described above, in the first embodiment of the present invention, the main subject is detected by using the output of the image sensor having the widest focal length. Then, when the main subject is detected, an image obtained by the image pickup element having the focal length at which the main subject is the largest is output to the display unit. As a result, the main subject can be displayed large and clear.

  On the other hand, if the main subject cannot be detected, an image obtained by the widest-angle image sensor is displayed on the display unit. As a result, the user can display an image with the widest angle of view that can be edited later.

  Note that the mode for detecting the subject such as the face detection mode in step S1 is set using a touch panel provided in the display unit 109, for example. In step S6, an image other than the image obtained by the widest-angle image sensor may be output.

[Second Embodiment]
The following describes one case of a camera according to a second embodiment of the present invention. The configuration of the camera according to the second embodiment is the same as that of the camera shown in FIGS.

  FIG. 4 is a flowchart for explaining display processing in the camera according to the second embodiment of the present invention. Note that the processing according to the flowchart shown in FIG. 4 is performed under the control of the system control unit 108.

  Now, it is assumed that the display process is started and the main subject is detected in the output (that is, the image) of the image sensor 102. The system control unit 108 controls the video analysis unit 111 to analyze the image obtained by the image sensor 102 (step S11). Then, the system control unit 108 determines whether or not a main subject (for example, a face) is still detected according to the analysis result (step S12).

  When the main subject is detected (YES in step S12), the system control unit 108 performs digital zoom for enlarging the main subject by driving the image sensor 102 by the image sensor control unit 107 (step S13). Then, the system control unit 108 controls the video selection unit 112 to give the digital zoomed image to the light field image processing unit 105.

  As a result, the image data processed by the light field image processing unit 105 is sent to the display unit 109 for display (step S14). When an image is displayed on the display unit 109, the system control unit 108 ends the display process.

  For example, if the main subject moves and no main subject exists in the image obtained by the imaging device 102 having the current focal length, that is, the main subject is not detected (NO in step S12), the system control unit 108 determines whether the image sensor is the widest-angle image sensor (step S15).

  If the image sensor from which the main subject has been captured is not the widest-angle image sensor 102 (NO in step S15), the system control unit 108 can obtain the next image sensor 102 with the shortest focal length. The selected image is selected (step S16). Then, the system control unit 108 returns to the process of step S11 to control the video analysis unit 101, and then analyzes an image obtained by the imaging element 102 having a short focal length.

  If the image sensor 102 from which the image in which the main subject has been present has been obtained is the widest-angle image sensor 102 (YES in step S15), the system control unit 108 determines that there is no image in which the main subject exists. Then, the video selection unit 112 is controlled to continue outputting the image obtained by the widest-angle image sensor 102 (step S17). Then, the system control unit 108 proceeds to the process of step S14.

  Thus, in the second embodiment of the present invention, the image is switched from the telephoto image sensor 102 to the wide angle image sensor 102 in accordance with the detection result of the main subject. When the main subject is detected, the main subject is enlarged and displayed. As a result, the user can observe the enlarged main subject, so that the main subject can be easily captured.

  As is clear from the above description, in the example shown in FIG. 2, the system control unit 108 and the video analysis unit 111 function as detection means, and the system control unit 108, the video selection unit 112, and the light field image processing unit 105 It functions as a display control means. The video analysis unit 111 and the system control unit 108 function as a determination unit.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the image processing apparatus. In addition, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the image processing apparatus. The control program is recorded on a computer-readable recording medium, for example.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

DESCRIPTION OF SYMBOLS 101 Optical system 102 Image pick-up element 103 Image processing LSI
104 signal recording unit 105 light field image processing unit 106 focus control unit 107 image sensor control unit 108 system control unit 109 display unit 111 video analysis unit 112 video selection unit

Claims (8)

An image processing apparatus that processes a plurality of images obtained from a plurality of imaging elements respectively corresponding to a plurality of imaging lenses including optical systems having different focal lengths ,
Detecting means for detecting a specific subject from the plurality of images;
When the specific object is detected from the plurality of images by the detection means, display control for displaying on the display unit an image obtained at the maximum telephoto with the longest focal length among the images in which the specific object is detected Means,
An image processing apparatus comprising: The detecting device, the image processing according to claim 1, characterized in that detects the specific subject in the most short focal length image obtained at the top angle of the images obtained from said plurality of imaging elements apparatus. 3. The image processing according to claim 1, wherein if the specific subject is not detected from the plurality of images , the display control unit displays the image obtained at the widest angle on the display unit. apparatus. Determining means for determining whether or not the specific subject still exists in the image in a state where the specific subject is detected in the image by the detecting unit;
When the determination unit determines that the specific subject does not exist in the image, the display control unit displays the image at the widest angle on the display unit when the image is an image obtained at the widest angle. The image processing apparatus according to claim 3 , wherein the image processing apparatus displays the image. 5. The display control unit according to claim 4 , wherein when the determination unit determines that the specific subject exists in the image, the display control unit displays an enlarged image of the specific subject on the display unit. The image processing apparatus described. The plurality of imaging lenses;
The plurality of image sensors;
The image processing apparatus according to claim 1, further comprising: A control method for an image processing apparatus that processes a plurality of images obtained from a plurality of imaging elements respectively corresponding to a plurality of imaging lenses including optical systems having different focal lengths ,
A detection step of detecting a specific subject from the plurality of images;
When the specific subject is detected from the plurality of images in the detection step, display control for displaying on the display unit an image obtained at the maximum telephoto with the longest focal length among images in which the specific subject is detected Steps,
A control method characterized by comprising: A control program used in an image processing apparatus for processing a plurality of images obtained from a plurality of imaging elements respectively corresponding to a plurality of imaging lenses including optical systems having different focal lengths ,
In the computer provided in the image processing apparatus,
A detection step of detecting a specific subject from the plurality of images;
When the specific subject is detected from the plurality of images in the detection step, display control for displaying on the display unit an image obtained at the maximum telephoto with the longest focal length among images in which the specific subject is detected Steps,
A control program characterized by causing

Images