JP2016099322A - Imaging device, control method of imaging device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire, in an image including a subject close to an imaging device and a subject distant from the imaging device, accurate distance information for each of the subjects (areas) in a device that can acquire an image with parallax.SOLUTION: An imaging device includes: imaging means for photographing optical images of subjects formed in an imaging optical system to create a plurality of image signals with parallax; distance distribution creation means for acquiring distance information on the subjects from the plurality of image signals to create a distance distribution of the subjects on the basis of the distance information; and control means for controlling the imaging means and distance distribution creation means to create a distance distribution of images including the subjects. The control means sets the depth of field in imaging for creating the plurality of image signals with parallax to be as shallow as to allow acquisition of the distance information on the subject formed in the imaging optical system from the plurality of image signals.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus capable of capturing a parallax image, and more particularly to an imaging apparatus capable of acquiring subject distance information from a parallax image.

  2. Description of the Related Art Conventionally, there is an apparatus that generates print data for a printing apparatus that prints a three-dimensional image having unevenness such as a 3D relief. For example, Patent Document 1 acquires distance information for each of a plurality of regions of an image, generates thickness information at the time of printing corresponding to the image information from the distance information for each region, and records the image information and the thickness information in association with each other. doing.

  For example, if you want to print a 3D relief image that clearly shows from a near subject to a far subject like pan focus, both near and far subjects can be used to express the unevenness of the clearly visible part. It is necessary to acquire distance information accurately. However, in Patent Document 1, it is possible to generate print data that understands the relationship between the subject and the background, but it is difficult to generate print data that understands the unevenness of the subject.

  On the other hand, in Patent Document 2, an omnifocal image focused on a range wider than the focused range of the image is generated from an image focused on a specific depth, and a blur parameter separately obtained from the omnifocal image is further generated. Thus, a blurred image group is generated, and distance information is obtained by comparison for each image region. A blurred image having a different blurred image group is selected for each image area, and a distance distribution is created based on distance information acquired from the selected blurred image.

Japanese Patent No. 4888223 International Publication No. 11/158515

  However, in the process disclosed in Patent Document 2 described above, a plurality of images of a specific depth are captured, an omnifocal image is generated therefrom, and a plurality of blurred images are generated from the omnifocal image using blur parameters. The distance is calculated using them. Therefore, the processing load is heavy because the distance map is finally synthesized. Further, there is a problem that correct distance information cannot be obtained when the number of blurred images is small and the depth variation is small.

  Here, in recent years, an imaging device capable of acquiring an image with parallax has been developed, and a distance from the acquired parallax to a subject can be calculated. However, in this method, when pan-focus shooting is performed, distance information cannot be obtained because a near object and a distant object are in an allowable circle of confusion and there is almost no parallax. In addition, since the reliability of the parallax obtained is low in an image area that is largely blurred and there is a distance measurement deviation due to aberration, it is difficult to obtain highly accurate distance information other than the focal plane near the in-focus position.

  In view of the above problems, the present invention makes it possible to acquire accurate distance information for each subject (region) of an image including a near subject and a far subject in an imaging apparatus capable of obtaining an image with parallax. With the goal.

  In order to achieve the above object, an imaging apparatus according to the present invention includes an imaging unit that captures an optical image of a subject formed by an imaging optical system and generates a plurality of image signals having parallax, and a plurality of image signals. A distance distribution generating means for acquiring distance information of the subject from the information, and generating a distance distribution of the subject based on the distance information, and a control means for controlling the imaging means and the distance distribution generating means to generate a distance distribution of the image including the subject And the control means sets the depth of field in imaging for generating a plurality of image signals having parallax to a shallow depth from which the distance information of the subject imaged by the imaging optical system can be acquired from the plurality of image signals. .

  According to the present invention, in an imaging device capable of acquiring a parallax image, in addition to a captured image, a parallax image having a shallower depth of field is captured, and distance information generated from the obtained parallax image is determined as the distance of the captured image. It can be information. This makes it possible to acquire accurate distance information for each subject (region) in an image including a near subject and a far subject.

1 is a block diagram illustrating a configuration example of an imaging apparatus according to a first embodiment of the present invention. It is a figure which shows the structure of the pixel of the image pick-up element which the imaging device concerning 1st Example of this invention has. It is a figure which shows the flowchart of operation | movement of the imaging device concerning 1st Example of this invention. It is a figure for demonstrating operation | movement of the imaging device concerning 1st Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus 100 such as a digital camera according to the first embodiment of the present invention. In FIG. 1, an imaging apparatus 100 includes an imaging optical system 101, an imaging element 102, an A / D converter 103, an image processing unit 104, a temporary storage memory 105, a subject detection unit 106, a distance distribution generation unit 107, and a distance distribution synthesis unit 108. The CPU 109 is configured. The imaging optical system 101 includes a zoom lens, a focusing lens, and a diaphragm. The image sensor 102 photoelectrically converts the optical image of the subject imaged by the imaging optical system 101 to generate and output an image signal (electric signal).

  The A / D conversion unit 103 converts the image signal output from the image sensor 102 from an analog signal to a digital signal. The image processing unit 104 performs various types of image processing on the image data converted into a digital signal by the A / D conversion unit 103. The temporary storage memory 105 stores image data processed by the image processing unit 104 and various data.

  The subject detection unit 106 reads the image processed by the image processing unit 104 from the temporary storage memory 105, and detects whether or not a subject exists in the image. The distance distribution generation unit 107 generates a distance distribution for the subject area detected by the subject detection unit. The distance distribution combining unit 108 combines the distance distributions of the plurality of subject areas generated by the distance distribution generation unit 107, and generates a distance distribution indicating the distances of the plurality of subject areas.

  The CPU 109 executes a program stored in a memory (not shown), thereby controlling each unit of the above-described imaging device 100 to realize various processing operations.

  The configuration of the imaging apparatus illustrated in FIG. 1 is an example, and the configuration is not limited thereto, and any configuration that can execute the operation of the present embodiment described below is acceptable.

  The following is an example of a configuration in which a captured image and an image with a shallow depth of field are captured, and distance information generated from a parallax image obtained by capturing an image with a shallow depth of field is stored as the distance information of the captured image. This embodiment will be described. First, the configuration of the imaging element 102 in the imaging apparatus 100 according to the present embodiment will be described.

  FIG. 2 is a diagram showing the structure of the image sensor 102, and is a cross-sectional view when one pixel is viewed from the side. In the figure, light from a subject is efficiently condensed on a photoelectric conversion unit by a microlens 201, and then only a desired wavelength region is selectively passed through a color filter 202. Reference numeral 203 denotes a semiconductor wiring layer. Light that has passed through the color filter 202 is received by a photodiode 204, which is a photoelectric conversion unit, and converted into an electrical signal by photoelectric conversion. A normal imaging device has one photodiode for one microlens 201, but in this embodiment, it has a structure in which two photodiodes 204 are arranged. Thus, it becomes possible to acquire a pupil division | segmentation image by setting it as the structure where two photodiodes 204 share one micro lens 201. FIG.

  In the present embodiment, still image shooting is performed with the imaging apparatus 100 illustrated in FIGS. 1 and 2, a subject is detected from the acquired still image, shooting is performed for each subject and distance distribution is generated according to the detection result, The distance distribution for each is synthesized. This processing operation will be described with reference to the flowchart shown in FIG.

  In step S301, the CPU 109 controls the imaging apparatus 100 to perform pan-focus imaging and acquire a captured image. Here, the pan-focus image is an image with a deep depth of field that appears to be focused on both a subject close to and a distant subject from the imaging device. In addition, as illustrated in FIG. 2, the imaging apparatus 100 includes two photodiodes 204 for one pixel, and can independently acquire pupil division image data. Strictly speaking, the two divided pupil images to be acquired are images having parallax, but it is considered that there is no parallax because the divided pupil images appear to be exactly overlapped with each other in a focused portion. An image 410 illustrated in FIG. 4 schematically illustrates an example of the captured image (pan focus image) captured in step S301.

  In step S <b> 302, the subject detection unit 106 performs subject detection on the captured image 410. The subject detection unit 106 may be configured to detect an image pattern such as a person, a face, a pet, a landmark, or the like from an image by template matching. You may comprise so that an area | region with a large quantity may be detected. Since such a detection method is a well-known technique, it is omitted here. In the case of the captured image 410 of FIG. 4, the subject detection unit 106 detects the building 402 as a subject at a position far from the person 401 and the person 401.

  In step S303, the CPU 109 determines the priority order in which order the distance distribution is generated for the subject detected in step S302. This determination process may be performed by the distance distribution generation unit 107. The priority order is determined based on, for example, the position and size of the subject in the image derived by the image processing unit 104, and the type of subject (such as increasing the priority of a person or a moving subject). In the example of the captured image 410 in FIG. 4, it is assumed that the person 401 has a higher priority than the building 402.

  In step S304, the CPU 109 controls the imaging apparatus 100 to focus and shoot the subject that has been given the highest priority in step S303, thereby obtaining a pupil division image. Here, in step S304, shooting is performed with a shallow depth of field as compared to pan-focus shooting in step S301. Since the person 401 is the top-priority subject in the photographed image 410, first, the person 401 is focused and photographed to obtain the photographed image 411.

  In step S <b> 305, the distance distribution generation unit 107 generates a distance distribution related to the subject imaged in step S <b> 304. Here, a method in which the distance distribution generation unit 107 generates a distance distribution related to the subject will be described.

  The two pupil-divided images acquired by the shooting in step S304 have a parallax even near the focal point due to the shallow depth of field, and the distance of the image region close to the in-focus position is calculated using this parallax. I can do it. Thus, the distance distribution of the focused subject is generated from the pupil divided image. In principle, the distance can be calculated for a portion where there is parallax in the two pupil-divided images, so that it is possible to generate a distance distribution not only for the focused subject but also for the entire image. However, since the reliability of the parallax obtained is low in the portion that is greatly blurred and there is a distance measurement deviation due to the aberration of the imaging optical system 101, it is possible to obtain a highly accurate distance distribution other than the focal plane near the in-focus position. Have difficulty. For this reason, the distance distribution in the region of the focused subject is generated from the captured image 411 in this embodiment instead of the distance distribution of the entire image. As a result, in step S305, a distance distribution 421 is generated only for the area of the person 401 in the captured image 410 of FIG.

  Note that in step S304, the purpose is to obtain a pupil-divided image for calculating the distance distribution, and it is not necessary to store the captured image itself captured in step S304 as image data, and the distance distribution generation in step S305 is completed. If necessary, the temporary storage memory 105 may be deleted.

  In step S306, the CPU 109 determines whether there is a subject whose distance distribution has not been generated. This determination process may also be performed by the distance distribution generation unit 107. If there is a subject whose distance distribution has not been generated, the process proceeds to step S304. If the distance distribution has been generated for all detected subjects, the process proceeds to step S307. In the example of the captured image 410 in FIG. 4, the generation of the distance distribution of the person 401 is completed, and the generation of the distance distribution of the building 402 is not yet performed, so the process returns to step S304 and the parallax focused on the building 402 as in the case of the person 401. An image is taken and a distance distribution 422 for the building 402 is generated. When both the distance distributions of the person 401 and the building 402 are generated, the process proceeds to step S307.

  In step S307, the distance distribution synthesis unit 108 synthesizes the distance distribution for each subject generated in steps S304 to S305, and generates a distance distribution including distance information of all subjects. In the example of FIG. 4, the distance distribution 421 and the distance distribution 422 are combined by the distance distribution combining unit 108 to generate the distance distribution 423. The distance distribution 423 includes distance information around the subjects of both the person 401 and the building 402.

  In step S308, the CPU 109 stores the distance distribution generated in step S307 as the distance distribution of the pan focus image captured in step S301. In the example of the captured image 410 in FIG. 4, a distance distribution 423 including distance information of the person 401 and the building 402 that are subjects in the image is stored in association with the image data of the captured image 410.

  In the above-described embodiment, the configuration of the image sensor 102 for acquiring the distance distribution has been described as a configuration for outputting image signals of two images with parallax. However, as long as the configuration is such that a subject image formed through different pupil regions is captured and a plurality of image signals are output, the configuration is not limited to the configuration of FIG. The structure arranged under the micro lens 201 may be sufficient.

  Further, in the example of FIG. 4 of the present embodiment, distance information of an area where no subject is detected is not acquired in the distance distribution 423. However, in the case where it is desired to print with unevenness only on the subject portion as in the case of 3D relief, the distance information of the region where the subject is not detected is ignored, and such a configuration may be used.

  On the other hand, when the subject is detected, the captured image 410 may be divided into rectangular regions, and the distance distribution may be acquired in each region. For example, from the brightness, color, spatial frequency, difference between adjacent areas, and the size of areas with small differences added together, it is determined whether the conditions set in advance for each subject such as the sky, flowers, buildings, etc. Then, all the areas of the captured image may be classified so as to be included in any one of the subject areas.

  As described above, according to the present invention, in an imaging device capable of acquiring an image with parallax, an image having a shallower depth of field than the captured image is captured, and distance information generated from the obtained parallax image is captured. Distance information. This makes it possible to acquire accurate distance information for each subject (area) in an image including a near subject and a far subject, and printing for a printing apparatus that prints a three-dimensional image having unevenness like a 3D relief. Data can be generated efficiently and accurately.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

Claims (10)

Imaging means for capturing an optical image of a subject imaged by the imaging optical system and generating a plurality of image signals having parallax;
Distance distribution generating means for acquiring distance information of the subject from the plurality of image signals and generating a distance distribution of the subject based on the distance information;
Control means for controlling the imaging means and the distance distribution generating means to generate a distance distribution of an image including the subject;
With
The control means sets a depth of field in imaging for generating a plurality of image signals having the parallax as shallow as possible to obtain distance information of a subject imaged by the imaging optical system from the plurality of image signals. An imaging apparatus characterized by that. A detection unit for detecting a subject from the image captured by the imaging unit;
The control means controls the imaging means and the distance distribution generation means according to the result of subject detection by the detection means to generate a distance distribution of each subject, and generates a distance distribution of the image from the distance distribution of each subject. The imaging apparatus according to claim 1, wherein:   When the detection unit detects different subjects, the distance distribution of each subject is a plurality of images having the parallax generated by imaging the optical image of each subject imaged by the imaging optical system with the imaging unit. 3. The distance distribution generated from the signal by the distance distribution generation unit, and the distance distribution of the image captured by the imaging unit is a distance distribution obtained by combining the distance distributions of the subjects. The imaging device described.   The control means determines a priority of generation of distance distribution by the distance distribution generation means for different subjects detected by the detection means, and controls the imaging means and distance distribution generation means according to the priority. The imaging apparatus according to claim 3, wherein a distance distribution of each subject is generated.   5. The imaging apparatus according to claim 3, wherein the distance distribution generation unit generates a distance distribution in an image region at a distance close to a focus position of the subject according to a result of subject detection by the detection unit. .   The detection unit divides the image captured by the imaging unit into rectangular regions, classifies the divided rectangular regions as regions of each subject, and the distance distribution generation unit is acquired from each rectangular region. The imaging apparatus according to claim 3 or 4, wherein a distance distribution of each subject is generated based on the distance information.   The imaging device according to claim 2, wherein the control unit stores the image captured by the imaging unit and a distance distribution of the image in association with each other. An image pickup apparatus control method comprising an image pickup means for picking up an optical image of a subject imaged by an image pickup optical system and generating a plurality of image signals having parallax,
A distance distribution generating step of acquiring distance information of the subject from the plurality of image signals, and generating a distance distribution of the subject based on the distance information;
A control step of controlling the imaging means and the distance distribution generation step to generate a distance distribution of an image including the subject;
With
In the control step, the depth of field in imaging for generating the plurality of image signals having the parallax is set to a shallow depth from which the distance information of the subject imaged by the imaging optical system can be acquired from the plurality of image signals. A control method comprising a step. A program for controlling an image pickup apparatus having an image pickup unit that picks up an optical image of a subject imaged by an image pickup optical system and generates a plurality of image signals having parallax,
Computer
Distance distribution generating means for acquiring distance information of the subject from the plurality of image signals, and generating a distance distribution of the subject based on the distance information;
Controlling the imaging means and the distance distribution generating means to generate a distance distribution of an image including the subject, and generating a plurality of image signals having the parallax; To function as a control means for making the distance information of the subject imaged by the imaging optical system shallow.   A computer-readable storage medium storing the program according to claim 9.

Images