JP2014230172A5 - - Google Patents

Description

The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program, and more particularly to a technique for generating an image focused on an arbitrary subject distance from output data after shooting.

On the other hand, in recent years, in an imaging apparatus such as a digital camera, an image focused on an arbitrary subject distance is generated from the output data after recording, by recording the traveling direction of the light intensity distribution and the traveling direction as output data. Technology has been proposed.

One such technique is to form light beams that have passed through different divided pupil regions of the imaging lens on each pixel (photoelectric conversion element) of the imaging device via a microlens array, thereby allowing light incident from various directions. There is a method (Light Field Photography) to record separately. The obtained output data (Light Field Data, hereinafter referred to as LF data) records the light flux that is incident from different directions on adjacent pixels. From the LF data, an arbitrary subject distance is set, and the output of a pixel that records a light beam that has passed through one point on the focal plane corresponding to the subject distance is added, so that a specific subject distance is obtained after shooting. It is possible to artificially generate (reconstruct) the pixels of the image focused on.

However, when an image corresponding to an arbitrary focal plane is reconstructed from the LF data as described above, it is necessary to specify pixels to be added according to the focal plane and perform an addition process for each pixel to generate an image. . That is, in order to generate a reconstructed image on the focal plane corresponding to the selected subject distance, time for calculation processing is required each time.

For example, when generating a reconstructed image focused on one subject, the user needs to select the subject distance to be generated. At this time, there is a possibility that the subject distance focused on the subject cannot always be selected. That is, until the reconstructed image is generated, the user cannot determine whether the selected subject distance is the subject distance focused on the subject. Therefore, it is necessary to repeatedly select the subject distance and generate the reconstructed image. As a result, it may take time until a reconstructed image having a desired focus state is obtained.

The present invention has been made in view of the above-described problems, and an image processing apparatus, an imaging apparatus, and a control method for generating an image signal that can easily specify a subject distance that is suitably focused on a desired subject. And to provide a program.

In order to achieve the above object, an image processing apparatus of the present invention is characterized by having the following configuration. Specifically, the image processing device generates light field data that is light field information of the subject , and generates a reconstructed image focused on a specific subject distance from the light field data acquired by the acquisition unit. Generating means for generating a plurality of reconstructed images having different focus states for a predetermined region in a reconstructed image that can be generated from light field data; and a plurality of reconstructed images generated by the generating means. And storage means for storing the light field data in association with the light field data.

With such a configuration, according to the present invention, it is possible to generate an image signal that can easily specify a subject distance that is suitably focused on a desired subject.

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example in which the present invention is applied to a digital camera that can generate an image focused on an arbitrary subject distance from LF data after shooting as an example of an image processing apparatus will be described. . However, the present invention can be applied to any device capable of generating an image focused on an arbitrary subject distance from LF data.

・ "Reconstructed image"
An image generated from LF data and focused on an arbitrary subject distance. Specifically, the pixels of the LF data are rearranged according to the pixel arrangement on the focal plane (reconstruction plane) corresponding to the object distance to be generated, and the pixel values of the pixels corresponding to one pixel of the reconstructed image are summed. Get the pixel value of the pixel. The pixel arrangement on the reconstruction plane is determined based on the incident direction of the incident light beam when the image sensor is present on the reconstruction plane. One pixel of the reconstructed image can be generated by adding pixel values of pixels corresponding to one microlens in the pixel arrangement.

The image processing unit 109 applies various image processing such as color conversion processing to the LF data to which the correction processing by the DFE 108 is applied. In the present embodiment, the image processing unit 109 also performs processing for generating an image (reconstructed image) that focuses on an arbitrary subject distance from the LF data. In the present embodiment, the image processing unit 109 includes a face detection unit 121, a face recognition unit 122, and a distance detection unit 123.

The face detection unit 121 detects an area corresponding to a human face pattern from the reconstructed image generated from the LF data by the image processing unit 109. In the face detection process, for example, feature points such as eye, nose and mouth end points and face contour points are extracted from the reconstructed image, and the size of the face area of the subject is acquired based on these feature points. Note that when photographing LF data, it is considered that there are human subjects at various positions in the photographing range with respect to the digital camera 100. Therefore, the face detection process is a reconstructed image generated for various subject distances. May be done about. Alternatively, for example, a reconstructed image generated only with pixels corresponding to the light beam that has passed through one divided pupil region among the light beams separated by the microlens 201 is equivalent to an image photographed in the aperture state. You may utilize for a face detection process. That is, since such a reconstructed image is an image having a deep depth of field as in the aperture state, it is possible to simultaneously detect human subjects having various distances from the digital camera 100.

The distance detection unit 123 detects the distance between each person subject existing in the shooting range at the time of shooting the LF data and the digital camera 100. The distance detection unit 123 detects the distance of each human subject by a phase difference detection method using a line sensor (not shown), for example. Alternatively, when acquiring LF data as in the present embodiment, for example, using two types of images generated using light beams that have passed through two different divided pupil regions, the distance between each person subject using the phase difference detection method May be detected. Also, as described above, when the reconstructed image is generated while changing the subject distance and the face detection processing is performed by the face detection unit 121, the subject distance that gives the highest contrast evaluation value is specified for each face region. Thus, the distance detection unit 123 may detect the distance of each person subject.

The display unit 110 is a display device included in the digital camera 100 such as a small LCD. The display unit 110 displays an image generated by the image processing unit 109 and focused on an arbitrary subject distance. As described above, the LF data obtained by A / D converting the analog image signal output from the imaging unit 106 according to the present embodiment does not connect the images in adjacent pixels. Therefore, not the LF data but the image data generated by the image processing unit 109 is displayed on the display unit 110.

The recording medium 111 is a recording device that is detachably connected to the digital camera 100 such as a built-in memory of the digital camera 100, a memory card, or an HDD. The recording medium 111 records LF data and an image focused on an arbitrary subject distance generated from these LF data.

In step S <b> 502, the face detection unit 121 detects the face area of the human subject included in the LF data under the control of the control unit 101. Further, the distance detection unit 123 acquires information on the subject distance focused on each face area under the control of the control unit 101. Information on the position and size of the obtained face area and information on the subject distance focused on the face area are stored in the RAM 103, for example.

In step S <b> 503, the image processing unit 109 generates a reconstructed image that is a representative image of LF data under the control of the control unit 101. The reconstructed image serving as the representative image may be an image focused on the subject distance corresponding to the setting of the focus lens of the imaging optical system 104 at the time of shooting, for example. Specifically, the image processing unit 109 acquires the pixel value of each pixel of the reconstructed image serving as the representative image by adding the pixels corresponding to the light beams that have passed through each microlens 201.

In the digital camera 100 of the present embodiment, the reconstruction generated to focus on a plurality of different subject distances for a predetermined region so as to be an index for determining the subject distance for generating the reconstructed image from the LF data. An image (reference image) is recorded in association with the LF data. That is, the recorded LF data is configured and recorded so that a reference image having a different focus state can be referred to for the region without generating a reconstructed image corresponding to the entire imaging range. The area may be set in advance before photographing the LF data by performing a touch operation on the surface of the display unit 110 by the user. In this step, it is determined whether or not such an area has been set. . In the digital camera 100 of the present embodiment, such an area is not only an area set by the user,
A face area of a person in which information about a feature amount for human recognition is registered. Two types of face areas existing in a central area of a shooting angle of view where a main subject is assumed to be present can be used. In the following description, priority is set for these areas in the order of "area set by the user", "face area of the person whose feature value is registered", and "face area existing in the center area" A reference image is generated for an area selected according to any one of the criteria. However, the implementation of the present invention is not limited to this, and a plurality of reference images having different focus states are generated and recorded in association with LF data for a predetermined region selected according to an arbitrary condition. Applicable if available.

In step S <b> 505, the control unit 101 determines subject distances for generating a plurality of reference images having different focus states for the region set by the user. In the present embodiment, the plurality of reference images generated for a region include at least a reconstructed image reconstructed with respect to the focal plane corresponding to the subject distance focused on the subject in the region. If the area is not an area corresponding to a human face area, information about the distance between the subject in the area and the digital camera 100 is acquired in this step, and then the subject distance focused on the subject generates a reconstructed image. Is determined as one of the subject distances. In the digital camera 100 of the present embodiment, for the set region, in addition to the subject distance that focuses on the subject in the region, six types of subject distances are determined for generating a reconstructed image. For example, the six types of subject distances may be selected so as to have an even distribution over a range of subject distances in which a reconstructed image focused from LF data can be generated as shown in FIG. FIG. 6 shows an example in which the subject distance focused on the subject in the area is 6 m, and the range of the subject distance from which the reconstructed image can be generated from the LF data is 1 m to 11 m. 3m, 5m, 7m, 9m, and 11m are determined for generating a reconstructed image.

Further, the control unit 101 considers that the face area detected in S502, which is an area not set by the user, is low in importance, and for example, sets the shortest subject distance in the range of subject distances that can be generated, This is determined as the subject distance for generating the reference image.

In step S507, as in step S505, the control unit 101 determines seven types of subject distances including the subject distance focused on the person for the face area of the person whose feature amount has been registered in advance for the reference image. Decide for production. The control unit 101 determines, for example, the shortest subject distance in the range of subject distances that can be generated for generating a reference image for a face area of a person whose feature value is not registered.

In step S509, as in step S505, the control unit 101 determines, for the face area existing in the central area, seven types of subject distances including a subject distance focused on a person in the face area for generating a reference image. To do. For the face region that does not exist in the central region, the control unit 101 determines, for example, the shortest subject distance in the range of subject distances that can be generated for generating a reference image.

In S510, under the control of the control unit 101, the image processing unit 109 generates a reference image that focuses on the determined subject distance for each region in which the reference image is generated. Then, the control unit 101 records the reference image and the detailed information of the reference image on the recording medium 111 in association with the LF data and the representative image, and completes the main photographing process. Each of the reference images includes information on the subject distance at which the image is generated.

In step S <b> 803, the control unit 101 determines whether a touch operation for selecting an area has been performed on the image displayed on the display unit 110. In the digital camera 100 of the present embodiment, the user browses the reference image generated for the touched region by performing a touch operation on the image as shown in FIG. 9A when browsing the LF data. be able to. At this time, the reference image is displayed so as to be selectable around the touched area as shown in FIG. 9B, and the user selects one of the displayed reference images to select the selected reference image. It is possible to view a reconstructed image for the subject distance for which is generated. That is, the reference image is used for selecting a subject distance for generating a reconstructed image, and the user can instruct generation of a reconstructed image after confirming the focus state of a desired region. That is, with such a configuration, in order to display a reconstructed image in a desired focus state, the user does not need to perform a trial-and-error for selecting a subject distance, and displays a reconstructed image in a desired focus state. It is possible to shorten the time required for the process.

In step S <b> 805, the control unit 101 determines whether any reference image among the reference images displayed on the display unit 110 has been selected by the user. If the control unit 101 determines that one of the displayed reference images has been selected, the control unit 101 stores the subject distance information corresponding to the selected reference image in the RAM 103, and then performs processing. To S806. When the control unit 101 determines that none of the displayed reference images is selected, the control unit 101 repeats the process of this step. The selection of the reference image may be performed by performing a touch operation on the corresponding reference image.

In step S <b> 806, the image processing unit 109 refers to the subject distance information corresponding to the reference image selected in step S <b> 805 stored in the RAM 103 under the control of the control unit 101, and reconstructs to focus on the subject distance. An image is generated from LF data. Then, the control unit 101 displays the generated reconstructed image on the display unit 110, and returns the process to S803.

As described above, the image processing apparatus of the present embodiment can generate an image signal that can easily specify a subject distance that is suitably focused on a desired subject. Specifically, the image processing apparatus can reconstruct an image signal corresponding to light beams having different combinations of divided pupil regions and incident directions that have passed through the imaging optical system of the imaging apparatus from the image signal. A predetermined area in the image is selected. Then, the image processing apparatus generates a plurality of reconstructed images having different focus states for the region, and stores them in association with the image signal. Further, at the time of reproducing the image signal, the image processing apparatus displays a plurality of reconstructed images having different focus states associated with the region so as to be an index for generating a reconstructed image for the image signal. When a reconstructed image is selected, a reconstructed image is generated from the image signal over the entire shooting range for the subject distance corresponding to the reconstructed image.

Claims (12)

An acquisition means for acquiring light field data which is light space information of the subject;
A generating unit that generates a reconstructed image focused on a specific subject distance from the light field data acquired by the acquiring unit, the focus on a predetermined region in the reconstructed image that can be generated from the light field data. Generating means for generating a plurality of reconstructed images in different states;
An image processing apparatus comprising: a storage unit that stores the plurality of reconstructed images generated by the generation unit in association with the light field data. Detecting means for detecting a person from a reconstructed image that can be generated from the light field data;
Recognizing means for recognizing whether or not the person detected by the detecting means is a predetermined person,
The predetermined area, the image processing apparatus according to claim 1, characterized in <br/> be a region corresponding to the recognized person to be predetermined person by the recognizing means. The generation means determines whether a person existing in a central area of a reconstructed image that can be generated from the light field data when there is no area corresponding to a person recognized as a person predetermined by the recognition means. The image processing apparatus according to claim 2, wherein the plurality of reconstructed images are generated with a corresponding region as the predetermined region . The image processing apparatus according to claim 1, further comprising an input unit that receives an instruction to set the predetermined area.   5. The image processing apparatus according to claim 1, wherein the plurality of reconstructed images include at least a reconstructed image in a focused state focused on a subject in the predetermined region.   The light field data is an image signal picked up by an image pickup device, and each of the pixels is an image signal corresponding to a luminous flux having a different combination of divided pupil regions and incident directions that have passed through the image pickup optical system of the image pickup device. The image processing apparatus according to claim 1, wherein the image processing apparatus is provided. An imaging unit that outputs an image signal in which each of the pixels corresponds to a luminous flux having a different combination of divided pupil regions and incident directions that have passed through the imaging optical system;
The image processing apparatus according to any one of claims 1 to 6,
An imaging device comprising: A selection means for selecting light field data which is light space information of the subject;
Control means for displaying an image corresponding to the light field data selected by the selection means on a display means;
Determining means for determining a region to be changed to a different focus state for the image displayed on the display means;
Generating means for generating a reconstructed image focused on a specific subject distance from the light field data,
The control means includes
For the area determined by the determining means, when a reconstructed image of the area having a different focus state is associated with the light field data, the reconstructed image of the area can be selected together with the image Displayed on the
When one reconstructed image among the reconstructed images of the region displayed on the display unit is selected, the object distance corresponding to the one reconstructed image generated by the generating unit is focused. A reconstructed image is displayed on the display means. An acquisition step in which an acquisition unit of the image processing apparatus acquires light field data that is light space information of the subject;
The generation means of the image processing device is a generation step of generating a reconstructed image focused on a specific subject distance from the light field data acquired in the acquisition step, and a reconstructable image that can be generated from the light field data. Generating a plurality of reconstructed images having different focus states for a predetermined region in the composition image;
A storage method of the image processing device, comprising: a storage step of storing the plurality of reconstructed images generated in the generation step in association with the light field data. . A selection step in which the selection means of the playback device selects light field data that is light space information of the subject;
A control step in which the control means of the playback device displays an image corresponding to the light field data selected in the selection step on a display means;
A determination step of determining a region to be changed to a different focus state with respect to the image displayed on the display unit;
Generating means for generating a reconstructed image focused on a specific subject distance from the light field data, the generating means of the playback device,
The control means in the control step,
For the region determined in the determining step, when a reconstructed image of the region having a different focus state is associated with the light field data, the display unit can select the reconstructed image of the region together with the image Displayed on the
When one reconstructed image of the reconstructed image of the area displayed on the display means is selected, the object distance corresponding to the one reconstructed image generated in the generating step is focused. A control method for a reproducing apparatus, wherein the reconstructed image is displayed on the display means.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 6.   The program for functioning a computer as each means of the reproducing | regenerating apparatus of Claim 8.

Images