JP6106026B2 - Image processing apparatus, imaging apparatus, playback apparatus, control method, and program - Google Patents

Description

The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program, and more particularly to a technique for generating an image focused on an arbitrary subject distance from output data after shooting.

  Digital data of an image captured by an imaging device such as a digital camera can be adjusted as desired by the user by using various image editing tools. One of the image editing functions desired by the user is a function of adjusting the in-focus state of the image after shooting. Conventionally, it is difficult to change the in-focus state of one image after shooting. For this reason, as disclosed in Japanese Patent Application Laid-Open No. H10-260260, there is a technique in which a plurality of images with different focus states are captured and recorded while the focus lens is driven, and an image in a desired in-focus state can be selected and used after shooting.

On the other hand, in recent years, in an imaging apparatus such as a digital camera, an image focused on an arbitrary subject distance is generated from the output data after recording, by recording the traveling direction of the light intensity distribution and the traveling direction as output data. Technology has been proposed.

One such technique is to form light beams that have passed through different divided pupil regions of the imaging lens on each pixel (photoelectric conversion element) of the imaging device via a microlens array, thereby allowing light incident from various directions. There is a method (Light Field Photography) to record separately. The obtained output data (Light Field Data, hereinafter referred to as LF data) records the light flux that is incident from different directions on adjacent pixels. From the LF data, an arbitrary subject distance is set, and the output of a pixel that records a light beam that has passed through one point on the focal plane corresponding to the subject distance is added, so that a specific subject distance is obtained after shooting. It is possible to artificially generate (reconstruct) the pixels of the image focused on.

JP 2005-277813 A

However, when an image corresponding to an arbitrary focal plane is reconstructed from the LF data as described above, it is necessary to specify pixels to be added according to the focal plane and perform an addition process for each pixel to generate an image. . That is, in order to generate a reconstructed image on the focal plane corresponding to the selected subject distance, time for calculation processing is required each time.

For example, when generating a reconstructed image focused on one subject, the user needs to select the subject distance to be generated. At this time, there is a possibility that the subject distance focused on the subject cannot always be selected. That is, until the reconstructed image is generated, the user cannot determine whether the selected subject distance is the subject distance focused on the subject. Therefore, it is necessary to repeatedly select the subject distance and generate the reconstructed image. As a result, it may take time until a reconstructed image having a desired focus state is obtained.

The present invention has been made in view of the above-described problems, and an image processing apparatus, an imaging apparatus, and a control method for generating an image signal that can easily specify a subject distance that is suitably focused on a desired subject. And to provide a program.

In order to achieve the above object, an image processing apparatus of the present invention is characterized by having the following configuration. Specifically, the image processing device generates light field data that is light field information of the subject , and generates a reconstructed image focused on a specific subject distance from the light field data acquired by the acquisition unit. Generating means for generating a plurality of reconstructed images having different focus states for a predetermined region in a reconstructed image that can be generated from light field data; and a plurality of reconstructed images generated by the generating means. And storage means for storing the light field data in association with the light field data.

With such a configuration, according to the present invention, it is possible to generate an image signal that can easily specify a subject distance that is suitably focused on a desired subject.

1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. The figure for demonstrating the relationship between the microlens array 105 which concerns on embodiment of this invention, and the imaging part 106. FIG. The figure for demonstrating the relationship between the light beam which passed each area | region of the exit pupil 301 which concerns on embodiment of this invention, and the photoelectric conversion element which photoelectrically converts this light beam. The figure which showed the response | compatibility with each area | region of the exit pupil 301 which concerns on embodiment of this invention, and the photoelectric conversion element matched with each micro lens The flowchart which illustrated the imaging | photography process performed with the digital camera 100 which concerns on embodiment of this invention. The figure for demonstrating the image for a reference produced | generated about an area | region in the imaging | photography process which concerns on embodiment of this invention. The figure which showed the detailed information about the image for a reference linked | related with LF data in the imaging | photography process which concerns on embodiment of this invention. The flowchart which illustrated the reproduction | regeneration processing performed with the digital camera 100 which concerns on embodiment of this invention. The figure which illustrated the display mode in the reproduction processing concerning the embodiment of the present invention

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example in which the present invention is applied to a digital camera that can generate an image focused on an arbitrary subject distance from LF data after shooting as an example of an image processing apparatus will be described. . However, the present invention can be applied to any device capable of generating an image focused on an arbitrary subject distance from LF data.

  In this specification, the following terms are defined and described.

・ "Light field data (LF data: Light Field Data)"
Data indicating three-dimensional information of the subject space. In the present embodiment, an image signal output from the imaging unit 106 included in the digital camera 100. Each pixel of the image signal indicates a signal intensity corresponding to a light beam having a different combination of the pupil region and the incident direction of the imaging optical system 104 that has passed. The LF data is also called ray space information. In the present embodiment, LF data is acquired by an optical system as shown in FIGS. 1 to 4, but the imaging means is not limited to this. For example, as in JP 2011-22796 A, a configuration in which a plurality of cameras with different viewpoints are collectively regarded as the imaging unit 106 may be used. In addition, unlike the optical system of FIG. 3, a configuration in which the light beam from the imaging optical system is imaged on the microlens array and the imaging device is provided on the imaging surface so that the object plane and the imaging device are in a conjugate relationship. But you can. Furthermore, the image from the imaging optical system is re-imaged on the microlens array (this is called re-imaging because the image once formed is diffused), and the image plane A configuration in which an image pickup element is provided in the apparatus may be used. A method of inserting a mask (gain modulation element) with an appropriate pattern into the optical path of the photographing optical system can also be used.

・ "Reconstructed image"
An image generated from LF data and focused on an arbitrary subject distance. Specifically, the pixels of the LF data are rearranged according to the pixel arrangement on the focal plane (reconstruction plane) corresponding to the object distance to be generated, and the pixel values of the pixels corresponding to one pixel of the reconstructed image are summed. Get the pixel value of the pixel. The pixel arrangement on the reconstruction plane is determined based on the incident direction of the incident light beam when the image sensor is present on the reconstruction plane. One pixel of the reconstructed image can be generated by adding pixel values of pixels corresponding to one microlens in the pixel arrangement.

<< Configuration of Digital Camera 100 >>
FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.

  The control unit 101 is a CPU, for example, and controls the operation of each block included in the digital camera 100. Specifically, the control unit 101 controls an operation of each block by reading an operation program for a shooting process or a refocus moving image generation process, which will be described later, stored in the ROM 102, and developing and executing the program on the RAM 103.

  The ROM 102 is, for example, a rewritable nonvolatile memory, and stores parameters necessary for the operation of each block in addition to the operation program for each block of the digital camera 100.

  The RAM 103 is a volatile memory. The RAM 103 is used not only as a development area for the operation program of each block included in the digital camera 100 but also as a storage area for storing intermediate data output in the operation of each block.

  The imaging unit 106 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 106 receives a timing signal output from a timing generator (TG) (not shown) according to an instruction from the control unit 101, and photoelectrically converts an optical image formed on the photoelectric conversion element surface of the imaging element by the imaging optical system 104. Convert and output analog image signal. Note that the imaging optical system 104 includes, for example, an objective lens, a focus lens, a diaphragm, and the like. In addition, the digital camera 100 according to the present embodiment includes a microlens array 105 between the imaging optical system 104 on the optical axis and the imaging element separately from the microlens provided in each photoelectric conversion element of the imaging element. .

<Relationship between microlens and photoelectric conversion element>
Here, the microlens array 105 provided between the imaging optical system 104 on the optical axis and the imaging element in the digital camera 100 of the present embodiment will be described with reference to the drawings.

  As shown in FIG. 2, the microlens array 105 of the present embodiment includes a plurality of microlenses 201. In FIG. 2, the optical axis of the imaging optical system 104 is the z axis, the horizontal direction of the digital camera 100 is the x axis, and the vertical direction is the y axis. In the example of FIG. 2, for the sake of simplicity, the microlens array 105 will be described as being configured by microlenses 201 arranged in 6 rows and 6 columns, but the configuration of the microlens array 105 is not limited to this. .

  In FIG. 2, the photoelectric conversion element 202 of the image pickup element that forms the image pickup unit 106 is shown by a lattice. A predetermined number of photoelectric conversion elements 202 are associated with each microlens 201, and in the example of FIG. 2, a photoelectric conversion element 202 of 6 × 6 = 36 pixels is associated with one microlens 201. Yes. The light beam that has passed through one microlens 201 is separated according to the incident direction, and is imaged on the corresponding photoelectric conversion element 202.

FIG. 3 illustrates light beams incident on the photoelectric conversion elements 202p _{1 to} p ₆ corresponding to one microlens 201. In FIG. 3, the upward direction corresponds to the vertically upward direction. In the figure, an optical path of a light beam incident on each photoelectric conversion element 202 viewed from the horizontal direction in a state where the digital camera 100 is in the horizontal position is illustrated. As shown in the figure, the photoelectric conversion elements 202p _{1 to} p ₆ arranged in the horizontal direction have regions a ₁ to a which are obtained by dividing the exit pupil 301 of the imaging optical system 104 into six in the vertical direction via one micro lens 201. The light beams that have passed through a ₆ are incident on each. In addition, the number attached | subjected to each area | region has shown the correspondence with the photoelectric conversion element 202 in which the light beam which passed through enters.

  In the example of FIG. 3, the optical path of the light beam incident on each photoelectric conversion element 202 viewed from the horizontal direction is shown, but the light beam separation is not limited to the vertical direction, and is similarly performed in the horizontal direction. That is, when the exit pupil 301 of the image pickup optical system 104 is classified into the regions as shown in FIG. 4A when viewed from the image pickup device side, the light beam that has passed through each region has a photoelectric as shown in FIG. Of the conversion elements 202, the light enters the photoelectric conversion elements with the same identification numbers. Here, it is assumed that the F numbers (F values) of the microlenses of the imaging optical system 104 and the microlens array 105 are substantially the same.

  The AFE 107 and the DFE 108 perform a correction process on the image signal generated by the imaging unit 106. Specifically, the AFE 107 performs reference level adjustment (clamp processing) and A / D conversion processing on the analog image signal output from the imaging unit 106, and outputs LF data to the DFE 108. The DFE 108 corrects a slight reference level shift or the like with respect to the input LF data.

The image processing unit 109 applies various image processing such as color conversion processing to the LF data to which the correction processing by the DFE 108 is applied. In the present embodiment, the image processing unit 109 also performs processing for generating an image (reconstructed image) that focuses on an arbitrary subject distance from the LF data. In the present embodiment, the image processing unit 109 includes a face detection unit 121, a face recognition unit 122, and a distance detection unit 123.

The face detection unit 121 detects an area corresponding to a human face pattern from the reconstructed image generated from the LF data by the image processing unit 109. In the face detection process, for example, feature points such as eye, nose and mouth end points and face contour points are extracted from the reconstructed image, and the size of the face area of the subject is acquired based on these feature points. Note that when photographing LF data, it is considered that there are human subjects at various positions in the photographing range with respect to the digital camera 100. Therefore, the face detection process is a reconstructed image generated for various subject distances. May be done about. Alternatively, for example, a reconstructed image generated only with pixels corresponding to the light beam that has passed through one divided pupil region among the light beams separated by the microlens 201 is equivalent to an image photographed in the aperture state. You may utilize for a face detection process. That is, since such a reconstructed image is an image having a deep depth of field as in the aperture state, it is possible to simultaneously detect human subjects having various distances from the digital camera 100.

  The face recognition unit 122 performs face recognition processing for recognizing whether the face area detected by the face detection unit 121 is a face area of a person whose feature amount is registered in advance. In the face recognition process, using the feature point arrangement information and the size of the face area described above, a similarity with a feature quantity of a person registered in advance is calculated, and whether or not the face area is the same person. Is judged.

The distance detection unit 123 detects the distance between each person subject existing in the shooting range at the time of shooting the LF data and the digital camera 100. The distance detection unit 123 detects the distance of each human subject by a phase difference detection method using a line sensor (not shown), for example. Alternatively, when acquiring LF data as in the present embodiment, for example, using two types of images generated using light beams that have passed through two different divided pupil regions, the distance between each person subject using the phase difference detection method May be detected. Also, as described above, when the reconstructed image is generated while changing the subject distance and the face detection processing is performed by the face detection unit 121, the subject distance that gives the highest contrast evaluation value is specified for each face region. Thus, the distance detection unit 123 may detect the distance of each person subject.

The display unit 110 is a display device included in the digital camera 100 such as a small LCD. The display unit 110 displays an image generated by the image processing unit 109 and focused on an arbitrary subject distance. As described above, the LF data obtained by A / D converting the analog image signal output from the imaging unit 106 according to the present embodiment does not connect the images in adjacent pixels. Therefore, not the LF data but the image data generated by the image processing unit 109 is displayed on the display unit 110.

The recording medium 111 is a recording device that is detachably connected to the digital camera 100 such as a built-in memory of the digital camera 100, a memory card, or an HDD. The recording medium 111 records LF data and an image focused on an arbitrary subject distance generated from these LF data.

  The operation input unit 112 is a user interface that the digital camera 100 has, such as a power button and a shutter button. When the operation input unit 112 detects that the user interface has been operated by the user, the operation input unit 112 outputs a control signal corresponding to the operation to the control unit 101. In the present embodiment, the display unit 110 is a touch panel display including a touch panel sensor, and a touch operation detected by the touch panel sensor is detected by the operation input unit 112 and transmitted to the control unit 101.

<Shooting process>
A specific process of the LF data shooting process executed in the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. Note that this imaging process will be described as being started when, for example, a digital camera 100 is instructed to shoot LF data.

  In step S <b> 501, the control unit 101 causes the imaging unit 106 to capture the received light beam, acquires LF data to which processing related to LF data generation by the AFE 107, DFE 108, and image processing unit 109 is applied. Store.

In step S <b> 502, the face detection unit 121 detects the face area of the human subject included in the LF data under the control of the control unit 101. Further, the distance detection unit 123 acquires information on the subject distance focused on each face area under the control of the control unit 101. Information on the position and size of the obtained face area and information on the subject distance focused on the face area are stored in the RAM 103, for example.

In step S <b> 503, the image processing unit 109 generates a reconstructed image that is a representative image of LF data under the control of the control unit 101. The reconstructed image serving as the representative image may be an image focused on the subject distance corresponding to the setting of the focus lens of the imaging optical system 104 at the time of shooting, for example. Specifically, the image processing unit 109 acquires the pixel value of each pixel of the reconstructed image serving as the representative image by adding the pixels corresponding to the light beams that have passed through each microlens 201.

  In step S <b> 504, the control unit 101 determines whether the user has set an area in which reconstructed images with different focus states can be referred to.

In the digital camera 100 of the present embodiment, the reconstruction generated to focus on a plurality of different subject distances for a predetermined region so as to be an index for determining the subject distance for generating the reconstructed image from the LF data. An image (reference image) is recorded in association with the LF data. That is, the recorded LF data is configured and recorded so that a reference image having a different focus state can be referred to for the region without generating a reconstructed image corresponding to the entire imaging range. The area may be set in advance before photographing the LF data by performing a touch operation on the surface of the display unit 110 by the user. In this step, it is determined whether or not such an area has been set. . In the digital camera 100 of the present embodiment, such an area is not only an area set by the user,
A face area of a person in which information about a feature amount for human recognition is registered. Two types of face areas existing in a central area of a shooting angle of view where a main subject is assumed to be present can be used. In the following description, priority is set for these areas in the order of "area set by the user", "face area of the person whose feature value is registered", and "face area existing in the center area" A reference image is generated for an area selected according to any one of the criteria. However, the implementation of the present invention is not limited to this, and a plurality of reference images having different focus states are generated and recorded in association with LF data for a predetermined region selected according to an arbitrary condition. Applicable if available.

  The control unit 101 moves the process to S <b> 505 if it is determined that an area that allows the reference image to be referred to has been set, and moves the process to S <b> 506 if it is determined that the area has not been set.

In step S <b> 505, the control unit 101 determines subject distances for generating a plurality of reference images having different focus states for the region set by the user. In the present embodiment, the plurality of reference images generated for a region include at least a reconstructed image reconstructed with respect to the focal plane corresponding to the subject distance focused on the subject in the region. If the area is not an area corresponding to a human face area, information about the distance between the subject in the area and the digital camera 100 is acquired in this step, and then the subject distance focused on the subject generates a reconstructed image. Is determined as one of the subject distances. In the digital camera 100 of the present embodiment, for the set region, in addition to the subject distance that focuses on the subject in the region, six types of subject distances are determined for generating a reconstructed image. For example, the six types of subject distances may be selected so as to have an even distribution over a range of subject distances in which a reconstructed image focused from LF data can be generated as shown in FIG. FIG. 6 shows an example in which the subject distance focused on the subject in the area is 6 m, and the range of the subject distance from which the reconstructed image can be generated from the LF data is 1 m to 11 m. 3m, 5m, 7m, 9m, and 11m are determined for generating a reconstructed image.

Further, the control unit 101 considers that the face area detected in S502, which is an area not set by the user, is low in importance, and for example, sets the shortest subject distance in the range of subject distances that can be generated, This is determined as the subject distance for generating the reference image.

  On the other hand, if it is determined in S504 that no area has been set, the control unit 101 causes the face recognition unit 122 to execute face recognition processing for the detected face area in S506, and the feature amount is previously stored in the face area. It is determined whether or not there is a face area of the person who has been registered. If the controller 101 determines that there is a face area of a person whose feature value has been registered in advance in the detected face area, the control section 101 moves the process to S507, and if it does not exist, moves the process to S508.

In step S507, as in step S505, the control unit 101 determines seven types of subject distances including the subject distance focused on the person for the face area of the person whose feature amount has been registered in advance for the reference image. Decide for production. The control unit 101 determines, for example, the shortest subject distance in the range of subject distances that can be generated for generating a reference image for a face area of a person whose feature value is not registered.

  Further, when it is determined that the face area of the person whose feature amount has been registered in advance does not exist in the face area detected in S506, the control unit 101 determines in S508 that the reconstructed image that can be generated from the LF data. It is determined whether or not there is a face area located in the central area. If the control unit 101 determines that there is a face region located in the central region, the process proceeds to S509. If it is determined that there is no face region, the control unit 101 proceeds to S511.

In step S509, as in step S505, the control unit 101 determines, for the face area existing in the central area, seven types of subject distances including a subject distance focused on a person in the face area for generating a reference image. To do. For the face region that does not exist in the central region, the control unit 101 determines, for example, the shortest subject distance in the range of subject distances that can be generated for generating a reference image.

In S510, under the control of the control unit 101, the image processing unit 109 generates a reference image that focuses on the determined subject distance for each region in which the reference image is generated. Then, the control unit 101 records the reference image and the detailed information of the reference image on the recording medium 111 in association with the LF data and the representative image, and completes the main photographing process. Each of the reference images includes information on the subject distance at which the image is generated.

  The detailed information may be information as shown in FIG. In the example of FIG. 7, the detailed information is managed for each region in which the reference image is generated for one LF data. Specifically, AREA_ID is identification information set for each area, AREA_Rect is information representative of the area's representative coordinates (for example, upper left coordinates) and the horizontal and vertical sizes of the area, and AREA_Files is generated for the area This is information on the file name of the reference image. JPEG_File is information for specifying a representative image of LF data. AREA_Flag is information indicating an area where a plurality of reference images in different focus states are generated. In the present embodiment, “area set by a user”, “a face area of a person whose feature value is registered”, “1” is input for “the face region existing in the central region”.

  On the other hand, if it is determined in S508 that the face area located in the central area does not exist, the control unit 101 records the LF data and the representative image in association with each other in S511 without associating the reference image, and performs the main photographing process. Complete.

  As described above, by performing the imaging processing of the present embodiment, it is possible to record a reference image that serves as an index of a focus state when generating a reconstructed image from LF data, together with the LF data.

《Reproduction processing》
Next, details of the reproduction process when browsing the LF data recorded by the above-described shooting process in the browsing mode will be described with reference to FIG. Note that this playback process will be described assuming that the digital camera 100 is started when the digital camera 100 is activated in a browsing mode for browsing LF data or JPEG data recorded on the recording medium 111, for example. Further, in the following description, for the sake of simplicity, only a process when an instruction to browse LF data is given will be described. When there is an instruction to browse other image data, for example, the same processing as the reproduction processing performed in the conventional imaging apparatus may be performed, and thus the details are omitted in this specification.

  In step S <b> 801, the control unit 101 determines whether an instruction to browse LF data has been issued. If the control unit 101 determines that an instruction to browse the LF data has been issued, the control unit 101 moves the process to S802. If the control unit 101 determines that an instruction has not been given, the process of this step is repeated.

  In step S <b> 802, the control unit 101 reads a representative image associated with the LF data and causes the display unit 110 to display the representative image. Specifically, the control unit 101 applies a decoding process related to display of a representative image and a conversion process to a video signal to the image processing unit 109, and then outputs the obtained video signal to the display unit 110. Displays an image.

In step S <b> 803, the control unit 101 determines whether a touch operation for selecting an area has been performed on the image displayed on the display unit 110. In the digital camera 100 of the present embodiment, the user browses the reference image generated for the touched region by performing a touch operation on the image as shown in FIG. 9A when browsing the LF data. be able to. At this time, the reference image is displayed so as to be selectable around the touched area as shown in FIG. 9B, and the user selects one of the displayed reference images to select the selected reference image. It is possible to view a reconstructed image for the subject distance for which is generated. That is, the reference image is used for selecting a subject distance for generating a reconstructed image, and the user can instruct generation of a reconstructed image after confirming the focus state of a desired region. That is, with such a configuration, in order to display a reconstructed image in a desired focus state, the user does not need to perform a trial-and-error for selecting a subject distance, and displays a reconstructed image in a desired focus state. It is possible to shorten the time required for the process.

  If the control unit 101 determines that a touch operation for selecting an area has been performed, the control unit 101 proceeds to step S804. If it is determined that no touch operation has been performed, the control unit 101 repeats the process of this step.

  In step S <b> 804, the control unit 101 refers to the detailed information associated with the LF data, and displays a reference image corresponding to the area where the touch operation has been performed, superimposed on the image displayed on the display unit 110. At this time, the reference image is arranged around the area where the touch operation is performed as described above. In this playback process, it is assumed that detailed information is associated with the LF data and a reference image exists for the touched area. However, if this is not the case, control is performed without performing the process of this step. The unit 101 may return the process to S803.

In step S <b> 805, the control unit 101 determines whether any reference image among the reference images displayed on the display unit 110 has been selected by the user. If the control unit 101 determines that one of the displayed reference images has been selected, the control unit 101 stores the subject distance information corresponding to the selected reference image in the RAM 103, and then performs processing. To S806. When the control unit 101 determines that none of the displayed reference images is selected, the control unit 101 repeats the process of this step. The selection of the reference image may be performed by performing a touch operation on the corresponding reference image.

In step S <b> 806, the image processing unit 109 refers to the subject distance information corresponding to the reference image selected in step S <b> 805 stored in the RAM 103 under the control of the control unit 101, and reconstructs to focus on the subject distance. An image is generated from LF data. Then, the control unit 101 displays the generated reconstructed image on the display unit 110, and returns the process to S803.

  By doing in this way, the user can instruct | indicate the production | generation of a reconstruction image, after confirming the focus state of a desired area | region.

As described above, the image processing apparatus of the present embodiment can generate an image signal that can easily specify a subject distance that is suitably focused on a desired subject. Specifically, the image processing apparatus can reconstruct an image signal corresponding to light beams having different combinations of divided pupil regions and incident directions that have passed through the imaging optical system of the imaging apparatus from the image signal. A predetermined area in the image is selected. Then, the image processing apparatus generates a plurality of reconstructed images having different focus states for the region, and stores them in association with the image signal. Further, at the time of reproducing the image signal, the image processing apparatus displays a plurality of reconstructed images having different focus states associated with the region so as to be an index for generating a reconstructed image for the image signal. When a reconstructed image is selected, a reconstructed image is generated from the image signal over the entire shooting range for the subject distance corresponding to the reconstructed image.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

An acquisition means for acquiring light field data which is light space information of the subject;
A generating unit that generates a reconstructed image focused on a specific subject distance from the light field data acquired by the acquiring unit, the focus on a predetermined region in the reconstructed image that can be generated from the light field data. Generating means for generating a plurality of reconstructed images in different states;
An image processing apparatus comprising: a storage unit that stores the plurality of reconstructed images generated by the generation unit in association with the light field data. Detecting means for detecting a person from a reconstructed image that can be generated from the light field data;
Recognizing means for recognizing whether or not the person detected by the detecting means is a predetermined person,
The predetermined area, the image processing apparatus according to claim 1, characterized in <br/> be a region corresponding to the recognized person to be predetermined person by the recognizing means. The generation means determines whether a person existing in a central area of a reconstructed image that can be generated from the light field data when there is no area corresponding to a person recognized as a person predetermined by the recognition means. The image processing apparatus according to claim 2, wherein the plurality of reconstructed images are generated with a corresponding region as the predetermined region . The image processing apparatus according to claim 1, further comprising an input unit that receives an instruction to set the predetermined area.   5. The image processing apparatus according to claim 1, wherein the plurality of reconstructed images include at least a reconstructed image in a focused state focused on a subject in the predetermined region.   The light field data is an image signal picked up by an image pickup device, and each of the pixels is an image signal corresponding to a luminous flux having a different combination of divided pupil regions and incident directions that have passed through the image pickup optical system of the image pickup device. The image processing apparatus according to claim 1, wherein the image processing apparatus is provided. An imaging unit that outputs an image signal in which each of the pixels corresponds to a luminous flux having a different combination of divided pupil regions and incident directions that have passed through the imaging optical system;
The image processing apparatus according to any one of claims 1 to 6,
An imaging device comprising: A selection means for selecting light field data which is light space information of the subject;
Control means for displaying an image corresponding to the light field data selected by the selection means on a display means;
Determining means for determining a region to be changed to a different focus state for the image displayed on the display means;
Generating means for generating a reconstructed image focused on a specific subject distance from the light field data,
The control means includes
For the area determined by the determining means, when a reconstructed image of the area having a different focus state is associated with the light field data, the reconstructed image of the area can be selected together with the image Displayed on the
When one reconstructed image among the reconstructed images of the region displayed on the display unit is selected, the object distance corresponding to the one reconstructed image generated by the generating unit is focused. A reconstructed image is displayed on the display means. An acquisition step in which an acquisition unit of the image processing apparatus acquires light field data that is light space information of the subject;
The generation means of the image processing device is a generation step of generating a reconstructed image focused on a specific subject distance from the light field data acquired in the acquisition step, and a reconstructable image that can be generated from the light field data. Generating a plurality of reconstructed images having different focus states for a predetermined region in the composition image;
A storage method of the image processing device, comprising: a storage step of storing the plurality of reconstructed images generated in the generation step in association with the light field data. . A selection step in which the selection means of the playback device selects light field data that is light space information of the subject;
A control step in which the control means of the playback device displays an image corresponding to the light field data selected in the selection step on a display means;
A determination step of determining a region to be changed to a different focus state with respect to the image displayed on the display unit;
Generating means for generating a reconstructed image focused on a specific subject distance from the light field data, the generating means of the playback device,
The control means in the control step,
For the region determined in the determining step, when a reconstructed image of the region having a different focus state is associated with the light field data, the display unit can select the reconstructed image of the region together with the image Displayed on the
When one reconstructed image of the reconstructed image of the area displayed on the display means is selected, the object distance corresponding to the one reconstructed image generated in the generating step is focused. A control method for a reproducing apparatus, wherein the reconstructed image is displayed on the display means.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 6.   The program for functioning a computer as each means of the reproducing | regenerating apparatus of Claim 8.

Images