JP6298494B2 - Image reproducing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus typified by a digital camera, and more particularly to an image recording apparatus photographed by an imaging apparatus having a refocus function and an apparatus for reproducing the recorded image.

  2. Description of the Related Art In recent years, an imaging apparatus having a configuration in which, in addition to pixel signals, information on the incident direction of a light beam incident on an imaging element can be obtained by arranging a microlens array arranged at a ratio of one for a plurality of pixels in front of the imaging element Has been proposed in Non-Patent Document 1 and the like. As an application of such an imaging apparatus, in addition to generating a normal captured image based on an output signal from each pixel, a predetermined image processing is performed on the captured image, thereby allowing an arbitrary focal length. It is possible to reconstruct an image focused on.

  When such a refocusable image is displayed for viewing, there is a problem that it is difficult to determine which focus position the refocused image should be displayed first. In order to solve this problem, in Patent Document 1, an object is extracted from an image, and a refocus image focused on a portion where the object exists is displayed. Alternatively, imaging conditions at the time of image acquisition are assigned to the image data as tag data, and a refocus image is generated based on the tag information.

JP 2011-22796 A

Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, in the prior art disclosed in the above-mentioned patent document, processing for extracting an object is required before displaying an image. Even when information at the time of shooting is included as tag data, it is necessary to calculate the phase difference between multiple images with different viewpoints in order to determine the refocus plane after reconstruction. There is a problem that calculation cost occurs.

  Accordingly, an object of the present invention is to provide an image data recording device and reproducing device that can quickly display image data acquired by an imaging device capable of photographing a subject as a refocusable image as a desired refocus image. Is to provide.

According to one aspect of the implementation form, an image reproducing device for reproducing an image corresponding to the image data, to reconstruct the image data, generating means for generating an image corresponding to the virtual aperture Setting means for setting the virtual aperture value; and display control means for displaying an image corresponding to the virtual aperture value generated by the generating means on a display unit, the display control means Displays an image and a graphic user interface for instructing the virtual aperture value of the image set by the setting unit, and the display control unit includes a plurality of images generated by the generation unit. Are simultaneously displayed.
According to another aspect of the embodiment, there is provided an image reproduction device that reproduces an image corresponding to image data, and a generation unit that generates an image corresponding to a virtual aperture value by reconstructing the image data. Setting means for setting the virtual aperture value; and display control means for displaying an image corresponding to the virtual aperture value generated by the generating means on a display unit, the display control means Displays an image and a graphic user interface for instructing the virtual aperture value of the image set by the setting unit, and the display control unit includes a plurality of images generated by the generation unit. Is provided at the same time according to the priority order recorded together with the image data used to generate the plurality of images.

According to yet another aspect of the implementation form, method of controlling an image reproducing apparatus for reproducing an image corresponding to the image data, to reconstruct the image data, an image corresponding to the virtual aperture A generating step for generating, a setting step for setting the virtual aperture value, and a display control step for displaying the image corresponding to the virtual aperture value generated in the generating step on a display unit, The display control step displays an image and a graphic user interface for instructing the virtual aperture value of the image set in the setting step, and the display control step includes the generation There is provided a method for controlling an image reproducing device, wherein a plurality of images generated in the step are simultaneously displayed.
According to still another aspect of the embodiment, there is provided a method for controlling an image reproducing device that reproduces an image corresponding to image data, and generating an image corresponding to a virtual aperture value by reconstructing the image data. A generating step for setting, a setting step for setting the virtual aperture value, and a display control step for displaying the image corresponding to the virtual aperture value generated in the generating step on a display unit. In the display control step, an image and a graphic user interface for instructing the virtual aperture value of the image set in the setting step are displayed. In the display control step, the generation step a plurality of images generated in accordance with the priority order that is recorded together with the image data used to generate the plurality of images, and wherein the simultaneously displaying That the control method of the image reproducing device is provided.

  According to the present invention, it is possible to provide an image data recording apparatus and a reproduction apparatus for the recorded data that can quickly display a refocus image in a desired in-focus state.

It is a figure which shows the structure of the image data which has a reconstruction parameter in metadata. 1 is a block diagram illustrating a configuration of an imaging apparatus to which a recording apparatus according to a first embodiment of the present invention is applied. It is a figure which shows typically the structure of the imaging device and microlens array which are used with the imaging device of FIG. It is a figure which shows notionally the structure of the imaging optical system which consists of a photographic lens, a micro lens array, and an image pick-up element in the imaging device of FIG. FIG. 4 is a diagram schematically illustrating a correspondence relationship between a pupil region of a photographing lens of the imaging apparatus in FIG. 2 and pixels of the imaging element in FIG. 3. FIG. 4 is a diagram schematically showing generation of signals for focus detection and refocus plane detection using the output of the image sensor of FIG. 3. It is a figure which shows the relationship between the light ray in a different refocus surface, and its light reception pixel. FIG. 6 is a flowchart illustrating a recording operation according to the first embodiment of the present invention. It is a figure which shows notionally the image containing a several subject in a different distance, and the defocus map of the image. It is a figure which shows the flowchart of the reproduction | regeneration operation | movement of the image reproduction apparatus which concerns on 2nd Example of this invention. It is a figure which shows typically the display screen of the reproduction image in the image reproduction apparatus which concerns on 2nd Example of this invention. It is a figure which shows notionally other optical systems applicable to the Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 2 is a block diagram of a digital camera as an imaging apparatus to which the image recording apparatus according to the first embodiment of the present invention is applied. Reference numeral 201 denotes a photographic lens, which is composed of a plurality of lenses (not shown). The plurality of lenses include a movable focusing lens, and the focusing state of the subject can be adjusted by moving the focusing lens. Reference numeral 202 denotes a microlens array (hereinafter referred to as MLA), which is composed of a plurality of microlenses and is arranged near the focal position of the photographing lens 201. Light that has passed through different pupil regions of the photographing lens 201 enters the MLA 202 and is emitted separately for each pupil region. Reference numeral 203 denotes an image sensor, which includes a CCD, a CMOS image sensor, or the like, and is disposed near the focal position of the MLA 202. Details of the MLA 202 and the image sensor 203 will be described later.

  An AD converter 204 AD-converts an image signal output from the image sensor 203 into digital data, and an image processing unit 205 performs predetermined image processing on the digital data to obtain digital image data of a subject. . Reference numeral 206 denotes a photographing control unit that displays the digital image data obtained by the image processing unit 205 on a display unit 207 configured with a liquid crystal display or the like, and controls storage in the recording unit 208. The imaging control unit 206 controls each unit by loading and executing a program stored in a memory (not shown) by the CPU. In this case, the CPU may perform all or part of the functions of each unit, or may be configured by hardware.

  A face detection unit 209 detects a human face from the digital image data obtained by the image processing unit 205, and detects the position and size in the screen. The operation unit 210 is a part that accepts an operation from a user such as a button or a touch panel. In response to the accepted operation, the operation unit 210 starts a focusing operation, reproduces or erases digital image data stored in the recording unit 208, and the like. Various operations are performed. The photographing lens 201 is electrically and mechanically connected to the photographing control unit 206. The photographing control unit 206 can acquire information on the photographing lens by communication, and sends a focusing lens driving command and the like during the focusing operation. be able to.

  Next, the configuration of the taking lens 201, the MLA 202, and the image pickup device 203 in the image pickup apparatus shown in FIG. 2 will be described.

  FIG. 3 is a diagram illustrating the configuration of the image sensor 203 and the MLA 202. This figure shows the image sensor 203 and the MLA 202 as seen from the z direction parallel to the optical axis from the photographic lens 201. One microlens 302 is arranged so as to correspond to a plurality of unit pixels 301 (photoelectric conversion elements) constituting the virtual pixel 300 of the photographed image. The microlens 302 is one of those constituting the MLA 202. Here, it has been described that there are a total of 25 unit pixels 301 corresponding to one microlens in 5 rows and 5 columns. The coordinate axes attached to the figure indicate the optical axis direction as the z-axis, the plane perpendicular to the z-axis is a plane parallel to the imaging plane, and the x-axis (horizontal direction) and the y-axis in the imaging plane. (Vertical direction) is defined. FIG. 3 shows a part of the light receiving surface of the image sensor 203, and an actual image sensor has a great number of pixels arranged therein.

FIG. 4 is a view of a state in which a light beam emitted from the photographing lens 201 passes through one micro lens 302 and is received by the image sensor 203, as viewed from a direction perpendicular to the optical axis z. Light emitted from each of the pupil regions a _{1 to} a _{5 of the} photographing lens 201 and having passed through the micro lens 302 forms an image on the corresponding unit pixels p _{1 to} p ₅ of the image sensor 203.

FIG. 5A is a view of the aperture of the photographing lens 201 as viewed from the optical axis (z-axis) direction. FIG. 5B is a view of one microlens 302 and the unit pixel 300 disposed behind the microlens 302 as seen from the optical axis (z-axis) direction. As shown in FIG. 5A, when the pupil region of the photographing lens 201 is divided into the same number of regions as the pixels under one microlens, one pixel is separated from one pupil division region of the photographing lens 201. Light is imaged. However, it is assumed here that the F-numbers of the photographing lens and the microlens are substantially the same. When the pupil division region of the photographic lens shown in FIG. 5A is a _{11 to} a ₅₅ and the pixels shown in FIG. 5B are p ₁₁ to p ₅₅ , the correspondence between them is the optical axis (z-axis) direction. Point symmetry. Therefore, light emitted from the pupil division area a ₁₁ of the taking lens of the pixel 301 at the back microlenses to form an image on the pixel p _11. Similarly, it emitted from the pupil division area a _11, light passing through the different micro lenses to form an image on the pixel p ₁₁ in the pixel 301 in the microlens rear.

As described above, since the pixels p ₁₁ to p ₅₅ shown in FIG. 5B receive light that has passed through different pupil regions with respect to the photographing lens, the focus detection operation is performed using the outputs of these pixels. You can also. As shown in FIG. 6A, the pixel outputs p _{11 to} p ₅₅ corresponding to each microlens are added to generate two signals that are pupil-divided in the horizontal direction as shown in the following equation.

As shown in FIG. 4, the combined signal A expressed by Equation 1 looks at the light flux that has passed through the exit pupil areas a _{1 to} a ₂ of the photographing lens. As shown in FIG. 6 (b), if we lined up the A ₁ to A _n obtained from the pixel group corresponding to the n-number of micro lenses arranged in succession in the horizontal direction, one-dimensional image signal A _{i (i} = 1, 2, 3 ... n). Similarly, when a one-dimensional image signal B _i (i = 1, 2, 3... N) in which the composite signal B calculated by Expression 2 is arranged is generated, A _i and B _i are respectively on the right side of the exit pupil of the photographing lens. And the signal looking to the left. Therefore, the focus detection operation based on the phase difference detection method can be performed according to the result of detecting the relative position of A _i and B _i and multiplying the relative shift amount by the predetermined conversion coefficient. If A _i and B _i are generated at an arbitrary position in the screen, the focal position at that position can be calculated. Therefore, automatic focusing can be performed by driving the focusing lens according to the calculation result. it can.

Next, processing for reconstructing digital image data acquired by the photographing lens 201, the MLA 202, and the image sensor 203 into an image at an arbitrarily set focus position (refocus plane) will be described. FIG. 7 shows the state of light rays when the designated refocus plane is z = 0 and z = d. Considering the central pixel in FIG. 7, when z = 0, the light flux emitted from the microlens enters the pixels a3, b3, c3, 3d, and 3e, respectively. Therefore,
[Formula 3]
S3 = a3 + b3 + c3 + d3 + e3
S3 may be obtained as an output of a virtual pixel. The same applies to pixels such as S1 other than S3.

Next, when it is desired to reconstruct an image when the aperture of the lens is stopped, it is only necessary to use a light beam only in a region near the center of the photographing lens according to the aperture value. For example, [Equation 4]
S3 = b3 + c3 + d3
Then, it is possible to obtain a reconstructed image at z = 0 when narrowed down than that obtained by Equation 3,
[Formula 5]
S3 = c3
Then, the image at z = 0 when further narrowed down can be reconstructed. That is, the image in the narrowing direction can be reconstructed.
Here, for the sake of explanation, only five pixels arranged in the x direction have been described as an example. However, since pixels are actually arranged in the xy plane, the same concept is applied in two dimensions. For example, S3 in the case of full aperture is the sum of 25 pixel outputs.

On the other hand, when z = d, the light flux emitted from the surface with z = d is incident on the pixels a1, b2, c3, d4, and e5, respectively.
[Formula 6]
S3 = a1 + b2 + c3 + d4 + e5
S3 may be generated as follows. Thus, if the position z (refocus position) of the refocus plane and the aperture value F at the time of reconstruction are determined, a reconstructed image can be calculated. When this equation 6 is applied to other reconstruction pixels other than S3, a reconstruction image in the case of z = d can be obtained by the following matrix calculation.

を準備しておけば、全画素出力を行列にしたものにＭを乗算するだけで、容易に再構成画像を得ることができる点である。 In other words, the expression 7 means that if the position z of the refocus plane of the reconstructed image and the aperture value F at the time of reconstruction are known, the following conversion matrix M

Is prepared, a reconstructed image can be easily obtained simply by multiplying all pixel outputs in a matrix by M.

  Here, the recording operation of this embodiment will be described with reference to the flowchart of FIG. In the case of the present embodiment, this operation is performed under the control of the imaging control unit 206 of the camera. When the camera operation is started in step S801, it waits for the user to turn on the switch S1, which is a pre-shooting preparation operation, in step S802. Here, the pre-photographing preparation operation is an AE / AF operation, and S1 is turned on when the user operates an operation member included in the operation unit 209. Here, a case where a two-stage push-type push button switch widely used in a general camera or the like is half-pressed is set to ON of S1, and a case where it is fully pressed is set to ON of S2. If S1 is turned on, the process proceeds to step S803.

  Steps S803 to S806 are steps for performing AE / AF, which is a pre-shooting preparation operation. In step S803, AE is performed. The AE evaluates the exposure level of the image from the output signal level of the image sensor 203 acquired under a predetermined condition, and determines an exposure condition that minimizes the saturation of the output signal and blackout. As already described, the reconstructed image in the direction in which the aperture value increases (the direction in which the aperture is narrowed down) can be calculated according to Equations 3 to 5, and therefore it is desirable to shoot with the aperture open as much as possible.

  In S804 to S806, an AF operation is performed. As already described in the description of FIG. 6, the camera of FIG. 2 can calculate the in-focus position at an arbitrary position in the screen. Therefore, for example, as shown in FIG. 9A, for a subject in which buildings, persons, and cars are distributed at different distances, the screen is divided into 63 blocks of 9 × 7 in length, and the in-focus positions are set. If calculated, a defocus map shown in FIG. 9B can be obtained. In FIG. 9B, the defocus amount of the portion corresponding to the background in the screen is shown as db. In step S804, the defocus map is acquired in this way.

By grouping blocks with similar defocus values with respect to such a defocus map, the approximate number of subjects can be estimated (subject detection means). Further, when the face detection unit 209 detects the presence of a human face on the screen, the position can be known. Therefore, in the example of FIG. 9, it can be seen that there are three subjects at the positions where the defocus amounts are d1, d2, and d3, and the subject at the position of d2 is a person. Therefore, in this embodiment, priority is given to the subject according to the following rules (priority means).
Priority 1. Faces (If there are multiple faces, prioritize them in descending order of detected face size)
Priority 2. Prioritize subjects other than the face in descending order of defocus amount.

  When this rule is applied, in the example of FIG. 9, the person with the defocus d2 is the subject with the highest priority, the next is not the face but the defocus d1 car on the closest side, and finally the defocus d3 building. Become. In this way, in step S805, the subject is identified from the defocus map, and the priority order is assigned to the identified subject. In addition, although the prioritization rule of the thought which gives priority to a person's face was shown here, this is an example and is not this limitation. Further, the priority order may be manually set by a switch or the like instead of the priority ordering rule or prior to the rule.

  In step S806, the focusing lens is driven by the defocus amount d2 so as to focus on the person who is given the highest priority by prioritization, and the AF operation is completed. In step S807, the process waits until S2 is turned on by the user, and when S2 is turned on, the operation after step S808 is performed.

  In step S808, an operation of acquiring a defocus map (shooting information) again immediately before shooting is performed. Since the focusing lens is driven to focus on the person in step S806, when the defocus map at this time is acquired, the defocus amount of the person part becomes 0 and the car part is d11, as shown in FIG. 9C. The part of the building changes with d33. In FIG. 9C, the defocus amount of the portion corresponding to the background in the screen is shown as dbb.

  When the defocus amount of each subject has been acquired, the imaging operation is finally performed in step S809, and parameters for obtaining a reconstructed image focused on each subject are generated and added as metadata to the imaging data (parameter generation). means). Regarding the aperture value of the reconstructed image, the aperture value of the parameter to be recorded (imaging information) at the time of completion of imaging is assumed to be F at the time of imaging because the direction of narrowing can be changed later according to the user's intention. In addition, a conversion matrix M that can generate an image focused on each subject is added to the captured image in the order in which the priorities are set in step S805. That is, a conversion matrix M1 (a conversion matrix that generates a reconstructed image with a defocus amount of 0 and an aperture value F) that can reconstruct an image focused on a person is assigned as the first priority. The second rank is a conversion matrix M2 (a conversion matrix that generates a reconstructed image with a defocus amount d11 and an aperture value F) that can reconstruct an image focused on the car. The third place is a conversion matrix M3 (a conversion matrix for generating a reconstructed image with a defocus amount d33 and an aperture value F) that can reconstruct an image focused on a building, and as a result, image metadata as shown in FIG. As given. Since the conversion matrix has a relatively large amount of data, instead of the conversion matrix, only the position of the refocus plane (defocus amount) and aperture value required for generating the conversion matrix are converted into metadata in order of priority. You may remember. In this case, the first position is defocused 0 and aperture value F, the second position is defocus amount d11 and aperture value F, and the third position is defocus amount d33 and aperture value F as parameters. become. When the photographed image data and the refocus metadata generated in association with the subject as described above are recorded, the photographing operation is terminated.

  Although the recording apparatus of the present invention has been described by taking the image pickup apparatus that is a digital camera as an example, the present invention is not limited to this. For example, the image data and metadata may be output from the camera and recorded. Also, only the information for image data and metadata is stored in a removable recording medium at the time of shooting, and the face is detected, priority is set, and tag data is added and recorded by a processing device such as a PC. It is also possible to adopt a configuration.

  In the first embodiment, the configuration in which the parameters for image reconstruction are added to the captured image data and recorded in the metadata has been described. In the second embodiment, there is provided an image reproduction apparatus that enables an editing operation that allows a user to browse reconfigurable image data recorded according to the first embodiment and change the refocus and aperture value at that time. explain. In this embodiment, an example in which a reconstructed image is displayed and edited by an application according to the flowchart of FIG. 10 on a PC having an interface as shown in FIG. Further, it is assumed that the reproduced image data includes image data obtained by photographing the subject shown in FIG. As described in the first embodiment, this image is an image given as metadata of an image as shown in FIG. 1 with a conversion matrix M1 that can reconstruct an image focused on a person as the first place. is there. Similarly, the conversion matrix M2 that can reconstruct an image focused on a car is second, and the conversion matrix M3 that can reconstruct an image focused on a building is third, and is given as image metadata. Yes.

  FIG. 10 is a flowchart of the image reproduction operation according to the second embodiment. In the case of the present embodiment, as described above, this operation is performed by the PC CPU executing an application program to control each part of the PC, for example, the display unit.

  First, in step S1002, it waits for a user to give a reproduction request for image data that can be reconfigured by the user. If there is a reproduction request, the process advances to step S1003 to acquire the image requested for reproduction and its metadata. Here, the image data and the metadata are data recorded by the recording apparatus according to the first embodiment, and are data provided directly from the recording apparatus or via a detachable recording medium.

  The interface of the PC application shown in FIG. 11 is roughly divided into left and right, and the right side is an area 112 for displaying a plurality of reconstructed images. In step S1003, the reconstruction parameters in the metadata are read, and the images reconstructed based on the read reconstruction parameters (conversion matrices M1, M2, M3) are displayed from the top in order of priority. S1005. In this case, the reconstructed image focused on the person who is the subject with the highest priority is displayed at the top of the right side of the screen, and the reconstructed image focused on the car with the second highest priority is further displayed below it. Below that, a reconstructed image focused on the third highest priority building is displayed. When the number of reconstructed images that can be displayed in the reconstructed image display area 112 is limited to n (where n ≧ 1) due to the screen space display, the conversion matrix stored in the metadata Among them, the reconstructed image with the top n is displayed in the display area.

  If the reconstructed image based on the reconstructed parameter stored in the metadata is displayed, the process advances to step S1006 to wait for an edit instruction for changing the refocus or aperture value from the user. If no editing instruction is given in this step, the process advances to step S1007 to end the sequence. On the other hand, when an editing instruction is given, the operation is as follows. That is, the left side of FIG. 11 is an editing area 110 that displays the image being edited. In the editing area, there is an interface for editing the image being played, the refocus plane, and the aperture value. The user can drag the two gauges left and right to instruct the refocus plane and aperture value to be changed. it can. Each time the gauge is operated, a conversion matrix for generating a reconstructed image corresponding to the gauge is calculated, and a reconstructed image after editing is generated and displayed in step S1008. As a result, the user's instruction can be reflected and displayed on the image.

  If the user edits the refocus plane or aperture value and determines that the edited reconstructed image is to be saved, a request is made to save the edited image in step S1009. When there is a storage request, the user is also requested to input (set) the priority order of the edited image (priority setting means), and the conversion matrix generated in step S1008 in step S1011 is determined according to the priority order. Add to the metadata part of the image data. At the same time, the edited image is displayed at a position corresponding to the priority order of the reconstructed image display area 112 in FIG. 11, and the operation of this embodiment is terminated.

  For example, it is assumed that the reconstructed image focused on the person stored in the metadata with the highest priority in the initial stage of image reproduction is strictly an image focused on the ear of the person. On the other hand, it is assumed that the user reconstructs an image focused on a person's eyes by editing, and stores this conversion matrix with the highest priority. In this case, since the reconstructed image focused on the eyes of the person has the highest priority, the priority of the reconstructed image originally stored in the metadata is shifted down by one. As a result, the second priority is the image focused on the person's ear, the third priority is the image focused on the car, and the fourth priority is the image focused on the building. Note that the priority order may be changed even when the user changes the order of the images in the reconstructed image display area 112 or deletes the images. In this case, the latest priority order is recalculated in conjunction with the replacement or deletion, and the conversion matrix for generating the reconstructed image written in the metadata is also rewritten according to the priority order after the recalculation.

  In the present embodiment, the reproduction and editing operation of image data by an application on a PC has been described as an example, but the embodiment of the present invention is not limited to this. For example, it may be realized as a reproduction operation performed by the CPU of the imaging control unit 206 of the imaging apparatus according to the first embodiment executing a program and controlling each unit such as the display unit 207. Needless to say, it may be realized as an image processing function in a device other than a PC having a display function.

  According to the above-described first and second embodiments of the present invention, it is possible to provide image data that can quickly reconstruct and display a refocus image in a desired focused state as recording data. As soon as it becomes possible, the possibility of effective use can be expanded.

  Here, an example of another optical system applicable to the present embodiment will be described with reference to FIG. FIG. 12 is a diagram schematically illustrating a state in which light rays from an object (subject) form an image on the image sensor 206. FIG. 12A corresponds to the optical system described with reference to FIGS. 2 to 4, and is an example in which the MLA 202 is disposed in the vicinity of the imaging surface of the photographing optical system. FIG. 12B shows an example in which the MLA 202 is arranged closer to the object than the imaging plane of the photographing optical system. FIG. 12C shows an example in which the MLA 202 is arranged on the side farther from the object than the imaging plane of the photographing optical system. In the figure, the parts shown in the already described figures are denoted by the same reference numerals.

12, 1200 the object plane, 1200a, 1200 b is an appropriate point on the _object, a 1 ~a ₅ is a pupil plane of the photographing optical system, from 1201 to 1209 are respectively the specific microlens on MLA ing. In FIGS. 12B and 12C, 203a indicates a virtual image sensor, and 202a indicates a virtual MLA. These are shown for reference in order to clarify the correspondence with FIG. Further, the light beams passing through the regions a ₁ and a ₃ on the pupil plane out 1200a point on the object by the solid line, the light beams passing through the regions a ₁ and a ₃ on the pupil plane out 1200b point on the object Is shown by a broken line.

In the example of FIG. 12A, the MLA 203 is disposed in the vicinity of the imaging plane of the photographing optical system, so that the imaging element 203 and the pupil plane 1210 of the photographing optical system are in a conjugate relationship. Furthermore, the object plane 1200 and the MLA 202 are in a conjugate relationship. The microlens 1201 light beam emitted from 1200a point on this for the object, each light beam exiting the 1200b reaches the microlens 1202, the light beam from the region _{a 1} has passed through _{a 5} respectively provided under the microlens The corresponding pixel is reached.

In the example of FIG. 12B, the MLA 202 forms an image of a light beam from the photographing optical system, and the image sensor 203 is provided on the image formation surface. By arranging in this way, the object plane 1200 and the image sensor 203 are in a conjugate relationship. The light flux having passed through the region a ₁ on the pupil plane out 1200a point on the object reaches the microlens 1203, the light flux having passed through the region a ₃ on the pupil plane out 1200a point on the object microlens 1204 To reach. The light flux having passed through the region a ₁ on the pupil plane out 1200b point on the object reaches the microlens 1204, the light flux having passed through the region a ₃ on the pupil plane out 1200b point on the object microlens 1205 To reach. The light beam that has passed through each microlens reaches a corresponding pixel provided under the microlens. In this way, images are formed at different positions depending on the point on the object and the passing area on the pupil plane. If these are rearranged at positions on the virtual imaging surface 203a, the same information as in FIG. 12A can be obtained. That is, information on the pupil region (incident angle) that has passed through and the position on the imaging device can be obtained.

In the example of FIG. 12 (c), the MLA 202 re-images the light beam from the imaging optical system (this is called re-imaging because the imaged light beam once diffused is imaged), An image sensor 203 is provided on the image plane. By arranging in this way, the object plane 1200 and the image sensor 203 are in a conjugate relationship. The light flux having passed through the region a ₁ on the pupil plane out 1200a point on the object reaches the microlens 1207, the light flux having passed through the region a ₃ on the pupil plane out 1200a point on the object microlens 1206 To reach. The light flux having passed through the region a ₁ on the pupil plane out 1200b point on the object reaches the microlens 1209, the light flux having passed through the region a ₃ on the pupil plane out 1200b point on the object microlens 1208 To reach. The light beam that has passed through each microlens reaches a corresponding pixel provided under the microlens. Similar to FIG. 12B, information similar to that in FIG. 12A can be obtained by rearranging the positions on the virtual imaging surface 203a. That is, information on the pupil region (incident angle) that has passed through and the position on the imaging device can be obtained.

  FIG. 12 shows an example in which the position information and the angle information can be acquired using the MLA (phase modulation element) as the pupil dividing means. However, the position information and the angle information (equivalent to limiting the passing area of the pupil) are shown. Other optical configurations can be used as long as they can be obtained. For example, a method of inserting a mask (gain modulation element) with an appropriate pattern into the optical path of the photographing optical system can be used.

  Each of the processes shown in FIGS. 8 and 10 is realized by reading a program for realizing the function of each process from a memory (not shown) and executing it by the CPU of the imaging apparatus as described above. is there.

  However, the present invention is not limited to the above-described configuration, and all or some of the functions shown in FIGS. 8 and 10 may be realized by dedicated hardware. The memory described above may be configured by a computer-readable / writable recording medium. For example, a non-volatile memory such as a magneto-optical disk device or flash memory, a recording medium such as a CD-ROM that can only be read, a volatile memory other than a RAM, or a recording medium composed of a combination thereof. .

  Further, a program for realizing the functions of the processes shown in FIGS. 8 and 10 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. Each process may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. Specifically, this includes a case where a program read from a storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. In this case, after being written, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing. Is done.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” includes a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Thus, what holds a program for a certain period of time is also included in the “computer-readable recording medium”.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  A program product such as a computer-readable recording medium in which the above program is recorded can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and program product are included in the scope of the present invention.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (22)

An image reproduction device for reproducing an image corresponding to image data,
Generating means for generating an image corresponding to a virtual aperture value by reconstructing image data;
Setting means for setting the virtual aperture value;
Display control means for displaying an image corresponding to the virtual aperture value generated by the generation means on a display unit,
The display control means displays an image and a graphic user interface for instructing the virtual aperture value of the image set by the setting means,
The image reproduction apparatus, wherein the display control means simultaneously displays a plurality of images generated by the generation means. The image reproducing apparatus according to claim 1, wherein the image data includes information related to defocusing of a subject in the image data in metadata.   The image reproducing apparatus according to claim 1, wherein the graphic user interface changes the virtual aperture value instructed to the setting unit in response to a drag operation.   The image reproducing apparatus according to claim 3, wherein the graphic user interface changes the virtual aperture value instructed to the setting unit by dragging a gauge on a displayed bar.   The image reproducing apparatus according to claim 4, wherein the bar extends left and right with respect to the displayed image, and indicates a smaller aperture value as the gauge is positioned on the left. The generating unit generates an image corresponding to the virtual aperture value changed according to the distance of the drag operation;
The image reproduction apparatus according to claim 3, wherein the display control unit displays the image generated by the generation unit in accordance with the distance of the drag operation. The generating means generates an image of the virtual aperture value corresponding to the gauge every time the gauge is operated,
The image reproduction apparatus according to claim 4, wherein the display control unit displays the image of the virtual aperture value corresponding to the gauge generated by the generation unit.   The image reproducing apparatus according to claim 1, wherein the graphic user interface displays the image without overlapping the image.   The image reproducing apparatus according to claim 1, wherein the image data is image data captured from a plurality of viewpoints.   The image data is image data captured by an imaging unit having a plurality of microlenses and a plurality of photoelectric conversion elements respectively assigned to the plurality of microlenses. The image reproducing device according to any one of the above.   The said generation | production means produces | generates the image corresponding to the said virtual aperture value by selecting and using the pixel of the said image data according to the said virtual aperture value. The image reproducing device according to any one of the preceding claims.   The image reconstructed and generated by the generation unit is an image focused on one of a plurality of focal positions including a focal position different from a focal position at the time of photographing the image data. The image reproduction device according to any one of 1 to 11.   The generation unit reconstructs the image data and generates an image by combining and adding outputs of corresponding pixels determined by any one of a plurality of focal positions and the virtual aperture value. The image reproducing device according to claim 1.   14. The image reproducing apparatus according to claim 1, wherein the image data includes information on an aperture value at the time of photographing the image data in metadata.   The image reproduction apparatus according to claim 1, wherein the display control unit displays a plurality of images generated by the generation unit side by side.   The display control unit displays the plurality of images generated by the generation unit side by side in accordance with the priority order recorded together with the image data used to generate the plurality of images. The image reproducing device described. An image reproduction device for reproducing an image corresponding to image data,
Generating means for generating an image corresponding to a virtual aperture value by reconstructing image data;
Setting means for setting the virtual aperture value;
Display control means for displaying an image corresponding to the virtual aperture value generated by the generation means on a display unit,
The display control means displays an image and a graphic user interface for instructing the virtual aperture value of the image set by the setting means,
The image reproduction apparatus, wherein the display control means simultaneously displays a plurality of images generated by the generation means in accordance with a priority order recorded together with image data used to generate the plurality of images. A control method of an image playback device for playing back an image corresponding to image data,
A generation step of generating an image corresponding to a virtual aperture value by reconstructing the image data;
A setting step for setting the virtual aperture value;
A display control step of displaying the image corresponding to the virtual aperture value generated in the generation step on a display unit,
In the display control step, an image and a graphic user interface for instructing the virtual aperture value of the image set in the setting step are displayed.
In the display control step, a plurality of images generated in the generation step are simultaneously displayed. 19. The method of controlling an image reproducing device according to claim 18, wherein the image data includes information related to defocusing of a subject in the image data in metadata. A control method of an image playback device for playing back an image corresponding to image data,
A generation step of generating an image corresponding to a virtual aperture value by reconstructing the image data;
A setting step for setting the virtual aperture value;
A display control step of displaying the image corresponding to the virtual aperture value generated in the generation step on a display unit,
In the display control step, an image and a graphic user interface for instructing the virtual aperture value of the image set in the setting step are displayed.
In the display control step, the plurality of images generated in the generation step are simultaneously displayed according to the priority order recorded together with the image data used to generate the plurality of images. Control method.   21. A computer-executable program in which a procedure of a control method for an image reproduction device according to claim 18 is described.   21. A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an image reproduction device according to claim 18.

Images