JP5310895B2 - Image generating apparatus, image generating method, and program - Google Patents

Description

  The present invention relates to an image generation apparatus, an image generation method, and a program.

A technique for executing composition conversion such as deleting unnecessary subjects after taking a picture is known.
In relation to such a technique, Patent Document 1 discloses a method of performing interpolation in order to suppress a sense of incongruity when an image region is designated and an image composition is changed. In the technique described in Patent Document 1, the gap caused by the composition change is corrected using the pixel values of surrounding pixels.

  In addition, there is known a technique for acquiring information related to the position, direction, and amount of light entering a lens of a camera from a subject by acquiring images of the subject taken from different viewpoints. In relation to such a technique, Patent Document 2 acquires a plurality of images (light field images) obtained by photographing a subject, and images of the subject in which focal length, depth of field, and the like are changed from the plurality of images. Is disclosed.

JP-T-2005-509985 Special table 2008-515110 gazette

The information on the subject included in the image taken from one viewpoint is all the position where the light beam reaches and the amount of light (corresponding to the coordinates of each pixel and its pixel value, respectively). For this reason, if a subject (unnecessary subject) that gets in the way of such a planar image is deleted so as not to interfere with viewing, even if correction is performed using the technique described in Patent Document 1, it is possible to remove the subject from the subject. Interpolation that is not based on ray information is required. For this reason, the changed image is greatly different from the case where the unnecessary subject is actually erased and photographed. That is, there is a problem that the image after erasure does not reflect information from the subject.
On the other hand, Patent Document 2 discloses a technique for obtaining information indicating the three-dimensional shape of a subject, light ray information from a subject that is hidden from a certain viewpoint, and the like by photographing the subject from a plurality of viewpoints. Yes. However, Patent Document 2 does not disclose a method for generating a reconstructed image in which some subjects are deleted, and does not solve the above-described problem.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image generation apparatus, an image generation method, and a program for generating a reconstructed image that more accurately reflects photographing information of a main subject.

In order to achieve the above object, an image generation apparatus according to the present invention includes:
And shoot each one these multiple viewpoints, an image acquisition section which acquires a captured image including a plurality of sub-images,
Sub pixels constituting the sub-image, an extraction unit for extracting a sub-pixel corresponding to the deletion object to be deleted as a deletion pixel,
The correspondence relationship between the reconstructed pixels constituting the reconstructed image defined on the predetermined reconstructed surface and the sub-pixels is changed so that the degree of correspondence between the sub-pixels extracted as the deleted pixels is reduced. Change part,
A generation unit that generates a reconstructed image with a small influence of the deleted pixel by obtaining a pixel value of the reconstructed pixel from a pixel value of the sub-pixel using the changed correspondence relationship ;
It is characterized by providing.

  According to the present invention, it is possible to generate a reconstructed image that more reflects the shooting information of the main subject.

It is a figure which shows the structure of the digital camera which concerns on Embodiment 1 of this invention. 1 is a diagram illustrating a configuration of an optical system of a digital camera according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating an example of a light field image according to the first embodiment, where (a) is a conceptual diagram of the light field image, (b) is an example of the light field image, and (c) is an example of the light field depth map. Show. FIG. 6 is a diagram for explaining ray tracing according to the first embodiment. It is a figure which shows the example of the corresponding | compatible list which concerns on Embodiment 1. FIG. It is a figure which shows (a) physical structure of the image generation apparatus which concerns on Embodiment 1, and (b) functional structure. FIG. 6 is a diagram for explaining an operation for instructing deletion according to the first embodiment, where (a) is a provisional reconstructed image, (b) is an operation for selecting a deletion subject, and (c) is a deletion target. Operation (d) shows a reconstructed image after complete deletion, and (e) shows a reconstructed image after deletion at a fixed ratio. FIG. 5 is a diagram for explaining a reconstructed image and a reconstructed depth map according to the first embodiment, where (a) is a temporary reconstructed image, (b) is a temporary reconstructed depth map, and (c) is a deleted reconstructed image. (D) shows a deletion reconstruction depth map, respectively. FIG. 6 is a diagram for explaining processing for selecting a deletion target according to the first embodiment, where (a) shows a deletion operation, (b) shows a selection pixel list used for selection, and (c) shows a selection result. FIG. FIG. 6 is a diagram for explaining a deletion process according to the first embodiment, where (a) shows an LFI before deletion, (b) shows a temporary reconstructed image, (c) shows an LFI after deletion, and (d) shows an LFI after deletion. Deletion reconstructed images are shown respectively. 3 is a flowchart illustrating image output processing according to the first embodiment. 6 is a flowchart illustrating provisional reconstructed image generation processing according to the first embodiment. 6 is a flowchart illustrating a deletion target definition process according to the first embodiment. 6 is a flowchart illustrating a deletion depth definition process according to the first embodiment. 6 is a flowchart illustrating a deleted pixel extraction process according to the first embodiment. 6 is a flowchart illustrating a deletion reconstructed image generation process according to the first embodiment. It is a figure which shows the example of the class-depth corresponding | compatible list which concerns on other embodiment of this invention. It is a figure which shows the example of the selection pixel list | wrist which concerns on other embodiment of this invention.

  Hereinafter, a digital camera and an image generation apparatus (reconstructed image generation apparatus) according to an embodiment for carrying out the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

(Embodiment 1)
An image generation apparatus 30 (reconstructed image generation apparatus) according to Embodiment 1 is mounted on the digital camera 1 shown in FIG. The digital camera 1 has the following functions i) to vi).
i) Function for photographing a light field image composed of a plurality of sub-images obtained by photographing the subject from a plurality of viewpoints ii) Function for obtaining a depth coefficient indicating the depth of the subject iii) Reconstruction by reconstructing the subject image from the light field image A function for generating an image iv) A function for displaying a generated reconstructed image v) A function for receiving an operation for instructing a subject to be deleted (deleted subject) vi) A reconstructed image whose composition has been changed according to the received operation Function to Generate The image generation apparatus 30 is particularly responsible for vi) a function for generating a reconstructed image whose composition has been changed in accordance with an accepted operation.

As shown in FIG. 1, the digital camera 1 includes an imaging unit 10, an information processing unit 20 including an image generation device 30, a storage unit 40, and an interface unit (I / F unit) 50. With such a configuration, the digital camera 1 acquires ray information of a subject from the outside and generates a reconstructed image. At this time, a reconstructed image is generated such that the subject to be deleted (deleted subject) among the subjects is deleted so as not to obstruct the viewing of other subjects (main subjects).
Here, “deleting the deleted subject” means not only removing the deleted subject completely, but also reducing the effect of the ray information from the deleted subject on the reconstructed image to a level that does not interfere with the viewing of the main subject. To include.

  The imaging unit 10 includes an optical device 110 and an image sensor 120, and performs an imaging operation.

  As shown in FIG. 2, the optical device 110 includes a shutter 111, a main lens ML, and a sub lens array SLA (micro lens array). The optical device 110 captures light rays from the outside (subject) by the main lens ML, and projects an optical image obtained from the viewpoint of the optical center of each sub lens SL constituting the sub lens array SLA on the image sensor 120.

  The image sensor 120 includes, for example, an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and a transmission unit that transmits an electric signal generated by the image sensor to the information processing unit 20. . The image sensor 120 converts the optical image projected by the optical device 110 into an electrical signal with such a physical configuration and transmits the electrical signal to the information processing unit 20.

The shutter 111 controls the incidence and shielding of external light on the image sensor 120.
The main lens ML is composed of one or a plurality of convex lenses, concave lenses, aspherical lenses, and the like, and a virtual main lens connection between the main lens ML and the sub lens array SLA with the light of the subject OB at the time of photographing as an optical image. An image is formed on the image plane MIP. Note that the subject OB at the time of shooting is assumed to be a plurality of components that are separated from the main lens ML by different distances as shown in FIG.

  The sub lens array SLA includes M × N sub lenses (micro lenses) SL arranged in a lattice pattern on a plane. The sub-lens array SLA is an image constituting the image sensor 120 by using an optical image formed by the main lens ML on the main lens imaging plane MIP as an optical image observed from the viewpoint of the optical center of each sub-lens SL. An image is formed on the imaging surface IE of the sensor. A space formed by a plane formed by the main lens ML and a plane formed by the imaging surface IE is referred to as a light field.

For the main lens ML, a maximum diameter LD and an effective diameter ED can be defined. The maximum diameter LD is the physical diameter of the main lens ML. On the other hand, the effective diameter ED is a diameter of a region that can be used for photographing in the main lens ML. Of the main lens ML, the outside of the effective diameter ED captures and reconstructs an image because light beams input to and output from the main lens are blocked by various filters attached to the main lens ML and the physical structure around the main lens ML. This is an area that is not effective for the purpose (non-effective area).
The maximum diameter LD and the effective diameter ED are measured in advance and stored in the storage unit 40 at the time of factory shipment.

In the example of FIG. 2, among a plurality of subjects OB (subjects OB1 to OB3), a light beam from a portion POB where the subject OB2 is located passes through a portion (effective portion) that forms an effective diameter ED of the main lens ML. Projected onto the sub-lens SL. In this way, an area where light emitted from a portion POB of a certain subject OB passes through the effective portion of the main lens ML and is projected onto the sub-lens array SLA is referred to as a main lens blur MLB of the portion POB. Among these, the part where the chief ray reaches is called the main lens blur center MLBC.
Hereinafter, a plurality of subjects will be referred to as subjects OB1 to OB3 in order from the farthest (the distance from the main lens is large).

  The distance from the optical center of the main lens ML to the imaging plane MIP of the main lens is b1, the distance from the imaging plane MIP to the plane formed by the sub lens array SLA is a2, and the imaging plane IE of the image sensor 120 from the sub lens array SLA. Let c2 be the distance.

With the above-described configuration, the imaging unit 10 captures a light field image (LFI) including information on the light rays passing through the light field (arrival site, light amount, direction).
FIG. 3A shows an example of the LFI obtained by shooting the block-shaped subject OB.
This LFI is composed of images (sub-images SI, S _{11 to} S _MN ) corresponding to each of M × N sub-lenses SL (microlenses) arranged in a lattice pattern. For example, the upper left subimage S ₁₁ corresponds to an image obtained by photographing the object OB from the upper left, the sub-image S _MN the lower right corresponds to an image obtained by photographing the object OB from the lower right.

Each sub image is arranged at a position on the LFI corresponding to the position of the sub lens on which the sub image is formed.
The i-th row sub-images (horizontal row sub-images) Si1 to SiN are stereo images in which the image formed by the main lens ML is formed by the sub-lens SL arranged side by side in the i-th row of the sub-lens array SLA. Corresponds to an image. Similarly, in the j-th row sub-images (vertical one-row sub-images) S1j to SMj, the images formed by the main lenses ML are arranged vertically in the j-th column of the sub-lens array SLA (microlens array). This corresponds to a stereo image taken with the sub lens SL. In the present embodiment, each sub image is a gray scale image, and each pixel constituting the sub image has a pixel value (scalar value).

  The information processing unit 20 shown in FIG. 1 is physically composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), an internal bus, and an I / O port. The information processing unit 20 functions as the image processing unit 210, the depth estimation / correspondence definition unit 220, the image generation device 30, and the imaging control unit 230 with such a physical configuration.

The image processing unit 210 acquires an electrical signal from the image sensor 120 and converts the acquired electrical signal into image data based on imaging setting information stored in the imaging setting storage unit 410 of the storage unit 40. The image processing unit 210 transmits the image data and shooting setting information obtained by adding predetermined information to the shooting setting information to the depth estimation / correspondence definition unit 220.
The imaging setting information stored in the imaging setting storage unit 410 will be described later.

  Upon receiving the LFI and the shooting setting information, the depth estimation / correspondence definition unit 220 estimates the depth of the subject for each pixel on the sub-image on the LFI. At this time, the depth of the subject is calculated to the extent that the pixel corresponding to the subject is shifted on each sub-image. The size of the deviation is a coefficient indicating the depth. Then, information (light field depth map, LFDM) in which a coefficient indicating the estimated depth is arranged at the position of each pixel is generated. It is estimated that the subject reflected in the pixel is closer to the front (position closer to the main lens ML) as the depth coefficient is larger.

The LFDM defines a coefficient indicating the depth for each of the sub-pixels included in the LFI. FIG. 3C shows the LFDM for the LFI shown in FIG. In the LFDM, the deeper part (the back side of the cuboid) is darker and the front (front) is lighter than the LFI obtained by photographing the cuboid subject as shown in FIG. It is shown. Hereinafter, similarly, a deeper part in the depth map is indicated by dark pixels, and a front part is indicated by thin pixels.
Here, LFI and LFDM have been described as separated information, but LFI and LFDM are pixel values (information held by LFI) and depth coefficients (information held by LFDM) for pixels arranged at certain coordinates. ) May be recorded as one piece of data.
The LFDM can be generated using any known method for estimating the depth of each pixel of a multi-viewpoint image. In the present embodiment, it is assumed that the method described later is used.

  The depth estimation / correspondence definition unit 220 further obtains correspondence information that defines the correspondence between the pixels (reconstructed pixels) constituting the reconstructed image (temporary reconstructed image) before the composition change and the sub-pixels. The correspondence information is information indicating a sub-pixel (corresponding sub-pixel) in which the subject at the position of the reconstructed pixel is shown in the reconstructed image arranged on the reconstruction plane indicated by the reconstruction setting.

The depth estimation / correspondence definition unit 220 may generate correspondence information by any known method for extracting the corresponding sub-pixel of the reconstructed pixel (target pixel), but in the present embodiment, the depth estimation / correspondence definition unit 220 extracts it by the following ray tracing.
A method of ray tracing will be described with reference to FIG. The light beam from the target portion P (corresponding to the target pixel) of the subject passes through the main point of the main lens and reaches the arrival position of the microlens array (MLBC on the sub lens in FIG. 4). The position on the sub lens of the MLBC can be obtained based on the shooting setting. The range (main lens blur MLB, mesh area in FIG. 4) in which light from the target region reaches centered on MLBC is obtained from the lens characteristics. The diameter of the main lens blur MLB the distance between the main lens ML and the reconstruction plane RF a1, (calculated from the focal length _{f ML} of a1 and the main lens) distance b1 between the main lens and the image plane MIP, imaging plane MIP And the distance a2 between the sub lens array SLA and the effective diameter ED of the main lens.

  Next, among the sub lenses SL included in the sub lens array SLA, a part or all of the sub lenses SL are specified in the main lens blur MLB. Then, the identified sub lenses SL are sequentially selected as the target lens. At this time, the area w of the portion where the target lens and the main lens blur overlap is obtained from the position of the center of the main lens blur, the diameter of the main lens blur MLB, and the position and size of the sub lens defined by the shooting setting information.

  A pixel (corresponding pixel) on the sub-image at a position where the light beam from the target pixel is imaged by the selected sub-lens is extracted.

Specifically, the corresponding pixel (corresponding to the reaching point PE) is calculated according to the following procedure.
First, the distance b1 to the focal plane of the main lens corresponding to the reconstruction surface RF can be calculated from the following equation (1) using known numerical values a1 and _fML .

Further, a2 can be obtained by subtracting b1 calculated using the equation (1) from the known numerical value c1.
Further, the distance a1 between the reconstruction surface RF and the main lens, the distance b1 between the main lens imaging surface MIP and the sub lens, and a known numerical value x (distance between the target portion P and the optical axis OA) are expressed by the following equations: In (2), a distance x ′ between a point (imaging point PF) that forms an image from the target site P through the main lens ML and the optical axis OA is calculated.
x ′ = x · b1 / a1 (2)
Furthermore, the distance d from the optical axis OA to the principal point of the sub lens SL of interest, the distance x ′ calculated using the above equation (2), the distance c2 from the microlens array LA to the imaging surface IE, and the distance a2 Is used in the following equation (3) to calculate the distance x ″ between the arrival point PE and the optical axis OA.

  The same calculation is performed in the Y-axis direction, and the position of the reaching point PE is specified. Then, the pixel on the sub-image corresponding to the reaching point PE is set as the corresponding pixel. Then, the area w obtained for the target lens is set as the weight of the target pixel and the corresponding pixel.

FIG. 5 shows a correspondence list which is an example of correspondence information. The correspondence list in FIG. 5 includes coordinates (xr, yr) on the reconstructed image of the reconstructed pixel, and coordinates (xs, ys) on the LFI of the sub-pixel extracted by ray tracing for the reconstructed pixel at that coordinate. The weight (w) is recorded in association with it. A plurality of sub-pixels are extracted for one reconstructed pixel. For example, in the example of FIG. 5, (xs (1,1), ys (1,1)) to (xs (1, k ₁ ), ys are set for the first reconstructed pixel (coordinate (1,1)). K1 sub-pixels located at (1, k ₁ )) are extracted. The corresponding strengths (weights) of the extracted sub-pixels and reconstructed pixels are w (1,1) to w (1, k ₁ ).
The depth estimation / correspondence definition unit 220 transmits such correspondence information, LFI, and LFDM to the image generation apparatus 30.

The image generation device 30 uses the LFI and LFDM transmitted from the image processing unit 210 to generate an image (deleted reconstructed image) reconstructed after deleting the deleted subject.
The image generation device 30 stores the generated deletion reconstructed image in the image storage unit 430 of the storage unit 40. The configuration of the image generation device 30 and the processing for generating the deleted reconstructed image will be described later.

  The imaging control unit 230 controls the imaging unit 10 based on the imaging setting information stored in the imaging setting storage unit 410 of the storage unit 40 and images the subject OB using the imaging unit 10.

The storage unit 40 includes a main storage device including a RAM and the like, and an external storage device including a nonvolatile memory such as a flash memory and a hard disk.
The main storage device loads control programs and information stored in the external storage device and is used as a work area of the information processing unit 20.
The external storage device stores in advance a control program and information for causing the information processing unit 20 to perform processing to be described later, and transmits these control program and information to the main storage device in accordance with instructions from the information processing unit 20. Further, in accordance with an instruction from the information processing unit 20, information based on the processing of the information processing unit 20 and information transmitted from the interface unit 50 are stored.

  Functionally, the storage unit 40 includes an imaging setting storage unit 410, a reconstruction setting storage unit 420, and an image storage unit 430.

The imaging setting storage unit 410 stores imaging setting information. The imaging setting information includes a distance between the main lens ML and the sub lens array SLA, a focal length f _ML of the main lens, information for specifying an exposure time, an F value, a shutter speed, and the like as imaging parameters that can change during imaging. In addition, the imaging setting storage unit 410 stores information on the physical configuration of the digital camera 1 such as the position of each sub lens SL on the sub lens array SLA, the distance c2 between the sub lens array SLA and the imaging surface IE, and the like. .
The imaging setting storage unit 410 transmits imaging parameters to the imaging control unit 230.
Further, the imaging setting storage unit 410 adds information related to the physical configuration to the imaging setting information obtained by imaging the LFI by the imaging unit 10 and transmits the information to the image processing unit 210 as imaging setting information.

The reconstruction setting storage unit 420 stores setting parameters for generating an image by reconstructing from the LFI, which is stored at the time of initial shipment or input by the user using the operation unit 530.
The reconfiguration setting information includes information indicating the specific contents of the reconfiguration process and reconfiguration parameters. In the present embodiment, the reconstruction information is generated by displaying a temporary reconstructed image that is not deleted from the LFI, displayed, and information indicating that a deleted reconstructed image is generated based on an operation on the temporary reconstructed image, Information specifying the distance between the focus point of the image and the main lens ML.

  The image storage unit 430 stores output images (temporary reconstructed images and deleted reconstructed images) generated by the image generating device 30. The image storage unit 430 transmits the stored image to the I / O unit 510 and the display unit 520 of the interface unit 50.

  An interface unit (described as I / F unit in the figure) 50 is a configuration relating to an interface between the digital camera 1 and its user (user) or an external device, and includes an I / O unit 510, a display unit 520, And an operation unit 530.

  The I / O unit (Input / Output unit) 510 is physically composed of a USB (Universal Serial Bus) connector, a video output terminal, and an input / output control unit. The I / O unit 510 outputs the information stored in the storage unit 40 to an external computer, and transmits the information transmitted from the outside to the storage unit 40.

  The display unit 520 includes a liquid crystal display device, an organic EL (Electro Luminescence) display, and the like, and has a screen for inputting imaging parameters stored in the imaging setting storage unit 410 and a screen for operating the digital camera 1. indicate. Further, the display unit 520 displays the image stored in the image storage unit 430.

  The operation unit 530 detects, for example, various buttons provided in the digital camera 1 and a touch panel provided in the display unit 520, information on operations performed on the various buttons and the touch panel, and the storage unit 40, the information processing unit 20, and the like. The information on the user operation is transmitted to the storage unit 40 and the information processing unit 20.

Next, the configuration of the image generation device 30 will be described with reference to FIG.
As shown in FIG. 6A, the image generation device 30 is physically composed of an information processing unit 31, a main storage unit 32, an external storage unit 33, an input / output unit 36, and an internal bus 37. Is done.

  The information processing unit 31 includes a CPU and a RAM.

  The main storage unit 32 has the same physical configuration as the main storage device of the storage unit 40. The external storage unit 33 has the same physical configuration as the external storage device of the storage unit 40 and stores a program 38. The input / output unit 36 includes input / output terminals and I / O devices, and realizes input / output of information between the image generation apparatus 30 and each unit of the information processing unit 20, the storage unit 40, the interface unit 50, and the like. The internal bus 37 connects the information processing unit 31, the main storage unit 32, the external storage unit 33, and the input / output unit 36.

  The information processing unit 31, the main storage unit 32, the external storage unit 33, the input / output unit 36, and the internal bus 37 are an internal circuit of the information processing unit 20 of the digital camera 1, a storage unit 40, and an interface unit. 50 may be a functional block realized by.

  The image generation device 30 copies the program 38 and data stored in the external storage unit 33 to the main storage unit 32, and the information processing unit 31 executes the program 38 using the main storage unit 32. Processing for generating a deletion reconstructed image to be described later is executed.

  As shown in FIG. 6B, the image generation device 30 has the above-described physical configuration as an input unit 310, a reconstructed image generation unit 320, a deletion operation acquisition unit 330, and an output unit 340. Function.

The input unit 310 is a part in charge of the function of the image generating apparatus 30 to acquire information from each unit of the digital camera 1. The input unit 310 includes a reconfiguration setting acquisition unit 3110, an LFDM acquisition unit 3120, an LFI acquisition unit 3130, and a correspondence acquisition unit 3140.
The LFDM acquisition unit 3120 acquires the LFDM from the depth estimation / correspondence definition unit 220.

The reconfiguration setting acquisition unit 3110 acquires the reconfiguration setting information from the reconfiguration setting storage unit 420. The reconstruction setting information includes information indicating specific contents of the reconstruction process and information necessary for the image generation process described below (information indicating a reconstruction parameter and a threshold). In this embodiment, it is assumed that the reconstruction parameter defines that a reconstructed image is generated by the following process.
i) A temporary reconstructed image arranged on the reconstruction surface RF is generated using the correspondence information.
ii) A temporary reconstructed image is displayed and a user's operation for designating a deletion subject is accepted.
iii) Based on the deletion subject designation operation, an image from which the deletion subject has been deleted is reconstructed and output.
For this purpose, the reconstruction parameter is information for identifying the distance (reconstruction distance, distance a1) between the focus point (reconstruction plane) of the new image and the main lens ML, and the pixel corresponding to the deleted subject. Includes information such as configuration parameters.

The LFI acquisition unit 3130 acquires the LFI generated by the image processing unit 210.
The correspondence acquisition unit 3140 acquires correspondence information (correspondence list in FIG. 5) from the depth estimation / correspondence definition unit 220.
Each unit of the input unit 310 transmits the acquired information to each unit of the reconstructed image generation unit 320.

  The deletion operation acquisition unit 330 receives information indicating an operation for designating a deletion subject from the operation unit 530 of the digital camera 1. The deletion operation acquisition unit 330 transmits information indicating the received deletion subject designation operation to the reconstructed image generation unit 320.

The reconstructed image generation unit 320 includes the LFDM acquired by the LFDM acquisition unit 3120 and the LFI acquisition unit 3130 according to the reconfiguration setting acquired by the reconfiguration setting acquisition unit 3110 and the deletion operation information acquired by the deletion operation acquisition unit 330. A deletion reconstructed image is generated from the acquired LFI and the correspondence information acquired by the correspondence acquisition unit 3140.
In the present embodiment, a temporary reconstructed image (RI1) that is a reconstructed image whose composition has not been changed is once generated and output. When the user performs a deletion subject specifying operation using the temporary reconstruction image, a deletion reconstruction image (RI2) that is a reconstruction image excluding the deletion subject is generated. In addition, a reconstructed depth map (RDM) that defines the depth of each pixel is generated.

  An overview of processing executed by the reconstructed image generation unit 320 will be described with reference to FIG. The reconstructed image generation unit 320 reconstructs the temporary reconstructed image RI1 that is not subjected to the deletion process from the LFI, and outputs the reconstructed image RI1 to the display unit 520. In the example of FIG. 7A, RI1 is an image in which a background (white), a child (gray) existing in the vicinity of the focal plane RF, and an obstacle (hatched portion) existing in the foreground appear.

  Then, it is desired that the user who has watched RI1 displayed on the display unit 520 deletes the obstacle in the foreground so as not to disturb the child's image. At this time, the user first selects a deletion subject by an operation such as tracing an obstacle on the touch panel (the display unit 520 and the operation unit 530) as shown in FIG. 7B.

  Based on such a selection operation, the reconstructed image generation unit 320 extracts the deleted subject, highlights it, and displays it (FIG. 7C). The user confirms whether the currently highlighted subject matches the desired deletion target, and if it matches, touches the confirm button (YES in FIG. 7C) to confirm the subject to be deleted. To do. On the other hand, if different from the desired object, touch NO. In this case, the selection is once reset and the user redoes the selection.

  When the subject to be deleted is determined, the reconstructed image generation unit 320 generates and displays an image from which the deleted subject has been deleted (deleted reconstructed image, RI2). FIG. 7D shows an example of the deleted reconstructed image when the deleted subject is completely deleted. On the other hand, FIG. 7 (e) shows an example of an image reconstructed image when the influence of information from the deleted subject is reduced and a fixed ratio is deleted.

  In order to execute such processing, the reconstructed image generation unit 320 includes a temporary reconstructed image generation unit 3210, a selected pixel extraction unit 3220, a deletion depth definition unit 3230, a deletion pixel extraction unit 3240, and a deletion unit 3250. And a deleted reconstructed image generation unit 3260.

The temporary reconstructed image generation unit 3210 defines a reconstructed image on the reconstructed surface RF. Then, the pixel value of the pixel on the reconstructed image (reconstructed pixel) and its depth coefficient are determined as follows to generate a temporary reconstructed image (RI1) and its reconstructed depth map (RDM1). .
(1) Using one of the reconstructed pixels as a target pixel, the corresponding subpixel of the reconstructed pixel corresponding to the coordinates of the target image and its weight are acquired with reference to the correspondence information.
(2) The pixel value of the corresponding pixel is acquired with reference to the LFI.
(3) A corrected pixel value is obtained by multiplying the acquired pixel value by a weighting coefficient.
(4) The corrected pixel values are calculated for all the extracted corresponding pixels, and the sum is used as the pixel value of the target pixel. At this time, normalization may be performed in which the sum of pixel values is divided by the sum of weights.
(5) The depth coefficient of the extracted corresponding pixel is acquired with reference to LFDM.
(6) The mode value of the acquired depth coefficient is set as the depth coefficient of the corresponding part of the reconstructed depth map.
(7) Perform (1) to (6) for each reconstructed pixel to determine the pixel value and the depth coefficient.

The temporary reconstructed image generation unit 3210 transmits the generated temporary reconstructed image (RI1) and the reconstructed depth map (RDM1) to the selected pixel extraction unit 3220.
Examples of RI1 and RDM1 are shown in FIG. FIG. 8A shows an example of the temporary reconstructed image RI1. FIG. 8B shows a depth map RDM1 of RI1. A portion (subject in front) indicated by a solid line on RI1 in FIG. 8A is a portion (white) having a large depth coefficient on RDM1.
On the other hand, the part indicated by the alternate long and short dash line on RI1 corresponds to the deeper subject (OB2). This site is shown in gray on RDM1.
The portion of the subject (background, OB1) indicated by the dotted line on RI1 has a small depth coefficient. This site is shown in a darker color in RDM1.
In the deleted reconstructed image RI2 (FIG. 8C) from which the subject OB2 has been deleted, the subject OB1 ′ that is hidden behind OB2 appears on RI1. In RDM2 corresponding to RI2 (FIG. 8D), the portion corresponding to OB2 is black indicating the background.

  The temporary reconstructed image generation unit 3210 transmits the generated temporary reconstructed image (RI1) to the output unit 340 for output. Also, RI1 and RDM1 are transmitted to the selected pixel extraction unit 3220. The output unit 340 stores RI1 in the image storage unit 430 and displays the RI1 on the display unit 520.

  Display unit 520 presents RI1 to the user, and operation unit 530 receives an operation for selecting a subject on RI1 from the user. The delete operation acquisition unit 330 acquires the received operation.

The deletion operation acquiring unit 330 transmits information (coordinates of the designated part) indicating the operation for designating the part on the RI1 to the selected pixel extracting part 3220.
The selected pixel extraction unit 3220 extracts a reconstructed pixel (selected pixel) corresponding to the part designated by the user from RI1. Specifically, the coordinates (x, y) on the RI1 of the part designated by the user are acquired. Further, the depth value of the selected pixel is acquired. The acquired information is held as selection information.

An example of selection information acquired by the selected pixel extraction unit 3220 will be described with reference to FIG. In the example of FIG. 9A, a background (OB1) corresponding to a depth coefficient (depth index) of 1, a subject (OB2a, OB2b) corresponding to a depth index of 3, and a subject OB3 corresponding to 6. It is reflected in RI1.
When the user performs a touch operation as indicated by a black arrow in RI1, the selected pixel extraction unit 3220 generates a selected pixel list shown in FIG. 9B. The selected pixel list records the number of the selected pixel, the coordinates of the selected pixel, and the depth coefficient (depth index) of the selected pixel in association with each other. Note that the selected pixel is a reconstructed pixel corresponding to a part designated by the user through a touch operation. The coordinates of the selected pixel are coordinates on the reconstructed image of the part touched by the user. The depth coefficient of the selected pixel is the depth coefficient of RDM1 corresponding to the coordinates of the selected pixel.

The selected pixel extracting unit 3220 transmits the generated selected pixel list to the deletion depth defining unit 3230.
The deletion depth definition unit 3230 determines a depth index satisfying a predetermined condition among the depth indexes included in the transmitted selection pixel list as a depth coefficient (deletion depth coefficient) corresponding to the deleted subject.

The deletion depth coefficient may be determined by an arbitrary method such as a depth index having a number exceeding a predetermined number in the selected pixel list. Here, the following method is used. Specifically, if the number of selected pixels included in the selected pixel list is L and the number of depth indexes of a certain numerical value d is Ld, the ratio Qd is calculated using the following equation (5). Of the depth indexes included in the selected pixel list, those having a Qd exceeding a predetermined threshold are set as deletion depth coefficients. This threshold value is defined by the reconfiguration setting information.
Qd = Ld / L (5)

In the examples of FIGS. 9A and 9B, depth indices 3 and 6 are selected as the deletion depth coefficients. Therefore, the deletion depth definition unit 3230 deletes pixels having a depth index different from the deletion depth coefficient from the selected pixel list.
The deletion depth definition unit 3230 transmits the selected pixel list that has been deleted and information indicating the defined deletion depth coefficient to the deletion pixel extraction unit 3240.

The deleted pixel extraction unit 3240 compares the depth coefficient of the sub-pixel with the deleted depth coefficient, and extracts a deleted pixel that can be determined that the deleted subject is reflected based on the comparison result from the sub-pixel. The process of extracting the deleted pixel will be described later.
Then, the deleted pixel extraction unit 3240 transmits a temporary reconstructed image in which the reconstructed pixel corresponding to the extracted deleted pixel is emphasized to the output unit 340. The user can determine whether a desired subject has been selected using the image output at this time.
In the example of FIG. 9C, subjects (OB2a and OB3) corresponding to pixels having depth indices 3 and 6 are emphasized as deletion subjects and presented to the user based on the operation of FIG. 9A. . The depth index of OB2b corresponds to the deletion depth coefficient, but does not include the part specified by the user, so OB2b is not regarded as the deletion subject.

  When the user performs an operation of confirming the output image and confirming the deletion subject, the deletion pixel extraction unit 3240 transmits information indicating the extracted deletion pixel (coordinates on the LFI of the deletion pixel) to the deletion unit 3250.

The deletion unit 3250 deletes the deleted pixel from the LFI. Specifically, the pixel value of the pixel of the coordinate (deleted pixel) transmitted from the deleted pixel extraction unit 3240 on the LFI is set to NULL. Pixels with a pixel value of NULL are excluded from the calculation target when generating a deletion reconstructed image to be described later.
In the present embodiment, setting the pixel value of the deleted pixel to NULL is equivalent to not treating it as a corresponding pixel of the reconstructed pixel in the following processing. That is, it is synonymous with deleting the deleted pixel from the correspondence list (or setting the correspondence weight to 0).

  An example of processing for deleting the deleted pixel from the LFI will be described with reference to FIG. 10 by taking as an example a case where a plate-like subject OB1, a cube-like subject OB2, and a cylindrical subject OB3 are photographed. 10A shows a part of the LFI acquired by the LFI acquisition unit 3130, and FIG. 10B shows the temporary reconstructed image reconstructed by the temporary reconstructed image generation unit 3210. In the LFI, the image of each subject appears point-symmetrically.

  The LFI is composed of a plurality of sub-images obtained by photographing OB1, OB2, and OB3 from different angles. In FIG. 10, the sub-images are shown as squares arranged in a grid pattern. Further, the center of each sub-image is indicated by a one-dot chain line.

When deleting OB2, the deletion depth definition unit 3230 extracts the depth coefficient of OB2 as the deletion depth coefficient. Then, the deleted pixel extraction unit 3240 extracts sub-pixels (pixels in the horizontal line portion of FIG. 10A) included in the predetermined range from the deletion depth coefficient among the corresponding pixels corresponding to the coordinates of OB2 on the reconstructed image. . Then, the deletion unit 3250 sets the extracted pixel value of the sub-pixel to NULL. FIG. 10C shows a part of the deleted LFI (deleted LFI, DLFI) in which the NULL pixels are shown in black.
When the image is reconstructed from the deletion LFI using the correspondence list, a deletion reconstructed image (RI2) from which the deletion subject (OB2) is deleted can be generated as shown in FIG. At this time, OB1 is displayed in the portion where OB2 was present. This is because OB1 is shown in the corresponding pixels of the reconstructed pixel in the part where OB2 is present, in addition to the deleted pixel.

  The deletion unit 3250 transmits the deletion LFI to the deletion reconstructed image generation unit 3260. The deleted reconstructed image generation unit 3260 weights and adds the pixel values of the pixels whose pixel values are not set to NULL on the deleted LFI among the corresponding subpixels registered in the corresponding list for each of the reconstructed pixels. A reconstructed image (RI2) is generated.

  The deletion reconstructed image generation unit 3260 transmits the generated deletion reconstructed image to the output unit 340. The output unit 340 stores the transmitted deletion reconstructed image in the image storage unit 430. Then, the digital camera 1 displays the deleted reconstructed image on the display unit 520. Alternatively, it is transmitted to the outside via the I / O unit 510.

  Next, processing executed by the digital camera 1 will be described with reference to a flowchart. The digital camera 1 starts the image output process shown in FIG. 11 in response to the user operating the operation unit 530 and instructing photographing of the subject.

  In the image output process, first, the LFI is generated from the image information acquired by the image processing unit 210 from the imaging unit 10 and the shooting setting information acquired from the imaging setting storage unit 410 (step S101). Then, the generated LFI and shooting setting information are transmitted to the depth estimation / correspondence definition unit 220.

  Next, the depth estimation / correspondence definition unit 220 acquires shooting setting information. Further, reconfiguration setting information is acquired from the reconfiguration setting storage unit 420 (step S102). The imaging settings and reconstruction settings acquired at this time are also transmitted to the image generation apparatus 30.

Then, the depth estimation / correspondence definition unit 220 estimates the depth for each sub-pixel constituting the LFI, and the image generation apparatus 30 acquires the LFDM in which the depth coefficient is arranged (step S103).
Specifically, the depth estimation / correspondence definition unit 220 estimates the depth of the corresponding subject for each sub-pixel constituting the LFI, and generates an LFDM in which the depth coefficient indicating the estimated depth is associated with the sub-pixel. To do.

Here, the depth coefficient may be calculated by using an arbitrary method for estimating the distance of the subject of the LFI pixel, but in the present embodiment, it is calculated by the following method.
i) A certain sub-image constituting the LFI is set as a target sub-image.
ii) The target sub-image is divided into image regions constituted by pixels whose pixel value difference is included in a predetermined range. Then, one of the image areas is selected as the attention area.
iii) Extract the sub-images on the right side of the target sub-image (left side if not present, in this case, the left and right in this case) as SR1, SR2,. Note that k is a natural number determined in setting.
iv) The coordinates (x, y) of the center of gravity of the attention area are acquired. Note that the coordinates are defined with respect to an independent coordinate system for each sub-image with the center of each sub-image as the origin.
v) Let d be the current pixel shift. An area (corresponding area) corresponding to the attention area is arranged at a part corresponding to the attention area in the sub image SR1. At this time, the center of gravity of the corresponding area is shifted to the right side by d. The sum of squares (SSD) of the difference between the pixel value of each pixel in the attention area and the pixel value of the corresponding pixel in the corresponding area is calculated. Similarly, the corresponding area is shifted to the right side by 2d in SR2, and the SSD is calculated. Each SSD is acquired up to SRk, and the sum of absolute differences (SSSD) of each SSD is obtained. This is the evaluation value of d.
vi) An evaluation value is calculated for each pixel shift d in the range of possible parallax. Among them, the pixel shift (d) in which the obtained absolute difference value sum SSSD is the minimum is set as the pixel shift coefficient of the pixel included in the attention area.
vii) Pixel shift coefficients are calculated for all pixels of all sub-images, and the calculated pixel shift coefficients are arranged as corresponding LFDM depth coefficients at the corresponding pixel locations.

Note that the pixel shift coefficient of each pixel can be obtained by any known method that determines where the pixel corresponding to the target region of a certain image is located in another image and calculates the shift of the mutual position. . The pixel shift coefficient calculated by such processing is that the subject corresponding to the sub-pixel of interest appears as an image at a different position for each image due to the difference in the viewpoint at which each sub-image was captured. ) Shows the estimation results.
Here, the difference in the position of the viewpoint (sub-lens) for each sub-image is referred to as lens parallax, and the difference in position at which the corresponding image generated by the lens parallax appears is referred to as image parallax. The image parallax increases as the lens parallax increases. Also, the depth (the distance between the viewpoint and the subject) increases as the depth increases. When the lens parallax is known between the sub-images as in this embodiment, the image parallax (deviation coefficient) can be a depth coefficient indicating the estimated depth.

  Next, the depth estimation / correspondence definition unit 220 generates the correspondence list shown in FIG. 5 by the ray tracing described in FIG. 4 (step S104). Then, the image generation device 30 acquires the correspondence list generated by the depth estimation / correspondence definition unit 220.

  When the LFDM is acquired in step S103 and the correspondence list is acquired in step S104, the image generation apparatus 30 then generates a temporary reconstructed image (RI1) using the LFI, the LFDM, and the correspondence list (temporary reconstructed image generation processing). ) Is started (step S105). Here, temporary reconstructed image generation processing 1 shown in FIG. 12 is executed.

  In the temporary reconstructed image generation process 1, first, the temporary reconstructed image generation unit 3210 defines a temporary reconstructed image on the reconstructed surface RF defined by the reconfiguration setting information (step S201). Further, a pixel of interest is selected from the pixels (reconstructed pixels) included in the temporary reconstructed image (step S202).

  Next, the temporary reconstructed image generation unit 3210 extracts a corresponding pixel corresponding to the target pixel from the sub-pixels (step S203). Specifically, in the correspondence list in FIG. 5, subpixels registered in a row in which the coordinates of the reconstructed pixel coincide with the coordinates of the target image are extracted as corresponding pixels.

  Next, the temporary reconstructed image generation unit 3210 calculates the pixel value of the target pixel (step S204). Specifically, the pixel values of the corresponding pixels extracted in step S203 are weighted and averaged using the weights of the corresponding pixels, and the obtained value is used as the pixel value of the target pixel.

Similarly, the temporary reconstructed image generation unit 3210 calculates the depth coefficient of the target pixel (step S205). Specifically, the depth coefficient defined by LFDM for each corresponding pixel extracted in step S203 is weighted and averaged using the weight of each corresponding pixel, and the obtained value is used as the depth coefficient of the target layer pixel. And
Note that the mode value of the depth coefficient of the corresponding pixel may be used as the depth coefficient of the target pixel.

  Next, it is determined whether or not the above processing is completed with all the reconstructed pixels of the target layer as the target pixel (step S206). If there is an unprocessed reconstruction pixel (step S206; NO), the process is repeated from step S202 on the next unprocessed reconstruction pixel. On the other hand, when all the reconstructed pixels have been processed (step S206; YES), the temporary reconstructed image generation process 1 ends.

  Returning to FIG. 11, when the temporary reconstructed image is generated in step S105, the temporary reconstructed image generated by the output unit 340 is output (step S106) and displayed on the display unit 520.

  Next, a process of accepting a deletion operation performed by the user for the temporary reconstructed image displayed in step S106 and defining a deletion subject and a deletion pixel based on the operation (deletion target definition process, here a deletion target definition process 1) ) Is executed by the image generation apparatus 30 (step S107).

The deletion target definition process 1 executed in step S107 will be described with reference to FIG.
In the deletion target definition process 1, it is first determined whether the deletion operation acquisition unit 330 has detected an operation for selecting a subject to be deleted by the user (step S301). Specifically, it is determined whether an operation touching the temporary reconstructed image (RI1) has been detected on the touch panel including the display unit 520 and the operation unit 530.
When it cannot detect (step S301; NO), it repeats step S301 and waits until it can detect.

  On the other hand, when the selection operation can be detected (step S301; YES), the selected pixel extraction unit 3220 extracts a reconstructed pixel (selected pixel) corresponding to the selected portion (step S302). Specifically, the coordinates of the reconstructed pixel corresponding to the designated position are obtained by a touch operation on the touch panel.

  Next, the selected pixel extraction unit 3220 refers to the temporarily reconstructed pixel (RI1) and its depth map (RDM1) to obtain the depth coefficient of the selected pixel (step S303), selects the selected pixel number, coordinates, The depth coefficient is stored in the selected pixel list in FIG. 9B in association with each other.

  Next, it is determined whether or not it is a selection end operation (step S304). Specifically, it is determined whether or not an operation indicating selection end, such as releasing a finger from the touch panel or executing an operation for instructing selection end, has been detected. When such an operation is not detected (step S304; NO), the process is repeated from step S302 based on the operation for selecting the next part.

  On the other hand, when the selection end operation is detected (step S304; YES), the process proceeds to step S305. In step S305, processing for defining a deletion depth coefficient (deletion depth definition processing, here, deletion depth definition processing 1) is executed.

  Deletion depth definition processing 1 will be described with reference to FIG. In the deletion depth definition process, first, the deletion depth definition unit 3230 obtains a depth coefficient (depth index) recorded in the selected pixel list (step S401). In the example of FIG. 9B, since 3, 6 and 1 are recorded as the values of the depth index, these three depth indexes are acquired.

  Next, the deletion depth definition unit 3230 selects one of the depth coefficients acquired in step S401 as the attention depth coefficient (step S402).

  Next, a ratio Qd is calculated for the attention depth coefficient using Expression (5) (step S403). For example, 45/200 is calculated as the value of the ratio Qd for the depth index 3 in FIG.

  Next, the deletion depth definition unit 3230 determines whether the ratio calculated in step S403 is equal to or greater than a predetermined threshold (step S404). If it is equal to or greater than the predetermined threshold (step S404; YES), the deletion depth definition unit 3230 determines the depth of interest as the deletion depth coefficient (step S405). On the other hand, if it is smaller than the predetermined threshold (step S404; NO), the target depth coefficient is deleted from the selected pixel list based on the determination that the target depth coefficient is not the deletion depth coefficient (step S406). Specifically, from the rows of the selected pixel list, the rows where the depth index matches the target depth coefficient are deleted.

  Next, it is determined whether or not the above processing has been completed for all the depth coefficients acquired in step S401 (step S407). If there is an unprocessed depth coefficient (step S407; NO), the process is repeated from step S402 for the next unprocessed depth coefficient. On the other hand, if all depth coefficients have been processed (step S407; YES), the deletion depth definition process ends.

  Returning to FIG. 13, when the deletion depth coefficient is defined in step S305, the deletion pixel extraction unit 3240 selects one of the defined deletion depth coefficients as the attention depth coefficient (step S306).

  Next, the deleted pixel extraction unit 3240 executes processing for deleting the deleted pixel from the light field image (LFI) (deleted pixel extraction processing, here, deleted pixel extraction processing 1) (step S307).

The deleted pixel extraction process 1 executed in step S307 will be described with reference to FIG.
First, in the deletion pixel extraction process 1, the deletion pixel extraction unit 3240 selects one of the selection pixels stored in the selection pixel list generated for the temporary reconstructed image (RI1) as a target selection pixel (step S501). ). Then, the target selected pixel is incorporated into a selection area to be selected in the temporary reconstructed image (RI1).

  Next, the deleted pixel extraction unit 3240 extracts adjacent reconstructed pixels that are pixels adjacent to the selected region (step S502). Specifically, for each pixel included in the selected region, a reconstructed pixel that is not included in the selected region is extracted as an adjacent reconstructed pixel among pixels adjacent in the vertical and horizontal directions.

  Next, the deleted pixel extraction unit 3240 determines whether or not the adjacent reconstructed pixel extracted in step S502 includes an incorporation pixel that satisfies a predetermined condition (step S503). Here, the predetermined condition is that the difference between the pixel value of the adjacent reconstructed pixel and the average value of the pixels included in the selected area is equal to or less than a predetermined threshold determined by the reconstruction setting, and the depth coefficient is the deletion depth coefficient ( It is included in a predetermined range (deleted depth range) including a target depth coefficient. For example, when the reconstruction setting defines the deletion depth range as a range of ± 2 of the deletion depth coefficient, if the depth coefficient is 6, 4 to 8 are the deletion depth range. In the present embodiment, the condition that the adjacent reconstruction pixels are incorporated into the selection area is a condition that the depth coefficient of the adjacent reconstruction pixels is included in the deletion depth range.

  When the adjacent reconstructed pixel extracted in step S502 includes a transfer pixel (step S503; YES), the deletion pixel extraction unit 3240 selects the transfer pixel as a selection area based on the determination that the transfer pixel is a pixel corresponding to the deletion subject. (Step S504). Then, the process returns to step S502, and further incorporation pixels are extracted.

On the other hand, if there is no transfer pixel (step S503; NO), the process proceeds to step S505 based on the determination that there is no pixel corresponding to the deleted subject in the range adjacent to the selected area at the current time.
In step S505, the deleted pixel extraction unit 3240 determines whether there is a pixel that is not included in the selection area among the selection pixels stored in the selection pixel list. If there is a selected pixel that is not included in the selected area (step S505; YES), the process is repeated from S501 with that pixel as a new target selected pixel.

  On the other hand, if there is no selected pixel that is not included in the selected area (step S505; NO), the deleted pixel extraction unit 3240 determines that a reconstructed pixel corresponding to the deleted subject is included in the current selected area. Then, the corresponding pixel of the light field image (LFI) corresponding to the selected region of the temporary reconstructed image (RI1) is extracted (step S506). Specifically, sub-pixels defined as corresponding to the reconstructed pixels included in the selected area in the correspondence list are extracted.

  Next, the deleted pixel extraction unit 3240 extracts, as a deleted pixel, a pixel having a depth coefficient included in the deleted depth range from the corresponding pixels extracted in step S506 (step S507). Then, the deleted pixel extraction process ends.

  Returning to FIG. 13, when the deleted pixel is extracted in step S307, it is next determined whether or not the above processing has been completed for all the deletion depths defined in step S305 (step S308). If there is an unprocessed deletion depth (step S308; NO), the process is repeated from step S306 for the next unprocessed deletion depth. On the other hand, when all the deletion depths have been processed (step S308; YES), the process proceeds to step S309.

  In step S309, as illustrated in FIG. 7C, the deleted pixel extraction unit 3240 outputs a temporary reconstructed image in which the deleted subject corresponding to the extracted deletion pixel is highlighted to the output unit 340. Specifically, an image capable of discriminating the deleted subject is generated by processing such as setting the pixel value of the deleted subject to black in RI1 or a pixel value blinking between black and white. Then, as shown in FIG. 7C, the generated image is displayed.

  Next, it is determined whether or not the deletion target (deletion pixel) has been confirmed by confirming the highlighted deletion area (deletion subject) and executing a confirmation operation (step S310). If the user has not decided such as selecting NO in FIG. 7C (step S310; NO), the deletion area is reset and the process returns to step S301.

  On the other hand, when the user selects YES in FIG. 7C and the deletion area is determined (step S310; YES), the deletion pixel extraction unit 3240 determines the sub-pixel extracted in step S307 as the deletion pixel. . Then, the deletion target definition process 1 ends.

  Returning to FIG. 11, when a deleted pixel is extracted in step S <b> 107, the deletion unit 3250 next executes a deletion process. Specifically, the pixel value of the deleted pixel is set to NULL (step S108). Then, the deletion reconstructed image generation unit 3260 starts processing for generating the deletion reconstructed image RI2 (deletion reconstructed image generation processing, here, deletion reconstructed image generation processing 1) (step S109).

  Deletion reconstructed image generation processing 1 executed in step S109 will be described with reference to FIG. In the deleted reconstructed image generation process 1, the deleted reconstructed image generation unit 3260 selects one of the reconstructed pixels as a target pixel (step S601).

  Next, the deletion reconstructed image generation unit 3260 extracts a corresponding pixel corresponding to the target pixel from the sub-pixels (step S602). Specifically, in the correspondence list in FIG. 5, subpixels registered in a row in which the coordinates of the reconstructed pixel coincide with the coordinates of the target image are extracted as corresponding pixels.

  Next, the deleted reconstructed image generation unit 3260 determines whether or not the corresponding pixel extracted in step S602 includes a pixel having a NULL pixel value (deleted pixel) (step S603). If the deleted pixel is included (step S603; YES), the deleted pixel is excluded from the target of the subsequent calculation processing (step S604). On the other hand, if the deleted pixel is not included (step S603; NO), step S604 is skipped.

  Next, the deletion reconstructed image generation unit 3260 calculates the pixel value of the target pixel (step S605). Specifically, the pixel values of the corresponding pixels extracted in step S602 are weighted and averaged using the weights of the corresponding pixels, and the obtained value is used as the pixel value of the target pixel. At this time, the deleted pixel excluded in step S604 is not subject to calculation processing.

  At this time, if all the corresponding pixels are deleted pixels (or the ratio occupied by the deleted pixels is larger than a predetermined ratio), a predetermined default value (for example, black) is set as the pixel value of the reconstructed pixel. At this time, the user is warned that information that is not based on photographing appears in the deletion reconstruction pixel by the deletion operation.

Similarly, the deleted reconstructed image generation unit 3260 calculates the depth coefficient of the target pixel (step S606). Specifically, the depth coefficient defined by LFDM for each corresponding pixel extracted in step S602 is weighted and averaged using the weight of each corresponding pixel, and the obtained value is used as the depth coefficient of the target pixel. To do.
Note that the mode value of the depth coefficient of the corresponding pixel may be used as the depth coefficient of the target pixel.
At this time, the deleted pixel is excluded from the calculation processing target.

  Next, it is determined whether or not the above processing is completed with all the reconstructed pixels as the target pixel (step S607). If there is an unprocessed reconstruction pixel (step S607; NO), the process is repeated from step S601 for the next unprocessed reconstruction pixel. On the other hand, when all the reconstructed pixels have been processed (step S607; YES), the deleted reconstructed image generation process 1 ends because the deleted reconstructed image has been generated.

  Returning to FIG. 11, when the deletion reconstructed image is generated in step S109, the deletion reconstructed image generated by the output unit 340 is output to the image storage unit 430 (step S110). Then, the display unit 520 or the I / O unit 510 outputs the changed reconstructed image to the outside, and the image output process ends.

  As described above, the image generation apparatus 30 according to the present embodiment can generate a reconstructed image in which a subject to be deleted is deleted. Further, when deleting a subject, sub-pixels corresponding to the subject to be deleted are extracted, and pixel values of reconstructed pixels are determined based on pixel values of other sub-pixels. For this reason, the ratio of pixels that do not depend on the shooting information of the main subject (pixels of the subject to be deleted) appears in the reconstructed image (RI2) after deletion is small. That is, it is possible to generate the deleted reconstructed image as an image more reflecting the shooting information of the main subject.

  In addition, the image generation apparatus 30 according to the present embodiment performs a deletion process on a pixel having a depth coefficient estimated that the deleted subject is located, and deletes the image of the deleted subject from the reconstructed image. With this configuration, it is possible to delete a subject while maintaining an image having a depth that does not include the subject to be deleted. For this reason, an image of a subject located on the back, which is photographed on the LFI, appears at the deleted part. Therefore, it is possible to generate a deletion reconstructed image close to the case where the subject is actually removed.

  In addition, the image generation apparatus 30 according to the present embodiment generates a reconstructed depth map from the LFDM, and determines the depth coefficient of the subject based on the depth coefficient of the selected pixel (a part of the reconstructed pixel). For this reason, subpixels that match the depth of the determined deletion subject can be extracted as deletion pixels, and a deletion reconstructed image close to the case where the subject is actually removed can be generated.

  Furthermore, the digital camera 1 of the present embodiment has a configuration in which a temporary reconstructed image that is not subjected to deletion processing is generated and presented to the user. Therefore, the user can select a desired deletion target after confirming an image when deletion is not performed. Therefore, it is highly convenient for the user when deleting the subject.

  Further, correspondence information based on the photographing parameters (photographing setting information) is acquired, and a temporary reconstructed image and a deleted reconstructed image are generated based on the correspondence information. Therefore, it is possible to generate a highly accurate temporary reconstructed image and a deleted reconstructed image reflecting the conditions at the time of shooting.

(Modification)
The present invention is not limited to the above embodiment, and various modifications can be made.
For example, in the above embodiment, the deletion depth range is set as a predetermined range including a depth coefficient in which the depth idex extracted by the deletion operation is larger than a predetermined ratio. Then, among the corresponding sub-pixels of the reconstructed pixel corresponding to the deleted subject, the sub-pixel whose depth coefficient is included in the deleted depth range is determined as the deleted pixel.
However, the method of extracting the deleted pixel of the present invention is not limited to this.
For example, the digital camera 1 may use all subpixels belonging to the depth range designated by the user as deleted pixels. According to such a configuration, for example, when there is a belt-like obstacle in front of the main subject, a process of uniformly removing the obstacles can be easily achieved.
Alternatively, a configuration is possible in which a sub-pixel belonging to a user-specified depth range corresponding to an arbitrary part specified by the user is determined as a deletion pixel.

In the above embodiment, the selection area (the area on the reconstructed image corresponding to the deletion subject) is defined by the deletion pixel extraction process 1 in FIG. 15, but the method for defining the selection area is not limited to this.
For example, a configuration in which a reconstructed image is divided into image areas based on pixel values and depth coefficients in advance, and an area in which the number of selected pixels included in the image area is greater than a predetermined ratio is selected as the selection area. Is possible. At this time, a known arbitrary method such as a division / integration method or a graph cut method can be used as a method for dividing the image region.

  Note that the method of dividing the reconstructed image into image regions in advance can be applied to processing for defining the deletion depth range. For example, based on the selection operation, an area in which the number of selected pixels included in the image area is greater than a predetermined ratio is first extracted as a selection area. Then, it is possible to adopt a configuration in which the mode value of the depth coefficient of the selected region is set as the deletion depth coefficient, and a predetermined range is set as the deletion depth range from the depth coefficient.

Further, it is possible to divide the range of the depth coefficient assumed in the design of the digital camera into a plurality of classes (depth classes) and execute the deletion process. In this modification, a depth class is defined using a list (class-depth correspondence list in FIG. 17) that stores a depth class and a corresponding range of depth coefficients in association with each other. In the example of FIG. 17, three depth classes are defined for the depth index (0 to MAX) assumed in the setting. For example, the fact that the class index 2 and the corresponding depth indexes 3 to 5 are registered in the same row indicates that pixels having a depth index (depth coefficient) of 3 to 5 belong to the second class.
The method of assigning the depth to the class is arbitrary, such as pre-defining with the reconstruction setting, or clustering each LFDM depth value by a known clustering method. Examples of the clustering method include clustering using a ward method so that the distance between classes of depth values to which each class belongs is maximized, using a k-average method, and the like.

In the configuration using classes, a selection pixel list as shown in FIG. 18 is generated based on the selection operation as shown in FIG. Then, among all the selected pixels, a class index exceeding a predetermined ratio is set as a selection class. Then, the deletion pixel is extracted using the depth range assigned to the selected class as the deletion depth range.
According to such a configuration, the ratio for selection is calculated as one class for the allocated range. Therefore, even when the depth index of the deleted subject does not have a constant numerical value and fluctuates, the user can select and delete the desired deleted subject.

Further, in the above-described embodiment, the deletion unit sets the pixel value of the deleted pixel to NULL and excludes it from calculation when generating the deletion reconstruction.
However, in the present invention, the deleted reconstructed image may be generated using any method that reduces the ratio of the pixel value of the deleted pixel appearing in the reconstructed pixel to such an extent that it does not interfere with the viewing of the desired subject.
For example, the deleted reconstructed image can be generated by a configuration in which the deletion unit deletes the deleted pixel from the correspondence list in FIG. 5 so that the pixel value of the deleted pixel is not used in the reconstructed image generation process. Alternatively, instead of erasing the deleted pixel from the correspondence list, it is possible to reduce the weight w stored in the correspondence list for the deleted pixel to an extent that does not interfere with viewing. According to this configuration, an image deleted by a certain ratio can be generated as shown in FIG.
As an example of such a configuration, it is conceivable that the user specifies a ratio for reducing the weight of the deleted pixel using a slide bar or the like. With this configuration, the user can lower the degree to which the deleted subject appears in the deleted reconstructed image to a desired level.

In this way, the deletion unit performs processing such as setting the pixel value of the deleted pixel to NULL and excluding it from the processing target in the subsequent reconstruction processing, deleting the deleted pixel from the correspondence list, and reducing the weight of the deleted pixel. . These processes can be rephrased as changing the correspondence between the reconstructed pixel and the sub-pixel so as to reduce the degree of correspondence between the deleted pixel and the reconstructed pixel. Here, “reducing the degree of correspondence” includes setting the degree of correspondence to zero. For this reason, a deletion part can also be expressed as a change part.
By such processing, the image generation apparatus can effectively reduce the proportion of the sub-pixel pixel values that can be determined to include the deleted subject appear in the reconstructed image. As a result, an image of a corresponding pixel other than the deleted pixel appears in a portion where the deleted subject is present, so that it is possible to generate a deleted reconstruction image increase that is close to the image when the deleted subject is actually removed.

  Furthermore, the deletion target may not be determined based on the user operation. For example, a configuration such as automatically deleting a subject of the closest class or a subject in front of a desired focal plane is possible.

Furthermore, in the above description, the example in which the weighting coefficient is included in the correspondence information has been described. However, the correspondence information may be information that defines the sub-pixel corresponding to the reconstructed pixel, and it is not necessary to define the weight. No.
Furthermore, in the above description, the correspondence information is generated for each LFI based on the shooting setting information and the reconstruction setting. However, the present invention is not limited to this, and predetermined correspondence information stored in advance may be used. This configuration is suitable for reducing the amount of calculation while maintaining the accuracy of the allowable range when there is little room for the assumed imaging parameters to fluctuate and the reconstruction distance is constant.

  In the above embodiment, the image is described as a grayscale image. However, the image to be processed according to the present invention is not limited to a grayscale image. For example, the image may be an RGB image in which three pixel values of R (red), G (green), and B (blue) are defined for each pixel. In this case, the pixel value is similarly processed as an RGB vector value. Alternatively, the above processing may be performed using each of R, G, and B values as independent grayscale images. According to this configuration, a reconstructed image that is a color image can be generated from a light field image that is a color image.

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  A central part that performs processing for generating a deleted reconstructed image including the information processing unit 31, the main storage unit 32, the external storage unit 33, and the like uses a normal computer system, not a dedicated system. Is feasible. For example, a computer program for executing the above operation is stored and distributed in a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. You may comprise the center part which performs the process for deletion reconstruction image generation. Further, the computer program may be stored in a storage device included in a server device on a communication network such as the Internet, and the image generation device may be configured by downloading or the like by a normal computer system.

  When the functions of the image generation device are realized by sharing of an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program portion may be stored in a recording medium or a storage device. Good.

  It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board (BBS: Bulletin Board System) on a communication network, and the computer program may be distributed via the network. The computer program may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to the specific embodiment which concerns, This invention includes the invention described in the claim, and its equivalent range It is. Hereinafter, the invention described in the scope of claims of the present application will be appended.

(Appendix 1)
An image generation device that generates a reconstructed image defined on a predetermined reconstruction plane from a captured image composed of a plurality of sub-images obtained by capturing a plurality of subjects from each of a plurality of viewpoints,
A depth acquisition unit that acquires a depth coefficient indicating the depth of the subject corresponding to the sub-pixel for each of the sub-pixels constituting the sub-image of the captured image;
An extraction unit that extracts, from the sub-pixels, sub-pixels whose depth coefficient acquired by the depth acquisition unit satisfies a predetermined condition as deletion pixels corresponding to a deletion subject to be deleted;
Based on the pixel value of the reconstructed pixel constituting the reconstructed image based on the pixel value of the subpixel corresponding to the reconstructed pixel, the influence of the pixel value of the deleted pixel extracted by the extraction unit among the subpixels is small. A generation unit that generates a reconstructed image in which the deleted subject is deleted,
An image generation apparatus comprising:

(Appendix 2)
The extraction unit acquires the depth of the deleted subject, the depth coefficient acquired by the depth acquisition unit is a sub-pixel included in a predetermined range including the depth of the deleted subject, and the depth coefficient is a sub Pixel
The image generating apparatus according to Supplementary Note 1, wherein

(Appendix 3)
An acquisition unit for obtaining the depth of the reconstructed pixel from the depth coefficient of the sub-pixel acquired by the depth acquisition unit;
The extraction unit acquires at least a part of a portion on the reconstructed image where the deleted subject is located, and based on the depth obtained by the acquisition unit for the reconstructed pixel of the acquired portion, To get the depth,
The image generating apparatus according to appendix 2, characterized in that:

(Appendix 4)
A change unit that changes the correspondence between the reconstructed pixel and the sub-pixel so that the degree of correspondence between the deleted pixel and the reconstructed pixel is reduced;
The generating unit obtains the pixel value of the reconstructed pixel so that the influence of the pixel value of the deleted pixel is reduced using the correspondence relationship changed by the changing unit.
The image generation apparatus according to any one of appendices 1 to 3, wherein

(Appendix 5)
A creation unit that creates a temporary reconstructed image that is a reconstructed image including the plurality of subjects based on the correspondence before the change unit changes;
An output unit for outputting the temporarily reconstructed image created by the creation unit;
A reception unit for designating a part on the temporary reconstructed image output by the output unit;
Further comprising
The extraction unit extracts, as the deletion pixel, a sub-pixel corresponding to a deletion site specified by the reception unit among sub-pixels in which the depth coefficient satisfies a predetermined condition.
The image generating apparatus according to appendix 4, wherein:

(Appendix 6)
A correspondence acquisition unit that acquires information indicating a correspondence relationship between the reconstructed pixel and the sub-pixel, which is obtained based on a photographing parameter when the photographed image is photographed;
The change unit changes the correspondence relationship indicated by the information acquired by the correspondence acquisition unit.
The image generating apparatus according to appendix 4 or 5, characterized in that:

(Appendix 7)
Acquire a captured image consisting of multiple sub-images, shooting multiple subjects from each of multiple viewpoints,
For each of the sub-pixels constituting the sub-image of the acquired captured image, obtain a depth coefficient indicating the depth of the subject corresponding to the sub-pixel,
From the sub-pixels, the sub-pixels for which the acquired depth coefficient satisfies a predetermined condition are extracted as deleted pixels corresponding to the deleted subject to be deleted,
Based on the pixel value of the sub-pixel corresponding to the reconstruction pixel, the pixel value of the extracted deletion pixel among the sub-pixels based on the pixel value of the reconstruction pixel constituting the reconstruction image defined on the predetermined reconstruction plane By generating a reconstructed image in which the deleted subject is deleted by determining that the influence of the value is reduced,
An image generation method characterized by the above.

(Appendix 8)
On the computer,
A function for acquiring a captured image composed of a plurality of sub-images obtained by capturing a plurality of subjects from each of a plurality of viewpoints;
A function of acquiring a depth coefficient indicating the depth of the subject corresponding to the sub-pixel for each of the sub-pixels constituting the sub-image of the acquired captured image;
A function of extracting, from the sub-pixel, a sub-pixel for which the acquired depth coefficient satisfies a predetermined condition as a deletion pixel corresponding to a deletion subject to be deleted;
Based on the pixel value of the sub-pixel corresponding to the reconstruction pixel, the pixel value of the extracted deletion pixel among the sub-pixels based on the pixel value of the reconstruction pixel constituting the reconstruction image defined on the predetermined reconstruction plane A function of generating a reconstructed image in which the deleted subject is deleted by determining that the influence of the value is small;
A program characterized by realizing.

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 10 ... Imaging part, 110 ... Optical apparatus, 120 ... Image sensor, 20 ... Information processing part, 30 ... Image generation apparatus, 31 ... Information processing part, 32 ... Main memory part, 33 ... External memory part, 36 ... Input / output unit, 37 ... Internal bus, 38 ... Program, 210 ... Image processing unit, 220 ... Depth estimation / correspondence definition unit, 230 ... Imaging control unit, 310 ... Input unit, 3110 ... Reconfiguration setting acquisition unit, 3120 LFDM acquisition unit, 3130 ... LFI acquisition unit, 3140 ... correspondence acquisition unit, 320 ... reconstructed image generation unit, 3210 ... temporary reconstructed image generation unit, 3220 ... selected pixel extraction unit, 3230 ... deletion depth definition unit, 3240 ... Deletion pixel extraction unit, 3250 ... deletion unit, 3260 ... deletion reconstructed image generation unit, 330 ... deletion operation acquisition unit, 340 ... output unit, 40 ... storage unit, 410 ... imaging setting storage unit 420 ... Reconfiguration setting storage unit, 430 ... Image storage unit, 50 ... Interface unit (I / F unit), 510 ... I / O unit, 520 ... Display unit, 530 ... Operation unit, LFI ... Light field image, LFDM ... Light field depth map, OA ... optical axis, OB ... subject, OB1 ... subject, OB2 ... subject, OB2a ... subject, OB2b ... subject, OB3 ... subject, POB ... subject part, P ... target region, ML ... main lens, PF ... imaging point, MIP ... main lens imaging surface, PE ... arrival point, IE ... imaging surface, SLA ... sub lens array, SL ... sub lens, MLB ... main lens blur, MLBC ... main lens blur center, S11 SMN: Sub-image, RDM: Reconstructed depth map, RDM1: Reconstructed depth map, RDM2: Reconstructed depth map, RI ... temporary reconstructed image, RI2 ... remove reconstructed image, RF ... reconstruction plane

Claims (9)

And shoot each one these multiple viewpoints, an image acquisition section which acquires a captured image including a plurality of sub-images,
Sub pixels constituting the sub-image, an extraction unit for extracting a sub-pixel corresponding to the deletion object to be deleted as a deletion pixel,
The correspondence relationship between the reconstructed pixels constituting the reconstructed image defined on the predetermined reconstructed surface and the sub-pixels is changed so that the degree of correspondence between the sub-pixels extracted as the deleted pixels is reduced. Change part,
A generation unit that generates a reconstructed image with a small influence of the deleted pixel by obtaining a pixel value of the reconstructed pixel from a pixel value of the sub-pixel using the changed correspondence relationship ;
An image generation apparatus comprising: The generating unit generates a reconstructed image in which the deleted subject is deleted using the correspondence relationship changed by the changing unit.
  The image generating apparatus according to claim 1. For each of the sub-pixels, further comprising a depth acquisition unit that acquires a depth coefficient indicating the depth of the subject corresponding to the sub-pixel,
The extraction unit extracts, from the sub-pixel, a sub-pixel that satisfies a predetermined condition for the depth coefficient acquired by the depth acquisition unit as a deletion pixel corresponding to the deletion subject.
The image generation apparatus according to claim 1, wherein the image generation apparatus is an image generation apparatus. The extraction unit acquires the depth of the deleted subject, the depth coefficient acquired by the depth acquisition unit is a sub-pixel included in a predetermined range including the depth of the deleted subject, and the depth coefficient is a sub Pixel
The image generating apparatus according to claim 3 . An acquisition unit for obtaining the depth of the reconstructed pixel from the depth coefficient of the sub-pixel acquired by the depth acquisition unit;
The extraction unit acquires at least a part of a portion on the reconstructed image where the deleted subject is located, and based on the depth obtained by the acquisition unit for the reconstructed pixel of the acquired portion, To get the depth,
The image generation apparatus according to claim 4 . A second generation unit that generates a temporary reconstructed image based on the correspondence before the change unit changes;
An output unit that outputs the temporarily reconstructed image generated by the second generation unit ;
A designating unit for designating a part on the temporary reconstructed image output by the output unit;
Further comprising
The extraction unit, a sub-pixel corresponding to the part position designated by the designation section is extracted as the deletion pixel,
The image generation apparatus according to claim 1 , wherein the image generation apparatus is an image generation apparatus. A correspondence acquisition unit that acquires information indicating a correspondence relationship between the reconstructed pixel and the sub-pixel, which is obtained based on a photographing parameter when the photographed image is photographed;
The change unit changes the correspondence relationship indicated by the information acquired by the correspondence acquisition unit.
The image generation apparatus according to claim 1 , wherein the image generation apparatus is an image generation apparatus. And shoot each one these multiple viewpoints, acquiring a captured image including a plurality of sub-images,
Extracting the sub-pixels either et constituting a sub-image, the sub-pixels corresponding to the deletion object to be deleted as a deletion pixel,
The correspondence relationship between the reconstructed pixels constituting the reconstructed image defined on the predetermined reconstructed surface and the sub-pixels is changed so that the degree of correspondence between the sub-pixels extracted as the deleted pixels is reduced. Steps,
Using the changed correspondence relation, the from pixel values of the sub-pixels to obtain the pixel values of the reconstructed pixel and generating a reconstructed image is less affected the deletion pixel,
Image generation method, which comprises a. On the computer,
And shoot each one these multiple viewpoints, the ability to obtain the captured image including a plurality of sub-images,
The sub-pixel or we constituting a sub-image, a function of extracting a sub-pixel corresponding to the deletion object to be deleted as a deletion pixel,
The correspondence relationship between the reconstructed pixels constituting the reconstructed image defined on the predetermined reconstructed surface and the sub-pixels is changed so that the degree of correspondence between the sub-pixels extracted as the deleted pixels is reduced. function,
A function of generating a reconstructed image with a small influence of the deleted pixel by obtaining a pixel value of the reconstructed pixel from a pixel value of the sub-pixel using the changed correspondence relationship ;
A program characterized by realizing.