JP5942428B2 - Reconstructed image generating apparatus, reconstructed image generating method, and program - Google Patents

Description

  The present invention relates to a reconstructed image generation apparatus, a reconstructed image generation method, and a program.

  There is known a technique for acquiring the direction and light amount of all light rays that enter a lens of a camera from a subject by acquiring images of the subject taken from different viewpoints.

  In relation to such a technique, Patent Document 1 acquires a plurality of images obtained by photographing a subject and reconstructs a subject image in which a focal length, a depth of field, and the like are changed from the plurality of images. Is disclosed. In the technique of Patent Document 1, a pixel value of a pixel (reconstructed pixel) of a reconstructed image is determined by adding a sub-set of pixels (corresponding pixels) included in a captured image. Patent Document 1 mentions that the pixel value of the reconstructed pixel is determined by weighted addition of the pixel values of the corresponding pixels.

Special table 2008-515110 gazette

  In order to generate a reconstructed image reconstructed with high accuracy using the technique described in Patent Document 1, it is necessary to determine the weight corresponding to each of the reconstructed pixel and the corresponding pixel corresponding thereto. However, Patent Document 1 does not disclose a specific method for obtaining the weight value. Further, even if the correspondence between the reconstructed pixel and the corresponding pixel is determined by ray tracing and the weight is calculated based on the path and other conditions, the weight must be calculated for each pixel one by one, and the calculation amount is large. There was a problem of becoming enormous.

  The present invention has been made in view of the above circumstances, and a reconstructed image generating apparatus and a reconstructed image that require a small amount of calculation when generating a reconstructed image reconstructed with high accuracy from a plurality of images obtained by photographing a subject. An object is to provide an image generation method and a program.

In order to achieve the above object, an aspect of the reconstructed image generating apparatus according to the present invention is:
Using a plurality of sub-lenses corresponding to the plurality of viewpoints further capture the light main lens and the main lens captures to capture light from an object, a plurality of sub-images taken the subject from a different said viewpoint A light field image acquisition unit for acquiring a light field image composed of:
A prototype image definition unit that defines the original image of the reconstructed image is an image of the subject reconstructed from the light-field image,
Image weight definition that defines an image weight that is a coefficient indicating the strength of correspondence between a prototype pixel that is a pixel constituting the prototype image defined by the prototype image definition unit and the sub lens corresponding to the sub image And
A corresponding pixel extracting unit that extracts a corresponding pixel that is a pixel corresponding to the original pixel constituting the original image defined by the original image defining unit from the sub image pixels constituting the plurality of sub images;
And the pixel value of the corresponding pixel before SL corresponding pixel extracting portion is extracted, the said image weight said image weight definition portion is defined in accordance with the sub-lenses corresponding to said sub-image belonging paired pixels, on the basis Obtaining a pixel value of the reconstructed pixel of the reconstructed image and generating the reconstructed image;
I have a,
When an area in which the light beam from the subject passes through the main lens and is projected onto the plurality of sub lenses is a main lens blur area, the image weight is determined by the main lens blur area on each sub lens. Defined as a percentage,
It is characterized by that.
In order to achieve the above object, an aspect of the reconstructed image generation method according to the present invention is:
A plurality of sub-images obtained by photographing the subject from different viewpoints using a main lens that captures light rays from the subject and a plurality of sub-lens corresponding to each of a plurality of viewpoints that further capture the light rays captured by the main lens Obtaining a light field image comprising:
Defining a prototype image of a reconstructed image that is an image of the subject reconstructed from the light field image;
Defining an image weight that is a coefficient indicating the strength of correspondence between a prototype pixel that is a pixel constituting the prototype image and the sub lens corresponding to the sub image;
Extracting a corresponding pixel, which is a pixel corresponding to the original pixel constituting the original image, from sub image pixels constituting the plurality of sub images; and
A pixel value of the reconstructed pixel of the reconstructed image is obtained based on the extracted pixel value of the corresponding pixel and the image weight defined according to the sub lens corresponding to the sub image to which the corresponding pixel belongs. Generating the reconstructed image;
Including
When an area in which the light beam from the subject passes through the main lens and is projected onto the plurality of sub lenses is a main lens blur area, the image weight is determined by the main lens blur area on each sub lens. Defined as a percentage,
It is characterized by that.
In order to achieve the above object, an aspect of the program according to the present invention is as follows.
On the computer,
A plurality of sub-images obtained by photographing the subject from different viewpoints using a main lens that captures light rays from the subject and a plurality of sub-lens corresponding to each of a plurality of viewpoints that further capture the light rays captured by the main lens Processing to obtain a light field image consisting of
Processing for defining a prototype image of a reconstructed image that is an image of the subject reconstructed from the light field image;
A process of defining an image weight that is a coefficient indicating a strength of correspondence between a prototype pixel that is a pixel constituting the prototype image and the sub lens corresponding to the sub image;
A process of extracting a corresponding pixel that is a pixel corresponding to the original pixel constituting the original image from the sub image pixels constituting the plurality of sub-images;
A pixel value of the reconstructed pixel of the reconstructed image is obtained based on the extracted pixel value of the corresponding pixel and the image weight defined according to the sub lens corresponding to the sub image to which the corresponding pixel belongs. A process for generating the reconstructed image;
And execute
When an area in which the light beam from the subject passes through the main lens and is projected onto the plurality of sub lenses is a main lens blur area, the image weight is determined by the main lens blur area on each sub lens. Defined as a percentage,
It is characterized by that.

  According to the present invention, it is possible to reduce the amount of calculation required for generating a reconstructed image reconstructed with high accuracy from a plurality of images obtained by photographing a subject.

It is a figure which shows the structure of the digital camera which concerns on Embodiment 1 of this invention. 2 is a diagram illustrating a configuration of optical formation of the digital camera according to Embodiment 1. FIG. FIG. 2 is a diagram illustrating an example of a light field image according to the first embodiment, where (a) illustrates a conceptual diagram of the light field image, and (b) illustrates an example of the light field image and a reconstructed image. It is a figure which shows the (a) physical structure and (b) functional structure of the image reconstruction apparatus which concern on Embodiment 1. FIG. FIG. 3 is a diagram illustrating an example of an optical system and an optical image of the digital camera according to the first embodiment. 2A and 2B are diagrams for explaining a main lens blur according to the first embodiment, where FIG. 3A shows a main lens blur projected on a sub-lens array, and FIG. 2B shows a center coordinate of the main lens blur on the sub-lens. . It is a figure which shows the example of the MLB table which concerns on Embodiment 1, (a) shows the MLB index table contained in an MLB table, (b) shows the MLB weight table contained in an MLB table. 3 is a flowchart illustrating image output processing according to the first embodiment. 3 is a flowchart illustrating image reconstruction processing according to the first embodiment. 6 is a flowchart illustrating pixel value addition processing according to the first embodiment. It is a figure which shows the structure of the digital camera which concerns on Embodiment 2 of this invention. It is a figure which shows the example of the optical system of the digital camera which concerns on Embodiment 2, and an optical image. FIG. 6 is a diagram illustrating a functional configuration of an image reconstruction device according to a second embodiment. 4A and 4B are diagrams for explaining a sub-lens blur according to the second embodiment, in which FIG. 5A shows a sub-lens blur projected on the imaging surface, and FIG. 5B shows a center coordinate of the sub-lens blur on the image sensor. It is a figure which shows the example of the SLB table which concerns on Embodiment 2, (a) shows the SLB index table contained in a SLB table, (b) shows the SLB weight table contained in a SLB table. 10 is a flowchart illustrating pixel value addition processing according to the second embodiment. It is a figure which shows the structure of the optical system of the digital camera which concerns on other embodiment of this invention. (A) is a figure which shows the example of the MLB index table which concerns on other embodiment, (b) is a figure which shows an SLB index table.

  Hereinafter, a digital camera and a reconstructed image generating apparatus (image reconstructing apparatus) according to an embodiment for carrying out the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

(Embodiment 1)
The image reconstruction device 30 according to the first embodiment is mounted on the digital camera 1 shown in FIG. The digital camera 1 has a function of photographing a subject to obtain a light field image composed of a plurality of sub-images and generating an arbitrary reconstructed image. The image reconstruction device 30 is responsible for generating a reconstructed image from a light field image composed of a plurality of sub-images.

  As shown in FIG. 1, the digital camera 1 includes an imaging unit 10, an information processing unit 20 including an image reconstruction device 30, a storage unit 40, and an interface unit (I / F unit) 50. . With such a configuration, the digital camera 1 acquires light ray information from the outside and displays an image representing the light ray information.

  The imaging unit 10 includes an optical device 110 and an image sensor 120, and performs an imaging operation.

As shown in FIG. 2, the optical device 110 includes a shutter 111, a main lens ML, and a sub lens array SLA (micro lens array). The optical device 110 captures light rays from the outside by the main lens ML, and the sub lens array. An optical image obtained by using the optical center of each sub lens SL constituting the SLA as a viewpoint is projected onto the image sensor 120. In the present embodiment, the main lens ML is a single focus lens, and its focal length f _ML is fixed.

  The image sensor 120 converts an optical image projected by the optical device 110 into an electrical signal and transmits the electrical signal to the information processing unit 20. For example, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Material Oxide Semiconductor). And a transmission unit that transmits an electrical signal generated by the image sensor to the information processing unit 20.

The shutter 111 controls the incidence and shielding of external light on the image sensor 120.
The main lens ML is composed of one or a plurality of convex lenses, concave lenses, aspherical lenses, and the like, and a virtual imaging surface between the main lens ML and the sub lens array SLA using the light of the subject OB at the time of photographing as an optical image. The image is formed on the MIP. Note that the subject OB at the time of shooting is not limited to a single object, and may be a plurality of components separated from the main lens ML by different distances as shown in FIG.

  The sub lens array SLA includes M × N sub lenses (micro lenses) SL arranged in a lattice pattern on a plane. The sub-lens array SLA is an image sensor that constitutes the image sensor 120 as an optical image obtained by observing an optical image formed by the main lens ML on the imaging plane MIP with the optical center of each sub-lens SL as a viewpoint. An image is formed on the imaging surface IE. A space formed by a plane formed by the main lens ML and a plane formed by the imaging surface IE is referred to as a light field.

For the main lens ML, a maximum diameter LD and an effective diameter ED can be defined. The maximum diameter LD is the physical diameter of the main lens ML. On the other hand, the effective diameter ED is a diameter of a region that can be used for photographing in the main lens ML. Of the main lens ML, the outside of the effective diameter ED captures and reconstructs an image because light beams input to and output from the main lens are blocked by various filters attached to the main lens ML and the physical structure around the main lens ML. This is an area that is not effective for the purpose (non-effective area).
The effective diameter ED can be defined by measuring a part where light rays are blocked by the physical structure such as the filter at the time of shipment from the factory.

  In the example of FIG. 2, the light beam from the portion POB where the subject OB is present passes through the portion (effective portion) forming the effective diameter ED of the main lens ML and is projected onto the plurality of sub lenses SL. In this way, an area in which light emitted from the portion POB of the subject OB passes through the effective portion of the main lens ML and is projected onto the sub lens array SLA is referred to as a main lens blur MLB of the portion POB.

  The main lens ML is housed inside the lens barrel. The lens barrel is divided into an object side (outer lens barrel) and an image plane MIP side (inner lens barrel) at the optical center of the main lens ML. In FIG. 2, the length of the outer lens barrel is Lolb, and the length of the inner lens barrel is Lilb. The distance from the optical center of the main lens ML to the imaging plane MIP of the main lens is b1, the distance from the imaging plane MIP to the plane formed by the sub lens array SLA is a2, and the imaging plane of the image sensor from the sub lens array SLA. Let the distance of IE be c2.

With the above-described configuration, the imaging unit 10 captures a light field image (LFI) including information on all light rays that pass through the light field.
FIG. 3A shows an example of the LFI obtained by shooting the block-shaped subject OB.
This LFI is composed of images (sub-images S, S _{11 to} S _MN ) corresponding to each of M × N sub-lenses SL (microlenses) arranged in a grid pattern. For example, the upper left sub-image S ₁₁ corresponds to an image obtained by photographing the object OB from the upper left, the sub-image S _MN the lower right corresponds to an image obtained by photographing the object OB from the lower right.

The i-th row sub-images (horizontal row sub-images) S _{i1 to} S _{iN form} images formed by the main lens ML by the sub-lens SL arranged side by side in the i-th row of the sub-lens array SLA. It corresponds to a stereo image. Similarly, the sub-images in the j-th column (vertical sub-images) S _{1j to} S _Mj are images formed by the main lens ML in the vertical direction in the j-th column of the sub-lens array SLA (microlens array). This corresponds to a stereo image taken with the sub lenses SL arranged side by side.
Digital camera 1, as shown in FIG. 3 (b), to generate a reconstructed image (RI) reconstructed a subject as the image from the LFI.
In the present embodiment, each sub image S is a gray scale image, and each pixel constituting the sub image has a pixel value (scalar value).

  The information processing unit 20 shown in FIG. 1 is physically composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), an internal bus, and an I / O port. It functions as the reconstruction device 30 and the imaging control unit 220.

  The image processing unit 210 acquires an electrical signal from the image sensor 120 and converts the acquired electrical signal into image data based on imaging setting information stored in the imaging setting storage unit 410 of the storage unit 40. The image processing unit 210 transmits the image data and shooting setting information obtained by adding predetermined information to the shooting setting information to the image reconstruction device 30.

  The imaging setting information stored in the imaging setting storage unit 410 will be described later.

The image reconstruction device 30 generates a reconstructed image using the LFI transmitted from the image processing unit 210. The process in which the image reconstruction device 30 generates a reconstructed image will be described later.
The image reconstruction device 30 stores the generated reconstructed image in the image storage unit 420 of the storage unit 40.

  The imaging control unit 220 controls the imaging unit 10 based on the imaging setting information stored in the imaging setting storage unit 410 of the storage unit 40 and images the subject OB using the imaging unit 10.

The storage unit 40 includes a main storage device including a RAM (Random-Access Memory) and the like, and an external storage device including a nonvolatile memory such as a flash memory and a hard disk.
The main storage device loads control programs and information stored in the external storage device and is used as a work area of the information processing unit 20.
The external storage device stores in advance a control program and information for causing the information processing unit 20 to perform processing to be described later, and transmits these control program and information to the main storage device in accordance with instructions from the information processing unit 20. Further, in accordance with an instruction from the information processing unit 20, information based on the processing of the information processing unit 20 and information transmitted from the interface unit 50 are stored.

  The storage unit 40 is functionally composed of an imaging setting storage unit 410 and an image storage unit 420.

The imaging setting storage unit 410 stores imaging setting information. The imaging setting information includes a distance between the main lens ML and the sub lens array SLA, a focal length f _ML of the main lens, information for specifying an exposure time, an F value, a shutter speed, and the like as imaging parameters that can change during imaging.
The imaging setting storage unit 410 transmits imaging parameters to the imaging control unit 220.
In addition, the imaging setting storage unit 410 transmits imaging setting information of the imaging unit 10 to the image processing unit 210.

  The image storage unit 420 stores the image reconstructed by the image reconstruction device 30. The image storage unit 420 transmits the stored image to the I / O unit 510 and the display unit 520 of the interface unit 50.

  An interface unit (described as I / F unit in the figure) 50 is a configuration relating to an interface between the digital camera 1 and its user (user) or an external device, and includes an I / O unit 510, a display unit 520, And an operation unit 530.

  The I / O unit (Input / Output unit) 510 is physically composed of a USB (Universal Serial Bus) connector, a video output terminal, and an input / output control unit. The I / O unit 510 outputs the information stored in the storage unit 40 to an external computer, and transmits the information transmitted from the outside to the storage unit 40.

  The display unit 520 includes a liquid crystal display device, an organic EL (Electro Luminescence) display, and the like, and has a screen for inputting imaging parameters stored in the imaging setting storage unit 410 and a screen for operating the digital camera 1. indicate. In addition, the display unit 520 displays an image stored in the image storage unit 420.

  The operation unit 530 detects, for example, various buttons provided in the digital camera 1 and a touch panel provided in the display unit 520, information on operations performed on the various buttons and the touch panel, and the storage unit 40, the information processing unit 20, and the like. The information on the user operation is transmitted to the storage unit 40 and the information processing unit 20.

  As shown in FIG. 3B, the digital camera 1 captures an LFI, generates a reconstructed image RI reconstructed from the LFI, and outputs the reconstructed image RI.

Next, the configuration of the image reconstruction device 30 will be described with reference to FIG.
As shown in FIG. 4A, the image reconstruction device 30 includes an information processing unit 31a, a main storage unit 32, an external storage unit 33, an operation unit 34, a display unit 35, an input / output unit 36, And an internal bus 37.

  The information processing unit 31a includes a CPU (Central Processing Unit) and a RAM (Random Access Memory).

  The main storage unit 32 has the same physical configuration as the main storage device of the storage unit 40. The external storage unit 33 has the same physical configuration as the external storage device of the storage unit 40 and stores a program 38. The operation unit 34 has the same physical configuration as the operation unit 530. The display unit 35 has the same physical configuration as the display unit 520. The input / output unit 36 has the same physical configuration as the I / O unit 510. The internal bus 37 connects the information processing unit 31a, the main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, and the input / output unit 36.

  The information processing unit 31a, the main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, the input / output unit 36, and the internal bus 37 are internal to the information processing unit 20 of the digital camera 1. It may be a functional block realized by a circuit, the storage unit 40, and the interface unit 50.

  The image reconstruction device 30 copies the program 38 and data stored in the external storage unit 33 to the main storage unit 32, and the information processing unit 31 a uses the main storage unit 32 to execute the program 38. Then, processing for reconstructing an image to be described later is executed.

  The image reconstruction device 30 functions as an I / O unit 300, a storage unit 310, and an information processing unit 320 as shown in FIG.

The I / O unit 300 includes an operation unit 3010 that receives a user operation, an image acquisition unit 3030 that acquires an LFI from the image processing unit 210, and a shooting setting acquisition unit that acquires shooting setting information when the LFI is acquired. 3020 and an output unit 3040 that outputs an image reconstructed from the LFI.
In addition to the imaging parameters (the distance between the main lens ML and the sub-lens array SLA, the focal length f _ML of the main lens, the information specifying the exposure time, the F value, and the shutter speed), the shooting setting information includes: The existing values include the effective diameter ED of the main lens, the position information of each sub lens SL constituting the sub lens array SLA, and the distance between the sub lens array SLA and the imaging surface IE of the image sensor 120. Further, a distance c1 from the main lens ML to the sub lens array SLA, a distance c2 from the sub lens array SLA to the surface of the image sensor (imaging surface IE), a diameter of each sub lens, a relative position of each image sensor to the sub lens, Information.

  The storage unit 310 includes a reconstruction setting storage unit 3110, an MLB table storage unit 3120, and a reconstructed image storage unit 3130.

The reconstruction setting storage unit 3110 stores reconstruction setting information for reconstructing an image input by the user using the operation unit 3010. The reconfiguration setting information includes information indicating the specific contents of the reconfiguration process and reconfiguration parameters.
Here, a case where an image refocused with LFI is reconstructed will be described. At this time, the reconstruction setting information includes information for refocusing and information for specifying the distance between the focus point of the new image and the main lens ML.

When reconstructing an image, the MLB table storage unit 3120 stores an MLB table that is a table for designating a weight W _ML to be applied to each sub lens SL. The MLB table stored in the MLB table storage unit 3120 will be described later.

  The reconstructed image storage unit 3130 is a work area for temporarily storing a reconstructed image being generated, and stores the reconstructed image generated by the information processing unit 320.

  The information processing unit 320 includes the LFI acquired by the image acquisition unit 3030, the imaging setting information acquired by the imaging setting acquisition unit 3020, and the imaging setting information acquired by the imaging setting acquisition unit 3020, and the reconstruction image that matches the reconstruction setting stored in the reconstruction setting storage unit 3110. It is generated using the MLB table stored in the MLB table storage unit 3120. In order to execute such processing, the information processing unit 320 performs a prototype definition unit 3210, a focused reconstruction pixel selection unit 3220, a reaching point acquisition unit 3230, an MLB weight table acquisition unit 3240, and a sub lens selection unit 3250. And a corresponding pixel specifying unit 3260 and a pixel value adding unit 3270.

The function of each unit constituting the information processing unit 320 and the outline of the image reconstruction process executed by the information processing unit 320 will be described with reference to FIGS. 4B and 5.
Here, a case where an image is refocused to a position away from the main lens ML by a distance a1 will be described as an example.

The prototype definition unit 3210 of the information processing unit 320 sets a virtual subject OB on the reconstructed image to a plane (reconstruction plane RF) perpendicular to the optical axis of the main lens ML that is a distance a1 away from the main lens ML. To do. Then, an image template (original) corresponding to the subject OB is defined in accordance with the resolution designated by the reconstruction setting stored in the reconstruction setting storage unit 3110. The center of the prototype is assumed to be the optical axis OA.
A prototype of a reconstructed image is an image in which all pixel values are 0 (or not set).

  Next, the attention reconstruction pixel selection unit 3220 sequentially selects attention reconstruction pixels (corresponding to the attention site P of the subject OB) among the pixels (reconstruction pixels) constituting the reconstruction image.

  Further, the arrival point acquisition unit 3230 acquires a part (main lens blur center MLBC) where the principal ray from the attention part P reaches the sub lens array SLA. This process may be any known process for determining how light rays are projected from the subject through the lens, but here, a method described later is used.

Next, the MLB weight table acquisition unit 3240 acquires the MLB weight table corresponding to the shooting setting information and the position of the main lens blur center MLBC from the MLB table storage unit 3120.
Here, an outline of the processing for acquiring the main lens blur center MLBC and the MLB table stored in the MLB table storage unit 3120 will be described with reference to FIGS.

  First, an example of the main lens blur MLB projected on the sub lens array SLA will be described with reference to FIG. A region where the principal ray emitted from the region of interest P reaches the sub lens array SLA through the main lens ML is indicated by using a black circle (●) in FIGS. This part is called a sub-lens arrival point or a main lens blur center MLBC. When the optical axis of the main lens ML and the sub lens array SLA intersect perpendicularly, the main lens blur MLB can be approximated to a circle having the main lens blur center DMBC and a diameter of the main lens blur MLBL. The main lens blur MLB is a shaded area in FIGS. 6A and 2.

As can be seen from FIG. 5, the main lens blur diameter DMLB can be obtained by the following equation (1) based on the similarity of triangles.
DMLB = ED · a2 / b1 (1)
Here, the effective diameter ED is a known numerical value. The unknown numerical values a2 and b1 are calculated as follows.
b1 can be calculated from the following equation (2) using known numerical values a1 and _fML .

Further, a2 can be obtained by subtracting b1 calculated using the equation (2) from the known numerical value c1.
That is, the main lens blur diameter DMLB includes a distance a1 between the reconstruction surface RF and the main lens ML indicated by the reconstruction setting, an effective diameter ED indicated by the imaging setting information, and a distance c1 between the main lens and the sub lens array. This is a number determined by the focal length f _ML of the main lens.

The arrival point acquisition unit 3230 obtains the position of the main lens blur center MLBC (which sub-lens SL is located on which sub-lens SL).
Specific processing will be described later.

The sub lens SL including the main lens blur center MLBC is defined as a sub lens SL (0, 0). A coordinate system is defined with the sub lens SL (0, 0) as the center, and the Y axis is Y _ML and the X axis is X _ML (FIG. _6A ).
A sub lens SL (x _ml , y _ml ) that has moved by x _ml in the y-axis direction and y _ml in the Y-axis direction around the sub lens SL (0, 0) is defined as a sub-lens SL (x _ml , y _ml ).

FIG. 6B shows an example in which the sub lens SL (0, 0) is enlarged and divided into partial areas SASL. In the example of FIG. 6B, the horizontal axis is i _ML , the vertical axis is j _ML , and the upper left is the origin, which is divided into 5 × 5 squares. If each partial area SASL is expressed as SASL (i _ML , j _ML ), the area where the main lens blur center MLBC exists (central area CA) in FIG. 6B is the partial area SASL (2, 3). is there. At this time, (2, 3) is expressed as the coordinates of the center area CA (center coordinates CP).

When the central coordinate CP and the main lens blur diameter DMLB are determined, the area occupied by the main lens blur MLB on each sub-lens SL (x _ml , y _ml ) is determined. The light emitted from the site of interest P can be approximated by being projected almost evenly on the main lens blur MLB. Therefore, the area occupied on the sub-lens SL there is a main lens blur MLB is how reached either on the corresponding target sites that sub lens the light emitted from the P, be approximated as corresponding to the sum of the light I can do it.
Therefore, in the present embodiment, the area of the main lens blur MLB occupying the sub lens SL is defined as a coefficient (weight W _ML ) indicating the strength of correspondence between each sub lens SL and the target region P. The weight W _ML (sub image weight) corresponds to the strength of the correspondence between the sub image corresponding to the sub lens SL and the reconstructed pixel corresponding to the target region. When each sub lens SL has substantially the same area, the weight W _ML can be defined as a ratio (a real number from 0 to 1) occupied by the main lens blur on the sub lens. In the following description, it is assumed that the weight W _ML is a real number from 0 to 1.
When each sub lens has a different area, the area occupied by the main lens blur MLB on the sub lens may be set as the weight W _ML from the position information of each lens.

When the distance a1 from the reconstruction surface RF to the center of the main lens is sufficiently larger than the distance b1 and the distance a2, the main point is irrespective of the position of the region of interest P (distance x from the optical axis of the main lens ML). It can be considered that the shape of the lens blur MLB does not change when viewed from the center of the main lens blur MLBC. That is, the weight W _ML corresponding to each sub-lens SL can be approximated by being determined by the relative position from the sub-lens including the center coordinate CP, regardless of the absolute position of the sub-lens SL including the main lens blur center MLBC.
Therefore, in the present embodiment, a table that defines a set of main lens blur weights W _ML corresponding to the parameter a that is an index indicating the distance a1 to the reconstruction surface and the center coordinates CP (i _ML , j _ML ). (MLB table) is defined and stored in the MLB table storage unit 3120 in advance.

  The parameter a is an index indicating a predetermined range of the distance a1 (the distance between the reconstruction surface RF and the center of the main lens). For example, the parameter a is an integer from 0 to 7, and the parameter a is 7, the distance a1 is infinite to 100 m, and 0 is the distance a1 from 1 to 10 m. Set as there is. The correspondence between the parameter a and the distance a1 is obtained by experiment and is stored in the reconstruction setting storage unit 3110 in advance. Alternatively, it may be a numerical value that can be arbitrarily designated by the user.

As shown in FIG. 7, the MLB table includes an MLB index table and an MLB weight table. The MLB index table (FIG. 7A) is a table that stores the central coordinates CP (i _ML , j _ML ) and the parameter a and an index indicating an ML weight list corresponding to the center coordinate CP (i _ML , j _ML ).
In the example of FIG. 7A, the parameter a takes Na values of 1 to Na, and the case where the center coordinate CP is any one of (0, 0) to (4, 4) corresponds to each case. Indexes (index # 01 to index #NaN) indicating the MLB weight table to be stored are stored in association with each other.

The MLB weight table stores the weight W _ML that can be defined for the value of the parameter a corresponding to the index and the center coordinate CP in association with each of the sub-lenses SL at the relative positions to be considered.
In the example of FIG. 7B, the MLB weight table indicates that the relative position with respect to the sub-lens SL at the relative position to be considered, that is, the central sub-lens (0, 0) is (+ X _ml Max, + Y _ml Max). The weight W _ML is registered for each sub lens SL (x _ML , y _ML ) that is (−X _ml Max, −Y _ml Max) (here, (−6, −6) to (6, 6)). It is a table. Note that X _ml Max and Y _ml Max are natural numbers determined in advance by experiment, and when the closest subject that can be set is photographed at the focal length corresponding to the maximum zoom, that is, the main lens blur diameter becomes the largest. This corresponds to the number of sub-lenses covered by the main lens blur.

  In the example of FIG. 7, when the target center coordinate CP is (2, 3) and the distance a1 indicated by the reconstruction setting corresponds to the parameter a = 5, the index of the MLB weight table used is #k. It is.

In the MLB table, the maximum value (here, 1) of the weight W _ML is associated with the sub-lens SL that is within a predetermined range (a range that is completely within the main lens blur diameter DMLB) from the main lens blur center MLBC. The minimum value of the weight W _ML for sub-lenses SL in a position outside a predetermined range (the main lens fully disengaged position to a blur diameter DMLB) (in this case 0) is associated. For sub-lenses in the middle position, a middle value is defined. At this time, when the center coordinate CP is closer to the sub lens, the portion occupied by the main lens blur MLB becomes larger, and therefore a larger intermediate value is associated.
Further, as the distance a1 to the reconstruction surface is closer, the main lens blur diameter DMLB becomes larger, so that a larger weight W _ML is obtained for the farther sub-lens SL. Note that the main lens blur diameter DMLB also increases when the focal length f _ML is small, or when the effective diameter ED of the main lens ML is large. Therefore, if the other conditions are the same, a larger weight W _{ML for the} far sub lens SL. Are associated.

The MLB weight table acquisition unit 3240 obtains, from the MLB index table, the index of the MLB weight table that matches the parameter a that matches the distance a1 and the center coordinates CP (i _ML , j _ML ) calculated by the arrival point acquisition unit 3230. get. Then, the MLB weight table indicated by the index is acquired from the MLB table storage unit 3120.

Then, the sub lens selection unit 3250 selects one of the sub lenses SL (+ X _ml Max, + Y _ml Max) to (−X _ml Max, −Y _ml Max) as the target sub lens.

  Then, the corresponding pixel specifying unit 3260 specifies a part (corresponding to a corresponding pixel on the LFI) that reaches the imaging surface IE from the target part P through the target sub lens. This process may be executed using any known ray tracing method, but here, a method described later is used.

The sub-lens selection unit 3250 and the corresponding pixel specifying unit 3260 sequentially set the sub-lens SL (+ X _ml Max, + Y _ml Max) to (−X _ml Max, −Y _ml Max) as the sub-lenses of interest, and on the LFI thereof. The corresponding pixel and the weight W _ML corresponding to the sub lens defined by the MLB weight table are extracted.

When the corresponding pixels have been extracted for all of the sub lenses SL (+ X _ml Max, + Y _ml Max) to (−X _ml Max, −Y _ml Max), the pixel value adding unit 3270 has the pixel value of the corresponding pixel of each sub lens. Then, the weight _WML corresponding to each sub lens is multiplied and averaged to obtain the pixel value of the reconstructed pixel corresponding to the target region P.

  When the pixel value adding unit 3270 determines the pixel value of the reconstructed pixel of interest, the pixel value is temporarily stored in the reconstructed pixel storage unit 3130. Then, the reconstructed pixel selecting unit 3220 to the pixel value adding unit 3270 sequentially determines all reconstructed pixels as the reconstructed pixels of interest, and generates a reconstructed image.

Next, processing executed by the digital camera 1 and the image reconstruction device 30 will be described with reference to flowcharts (FIGS. 8 to 10).
When the user photographs the subject OB using the imaging unit 10, the digital camera 1 starts the image output process shown in FIG.

  In the image output process, first, the image processing unit 210 of the digital camera 1 receives information from the imaging unit 10 and generates an LFI. The image processing unit 210 transmits the acquired LFI to the image acquisition unit 3030 of the image reconstruction device 30. Thus, the image acquisition unit 3030 acquires the LFI (step S101).

The image processing unit 210 further acquires imaging setting information (including the focal length f _ML ) obtained by capturing the light field image from the imaging setting storage unit 410. The image processing unit 210 adds information such as the physical configuration of the imaging unit 10 to the acquired imaging setting information and transmits the information to the imaging setting acquisition unit 3020 of the image reconstruction device 30 as imaging setting information. Thus, the shooting setting acquisition unit 3020 acquires shooting setting information (step S102).

  Next, the image reconstruction device 30 starts the image reconstruction process shown in FIG. 9 (step S103). In the image reconstruction process, first, the prototype defining unit 3210 acquires parameters used for generating a reconstructed image stored in the reconstruction setting storage unit 3110 (step S201). Here, the distance a1 of the reconstructed image from the main lens ML to the reconstructed surface RF, the number of pixels in the reconstructed image, and the part to which each pixel corresponds (at the intersection on the main lens optical axis and the reconstructed surface RF). The reconfiguration setting includes information for instructing to generate a refocused reconstructed image, and the like.

  Next, based on the reconstruction setting, the prototype definition unit 3210 has the prototype of the reconstructed image (the pixel values of the reconstruction pixels are all on the reconstruction plane RF located at a distance a1 from the optical center of the main lens ML). 0 image) is defined (step S202).

  Next, with the counter variable set to i, the focused reconstruction pixel selection unit 3220 focuses on the i-th reconstruction pixel (corresponding to the focused region P in FIG. 5) (step S203).

  Then, the arrival point acquisition unit 3230 specifies the position of the point (main lens blur center MLBC) that reaches the sub lens array SLA from the target reconstruction pixel through the optical center of the main lens ML (step S204).

Specifically, in the main lens blur center MLBC, the distance in the x-axis direction between the intersection (reconstruction center) of the reconstruction surface RF and the optical axis and the target region P is x, and the optical axis of the main lens ML and the main The distance d2 in the x-axis direction of the lens blur center MLBC is calculated by the following equation (3).
d2 = x · c1 / a1 (3)
The same calculation can be performed for the y-axis. In this way, the position of the microlens blur center MLBC is specified from the position of the attention site P.

Next, on which sub-lens the main lens blur center MLBC is located. Further, the position on the sub lens SL is obtained from the position and size information of each sub lens indicated by the shooting setting information as the center coordinates CP (i _ML , j _ML ).

Then, the MLB weight table acquisition unit 3240 displays the center coordinate CP (i _ML , j _ML ) indicated by the MLB index table stored in the MLB table storage unit 3120, and the parameter a corresponding to the distance a1 indicated by the reconstruction setting, , MLB weight table matching the above is acquired (step S205).

  Then, the sub lens selection unit 3250 to the pixel value addition unit 3270 execute a process of determining the pixel value of the reconstructed pixel of interest using the MLB weight table (pixel value addition process, here, the pixel value addition process 1) ( Step S206).

The pixel value addition process (pixel value addition process 1) executed in step S206 will be described with reference to FIG. In the pixel value addition processing 1, first, the sub-lens selection unit 3250 selects a sub-lens (corresponding sub-lens) corresponding to a reconstructed pixel with a sub-lens SL including a main lens blur MLB (sub-lens whose MLB weight table has a non-zero weight). ) ( Step S301).

Next, the sub-lens selection unit 3250 selects the j-th corresponding sub-lens among the sub-lenses extracted using j as a counter variable as a target sub-lens ( step S302).

Then, the weight W _ML defined for the relative position (X _ML , Y _ML ) of the _target sub-lens when the lens including the main lens blur center MLBC is (0, 0) is acquired from the MLB weight table. (Step S303).

  Next, the corresponding pixel specifying unit 3260 extracts a pixel corresponding to the target reconstruction pixel (corresponding sub pixel, pixel corresponding to the reaching point PE) from among the pixels constituting the sub image corresponding to the target sub lens, The pixel value is acquired (step S304).

Specifically, the reaching point PE is calculated by the following procedure.
Using the subject distance a1 and the distance b1 in the following formula (4), the distance x ′ between the point (imaging point PF) at which the site of interest P forms an image through the main lens ML and the optical axis OA is calculated x ′ = X · b1 / a1 (4)
Furthermore, the distance d from the optical axis OA to the principal point of the j-th sub lens SL, the distance x ′ calculated using the above equation (4), the distance c2 from the sub lens array SLA to the imaging surface IE, and A distance x ″ between the arrival point PE and the optical axis OA is calculated by using the distance a2 from the imaging plane MIP to the sub lens array SLA in the following equation (5).

The same calculation is performed in the Y-axis direction, and the position of the reaching point PE is specified. Then, using the position information of the imaging device shown imaging setting information to identify the corresponding picture element in a destination point PE, and obtains the image pixel value. In this case, the position of the sub lens array SLA and the imaging surface IE is adjusted so that the corresponding sub pixel can be identified as one pixel to one pixel for any of the assumed distances a1 from the reconstruction surface. It shall be.

Next, it is determined whether or not the processing for extracting the corresponding sub-pixel has been executed for all the corresponding sub-lenses extracted in step S301 (step S305).
If it is determined that there is an unprocessed sub lens SL (step S305; NO), the counter variable j is incremented (step S306), and the process is repeated from step S302 with the next sub lens as the target sub lens.

On the other hand, if it is determined that the corresponding sub-pixel has been extracted for all the corresponding sub-lenses (step S305; YES), the pixel value adding unit 3270 determines the value of each corresponding sub-pixel and the corresponding sub-lens corresponding to the corresponding sub-pixel. The weighted average value is calculated using the defined weight W _ML and is set as the pixel value of the reconstructed pixel of interest (step S307).
Then, the pixel value addition process 1 ends.

Returning to FIG. 9, when the pixel value of the target reconstructed pixel is determined by the pixel value addition process, it is determined whether or not the process of determining the pixel value for all the reconstructed pixels has been executed (step S207).
If it is determined that pixel values have not been determined for all the reconstructed pixels (step S207; NO), the counter variable i is incremented (step S208), and the process is repeated from step S202 for the next reconstructed pixel of interest.
On the other hand, if it is determined that the pixel values have been determined for all the reconstructed pixels (step S207; YES), the output unit 3040 outputs the reconstructed pixels to the image storage unit 420 (step S209), and the image reconstruction process ends. .

Returning to FIG. 8, when the generation of the reconstructed image is completed in the image reconstruction process (step S103), the digital camera 1 outputs the reconstructed image stored in the image storage unit 420 to the display unit 520 (step S104). . Alternatively, it may be output to an external device using the I / O unit 510.
Then, the digital camera 1 and the image reconstruction device 30 end the image output process.

  As described above, according to the image reconstruction device 30 of the present embodiment, a reconstructed image is generated using the MLB weight table corresponding to the setting (shooting parameters) at the time of shooting. Therefore, it is possible to generate a reconstructed image with high image quality that is reconstructed with high accuracy with a small amount of calculation.

At this time, since the pixel value is determined from a plurality of MLB weight tables using a table corresponding to the focal length of the main lens, a highly accurate reconstructed image reflecting the focal length of the main lens at the time of shooting can be obtained with a small amount of calculation. Can be generated.

  Further, since the pixel value is determined using the MLB weight table corresponding to the parameter a corresponding to the distance between the reconstruction surface and the main lens, there are few reconstructed images with high accuracy due to the weight matching the position of the reconstruction surface. Can be generated with computational complexity.

  Further, the pixel values are averaged using an MLB weight table corresponding to the center coordinates CP (corresponding to the main lens center MLBC). Therefore, it is possible to generate a reconstructed image with higher accuracy according to the position where the light emitted from each of the reconstructed pixels reaches the sub lens with a small amount of calculation.

  Since the reconstructed image is generated using the MLB weight table corresponding to these parameters, how much light from the reconstructed pixel has reached which sub-lens and the weight corresponding to the size of the corresponding sub-lens. The pixel value of the reconstructed pixel can be determined by weighted addition of the pixels. Therefore, the weight corresponding to many parameters such as the position of the main lens, the focal length, the position of the sub lens, the arrival point, etc. can be obtained with a small amount of calculation. By using this weight, a highly accurate reconstructed image can be generated with a small amount of calculation.

  In this MLB weight table, a small weight is defined for a sub lens located at a predetermined distance (corresponding to the main lens blur diameter, main distance) from the center coordinate CP. The main lens blur diameter increases as the distance to the reconstruction surface decreases and as the focal length of the main lens decreases. On the other hand, it increases as the effective diameter of the main lens increases. For this reason, the MLB weight table is set so that the main table diameter decreases as either the distance to the reconstruction surface or the focal length of the main lens decreases or the effective diameter increases. By using such an MLB weight table, a highly accurate reconstructed image generated using a weight corresponding to the main lens blur diameter can be generated with a small amount of calculation.

  In addition, according to the digital camera 1 of the present embodiment, since an image is reconstructed using the image reconstruction device 30, a light field image is taken, and precise reconstruction pixels are reduced from the light field image with a small amount of calculation. Can be generated, displayed and recorded.

(Embodiment 2)
Next, the image reconstruction device 31 according to the second embodiment of the present invention will be described. The image reconstruction device 31 is included in the digital camera 2 shown in FIG.
Compared to the digital camera 1 according to the first embodiment, the digital camera 2 is different in that the imaging unit is the imaging unit 11 having the configuration illustrated in FIG. 12 and the image reconstruction device 31 performs image reconstruction processing. .

As shown in FIG. 12, the optical device 113 of the imaging unit 11 forms an image of light from the target region P on a plane (sub-lens imaging plane SIP) in which the sub-lens array SLA is different from the imaging plane IE of the image sensor 120. It is characterized by making it. Therefore, for one region of interest P, region that reaches the imaging surface IE through the respective sub-lens (sub-lens blur SLB, oval white 1 2) may not fit within one pixel . Therefore, the pixel value of the target reconstruction pixel is determined using a table (SLB table) for performing weighted addition for each pixel included in the sub-lens blur SLB.

  The physical configuration of the image reconstruction device 31 is the same as that of the image reconstruction device 30 according to the first embodiment.

As illustrated in FIG. 13, the image reconstruction device 31 functions as an I / O unit 300, a storage unit 311, and an information processing unit 321.
The I / O unit 300 is the same as the I / O unit 300 of the image reconstruction device 30 according to the first embodiment. However, the shooting setting information acquired by the shooting setting acquisition unit 3020 includes the diameter of each sub lens (sub lens diameter DSL), the distance c2 between the sub lens array SLA and the imaging surface IE, and the focal length f _{SB of} each sub lens. And information indicating that. The focal lengths f _SB of the sub lenses SL are all common, and are invariable values unique to the digital camera 2.

The storage unit 311 further includes an SLB table storage unit 3140 in addition to the storage unit 310 of the image reconstruction device 30 according to the first embodiment.
The SLB table storage unit 3140 stores an SLB table that is a table that defines the weight W _SL applied to the corresponding pixel corresponding to the sub lens. The SLB table will be described later.

  The information processing unit 321 includes the LFI acquired by the image acquisition unit 3030, the shooting setting information acquired by the shooting setting acquisition unit 3020, and the shooting setting information acquired by the image setting unit 3020, and the reconstruction image that matches the reconstruction setting stored in the reconstruction setting storage unit 3110. It is generated using the MLB table stored in the MLB table storage unit 3120 and the SLB table stored in the SLB table storage unit 3140. In order to execute such processing, the information processing unit 321 performs a prototype definition unit 3210, a focused reconstruction pixel selection unit 3220, a reaching point acquisition unit 3230, an MLB weight table acquisition unit 3240, and a sub lens selection unit 3250. An imaging plane arrival point acquisition unit 3280, an SLB weight table acquisition unit 3290, a corresponding pixel specification unit 3261, and a pixel value addition unit 3271.

  The functions of the prototype definition unit 3210 to the sub lens selection unit 3250 are the same as the parts of the same name in the information processing unit 320 according to the first embodiment.

The imaging plane arrival point acquisition unit 3280 acquires the arrival point PE at which the principal ray from the target site P passes through the target sub lens and reaches the imaging plane IE using the above-described equation (5). Then, coordinates (i _SL , j _SL ) indicating where the arrival point PE is on the image sensor are specified from the position information of the image sensor included in the image setting information.

Then, the SLB weight table acquisition unit 3290 refers to the SLB table stored in the SLB table storage unit 3140 and matches the parameter a corresponding to the distance a1 indicated by the reconstruction setting and the coordinates (i _sL , j _sL ). The SLB weight table to be acquired is acquired.

Here, the sub lens blur SLB, the coordinates (i _SL , j _SL ), and the SLB table stored in the SLB table storage unit 3140 will be described with reference to FIGS.

  First, an example of the sub-lens blur SLB projected on the imaging surface IE will be described with reference to FIG. A region where the principal ray emitted from the target region P reaches the imaging surface IE through the main lens ML and the target sub lens is indicated by a black circle (●) in FIG. This portion is referred to as a reaching point PE or a sub lens blur center SLBC. The sub-lens blur SLB can be approximated to a circle whose diameter is the sub-lens blur diameter DSLB centered on the sub-lens blur center SLBC.

An image sensor including the sub lens blur center SLBC is assumed to be PS (0, 0). A coordinate system is defined around PS (0, 0), and the Y axis is Y _SL and the X axis is X _SL .
Then, an image sensor PS that has moved by _xSl in the Y-axis direction and _ySl in the Y-axis direction around PS (0,0) is _defined as PS ( _xsl , _ysl ).

The sub-lens blur diameter can be defined as follows.
First, the distance b2 between the sub-lens array SLA and the sub-lens imaging surface SIP is obtained using the following equation (6) with a known numerical value as a variable.
b2 = a2 · f _SB / (a2−f _SB ) (6)

Then, the distance c3 between the sub-lens imaging surface SIP and the imaging surface IE is obtained from the obtained b2 and the distance c2 that is a known numerical value using the following equation (7).
c3 = c2-c3 (7)

Furthermore, from the distance b2 obtained by Expression (6), the distance c3 used in Expression (7), and the diameter of the corresponding sublens (sublens diameter DSL) indicated by the imaging setting information, Expression (8) is used. The sub lens blur diameter DSLB is calculated.
DSLB = DSL · c3 / b2 (8)

That is, the sub lens blur diameter DSLB is a number determined by the distance a1 between the reconstruction surface RF and the main lens ML indicated by the reconstruction setting, the focal length f _ML of the main lens, the focal length f _SB of the sub lens, and the like. .

FIG. 14B shows a state in which the imaging element PS (0, 0) is enlarged and divided into partial areas SAPS. In the example of FIG. 14B, the image sensor PS is divided into 5 × 5 squares with the horizontal axis as i _SL , the vertical axis as j _SL , and the upper left as the origin. Assuming that each partial area SAPS is expressed as SAPS (i _SL , j _SL ), an area (arrival area EA) in which the sub-lens blur center SLBC (corresponding to the arrival point PE) exists in FIG. 2, 1). At this time, (2, 1) is expressed as the coordinates of the arrival area EA (arrival coordinates EP).

When the arrival coordinate EP and the sub-lens blur diameter DSLB are determined, the area occupied by the sub-lens blur SLB on each image sensor PS (x _sl , y _sl ) is determined. In this embodiment, the light emitted from the site of interest P is approximated when projected evenly on the sub-lens blur SLB. Therefore, it is assumed that the area occupied by the sub-lens blur SLB on the image sensor PS corresponds to how much the light emitted from the corresponding target site P reaches the image sensor, or the sum of the light. I can do it.

Therefore, in the present embodiment, the area of the sub lens blur SLB occupying the image sensor PS is a coefficient indicating the strength of correspondence between each image sensor PS (and the sub pixel corresponding to the image sensor) and the target site P. (Weight W _SL ). The weight W _SL corresponds to the strength of correspondence (sub-pixel weight) between the reconstructed pixel corresponding to the image sensor PS and the corresponding pixel corresponding to the region of interest. When each image sensor PS has substantially the same area, the weight W _SL can be defined as a ratio (a real number from 0 to 1) occupied by the sub-lens blur SLB on the image sensor PS. In the following description, it is assumed that the weight W _SL is a real number from 0 to 1.

Defines a table (SLB table) defining a set of sub lens blur weights W _SL corresponding to a parameter a which is an index indicating the distance a1 to the reconstruction plane and the arrival coordinates EP (i _sL , j _sL ). And stored in advance in the SLB table storage unit 3140.

The SLB table includes an SLB index table and an SLB weight table as shown in FIG. The SLB index table (FIG. 15A) is a table that defines an index indicating an SL weight list corresponding to the arrival coordinates EP (i _sL , j _sL ) and the parameter a.
In the example of FIG. 15A, the SLB index table takes Na values where the parameter a is 1 to Na and each of the arrival coordinates EP is any of (0, 0) to (4, 4). The index (index # 01 to index #NaN) indicating the SLB weight table corresponding to the case is stored in association with each other.

In the SLB weight table, the imaging element PS (that is, the sub-pixel corresponding to this) at the relative position to consider the weight W _SL determined based on the value of the parameter a corresponding to the index and the arrival coordinate EP. Are stored in association with each other.

In the example of FIG. 15B, the SLB weight table indicates that the relative position with respect to the sub-pixel at the relative position to be considered, that is, the pixel (arrival pixel) where the arrival coordinate EP is located is (+ X _sl Max, + Y _sl Max). It is a table registered for each sub-pixel (X _SL , Y _SL ) that is (−X _sl Max, −Y _sl Max) (here, (−4, −4) to (4, 4)). Note that X _sl Max and Y _sl Max are natural numbers determined in advance by experiment, and correspond to the number of image sensors PS covered by the sub lens blur when the sub lens blur diameter is the largest in the setting.

The SLB weight table acquisition unit 3290 sets an index of the SLB weight table that matches the parameter a that matches the distance a1 and the arrival coordinates EP (i _SL , j _SL ) calculated by the imaging surface arrival point acquisition unit 3280 as the SLB index. Get from table. Then, the SLB weight table indicated by the index is acquired from the SLB table storage unit 3140.

Corresponding pixel specifying unit 3261, all extracts sub-pixels defined in SLB weight table SLB weight table acquisition unit 3290 acquires, temporarily stores in association with the weight _{W SL.}

In the SLB table, the maximum value (here, 1) of the weight W _SL is associated with the image sensor PS that is within a predetermined range (a range that is completely within the sub-lens blur diameter DSLB) from the arrival point PE. The minimum value of the weight W _SL for image sensor PS in a position outside a predetermined range (sub-lens blur diameter DSLB from a fully disengaged position) (here 0) is associated. An intermediate value is defined for the image sensor PS at an intermediate position. At this time, when the arrival position is closer to the image sensor PS, the portion occupied by the sub-lens blur SLB becomes larger, and therefore a larger intermediate value is associated.
Further, the smaller the distance a1 from the reconstruction surface, the larger the main lens blur diameter DMLB, so that a larger weight W _SL is obtained for the farther sub-lens SL. Even when the focal length f _ML is large, the sub-lens blur diameter DSLB is large. Therefore, when the other conditions are the same, the farther imaging element PS is associated with a larger weight W _SL .

The corresponding pixel specifying unit 3261 from the sub-lens selection unit 3250 sequentially sets all the corresponding sub-lens as the target sub-lens, and sets pixel values, weights W _ML , and weights W _SL for all the sub-pixels corresponding to the target region P. , Are associated.

Then, the pixel value adding unit 3271 is the pixel value of the reconstructed pixels corresponding to the attention site P, determined by averaging weighted sum by using the weight W _ML and the weight W _SL the extracted pixel value in the above process.

  Next, processing executed by the digital camera 2 and the image reconstruction device 31 will be described with reference to a flowchart (FIG. 16).

  The digital camera 2 and the image reconstruction device 31 execute the image output process shown in FIG. 8 and the image reconstruction process shown in FIG. 9 as in the first embodiment.

  In step S206 of the image reconstruction process, the image reconstruction device 31 executes the pixel value addition process 2 illustrated in FIG. 16 as the pixel value addition process.

  In the pixel value addition process 2, first, steps S401 to S403 are executed in the same manner as steps S301 to S303 of the pixel value addition process 1 shown in FIG.

  Next, the imaging surface arrival point acquisition unit 3280 acquires the position of the arrival point PE using Expression (5) (step S404).

Then, the SLB weight table acquisition unit 3290 acquires an SLB weight table that matches the arrival coordinate EP (i _SL , j _SL ) and the parameter a corresponding to the distance a1 indicated by the reconstruction setting (step S405).

Then, the corresponding pixel specifying unit 3261 acquires the pixel value of the corresponding pixel (the pixel whose corresponding weight W _SL is not 0) defined in the SLB weight table and the weight W _SL (step S406).

Next, it is determined whether or not the processing for obtaining the pixel value of the corresponding pixel and its weight has been executed for all the corresponding sub-lenses extracted in step S401 (step S407).
If it is determined that there is an unprocessed sub lens SL (step S407; NO), the counter variable j is incremented (step S408), and the process is repeated from step S402 with the next sub lens as the target sub lens.

On the other hand, if it is determined that the corresponding sub-pixels have been extracted for all the corresponding sub-lenses (step S407; YES), the pixel value adding unit 3271 sets the value of each corresponding sub-pixel to the corresponding sub-lens corresponding to the corresponding sub-pixel. and the weight W _ML defined, and the weight W _SL corresponding to each sub-pixel, and calculates the weight average value is used to the pixel value of the target reconstructed pixel (step S409).

  As described above, according to the image reconstruction device 31 of the present embodiment, the reconstruction pixel is configured in consideration of not only the main lens blur but also the sub lens blur. Even if it is, it is possible to generate a reconstructed image with high accuracy. In addition, since the weight considering the sub-lens blur is calculated using the table, the necessary calculation amount for generating a high-quality reconstructed image considering the sub-lens blur is small.

(Modification)
The present invention is not limited to the above embodiment, and various modifications can be made.
For example, in Embodiments 1 and 2, the focal length f _ML of the main lens and the effective diameter ED of the main lens did not change. However, the present invention is not limited to this, and either or both of the focal length f _ML and the effective diameter ED may be varied.
A configuration example of the optical device 110 in such a case is shown in FIG. The optical device 110 in FIG. 17 can change the effective diameter ED by the user operating the diaphragm 112.

FIG. 18A shows an MLB index table when both the focal length f _ML of the main lens and the effective diameter ED of the main lens are variable. Since W _ML varies depending on the effective diameter ED and the focal length f _ML , the MLB index table corresponds to (i _ML , j _ML ) and parameter ED (a predetermined range of effective diameter ED) in addition to parameter a. Index of the MLB weight table corresponding to the case where each of the parameter #f _ML (natural number corresponding to a predetermined range of the focal length f _ML ) is changed.

On the other hand, the weight W _SL will vary depending on the focal length f _ML, it is constant even when the effective diameter ED of the main lens changes. Therefore, the SLB index table does not include the parameter ED as shown in FIG.
According to such a configuration, even when the main lens is a zoom lens or has a diaphragm, a reconstructed image can be generated in consideration of the focal length and diaphragm at the time of shooting. Therefore, the accuracy of the reconstructed image does not decrease even when the focal length or the aperture changes.

In the above-described embodiment, the weight W _ML and the weight W _SL are arbitrary numbers, but the W _ML and W _SL may take either 0 or 1. In this case, for example, the weight is set to 1 when the area of the main lens blur on the sub lens is equal to or larger than a predetermined threshold (for example, half or more of the area of the sub lens), and is set to 0 otherwise. Can do. The weight W _SL can be similarly defined. According to such a configuration, it is possible to reduce the calculation amount for generating the reconstructed image.

In the above embodiment, the pixel values of the corresponding pixels are averaged using weights when calculating the pixel values. However, the method of calculating the pixel value using the weight W _ML or the weight W _SL is not limited to this, and if the weight obtained for the corresponding pixel increases, the pixel value of the pixel greatly affects the pixel value of the calculated pixel. The pixel value can be obtained by any method that gives

The MLB table is not limited to the configuration exemplified in the above description. The MLB table, large weight W _ML for sub-lens SL is associated in the position close to the center coordinates CP, associated small weight W _ML for sub-lenses SL in the distant position. Further, as the distance a1 to the reconstruction surface is closer, the main lens blur diameter DMLB becomes larger, so that a larger weight W _ML is obtained for the farther sub-lens SL. In addition, when the focal length f _ML is small, or when the effective diameter ED of the main lens ML is large, and the other conditions are the same, it is desirable that a larger weight W _ML is associated with the far sub lens SL. It is.
The MLB table can be replaced with an arbitrary table in which the weighting coefficient and the sub lens SL are associated with each other under such conditions.

Further, the SLB table is not limited to the configuration exemplified in the above description. The SLB table associated weight is larger W _SL for image sensor PS in the position close to the arrival coordinates EP (predetermined range), it is associated a small weight W _SL for image sensor PS in the distant position. Further, as the distance a1 to the reconstruction surface is further increased, the sub lens blur diameter DSLB is increased, so that a larger weight W _{S L} is obtained for the farther sub lens SL. When the focal length f _ML is large and the other conditions are the same, a larger weight W _SL is associated with the farther image sensor PS.
The SLB table can be replaced with an arbitrary table in which the weighting coefficient and the image sensor PS are associated with each other under such conditions.

  In the above embodiment, the image is described as a grayscale image. However, the image to be processed according to the present invention is not limited to a grayscale image. For example, the image may be an RGB image in which three pixel values of R (red), G (green), and B (blue) are defined for each pixel. In this case, the pixel value is similarly processed as an RGB vector value. Alternatively, the above processing may be performed using each of R, G, and B values as independent grayscale images. According to this configuration, a reconstructed image that is a color image can be generated from a light field image that is a color image.

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  The central part that performs processing for image reconstruction composed of the information processing unit 31a, the main storage unit 32, the external storage unit 33, and the like is realized using a normal computer system, not a dedicated system. Is possible. For example, a computer program for executing the above operation is stored and distributed in a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. You may comprise the center part which performs the process for image reconstruction. Further, the image corrector may be configured by storing the computer program in a storage device included in a server device on a communication network such as the Internet and downloading it by a normal computer system.

  When realizing the functions of the image reconstruction device by sharing the OS (operating system) and application programs, or by cooperating the OS and application programs, store only the application program portion in a recording medium or storage device. Also good.

  It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board (BBS: Bulletin Board System) on a communication network, and the computer program may be distributed via the network. The computer program may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

  It is also possible to store either or both of the MLB table and the SLB table in an external device, transmit an index (such as parameter a) through the communication device, and acquire the MLB weight table from the external device.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to the specific embodiment which concerns, This invention includes the invention described in the claim, and its equivalent range It is. Hereinafter, the invention described in the scope of claims of the present application will be appended.

(Appendix 1)
A light field image acquisition unit for acquiring a light field image composed of a plurality of sub-images obtained by photographing a subject from different viewpoints;
A shooting parameter acquisition unit for acquiring a shooting parameter for acquiring the light field image;
A model definition unit that defines a model of a reconstructed image that is an image of a subject reconstructed from the light field image;
A table defining image weights, which are coefficients indicating the strength of correspondence between a reconstructed pixel that is a pixel constituting a reconstructed image defined by the template definition unit and at least one of the sub-images, A main table acquisition unit for acquiring a main table corresponding to the imaging parameters acquired by the imaging parameter acquisition unit;
A corresponding pixel extracting unit that extracts a corresponding pixel that is a pixel corresponding to a reconstructed pixel constituting a reconstructed image defined by the template defining unit from sub image pixels constituting the plurality of sub-images;
The pixel value of the reconstructed pixel, the pixel value of the corresponding pixel extracted by the corresponding pixel extracting unit, the reconstructed pixel defined by the main table acquired by the main table acquiring unit, and the sub-image to which the corresponding pixel belongs A reconstructed image generating unit that generates the reconstructed image by obtaining based on the image weight corresponding to
A reconstructed image generating apparatus comprising:

(Appendix 2)
The light field image is captured using a main lens that captures light rays from the subject, and a plurality of sub-lenses corresponding to each of the plurality of viewpoints that further capture light rays captured by the main lens,
The shooting parameter includes at least a focal length of the main lens,
The reconstructed image generating apparatus according to Supplementary Note 1, wherein

(Appendix 3)
A reconstruction setting acquisition unit that acquires a reconstruction setting including information indicating a distance between the surface including the reconstructed image and the main lens;
The main table acquisition unit acquires a main table according to the imaging parameter acquired by the imaging parameter acquisition unit and the reconstruction setting acquired by the reconstruction setting acquisition unit.
The reconstructed image generation apparatus according to Supplementary Note 2, wherein

(Appendix 4)
Corresponding to the reconstruction pixel from the imaging parameter acquired by the imaging parameter acquisition unit, the reconstruction setting acquired by the reconstruction setting acquisition unit, and the relative position of the main lens and the sub lens stored in advance. A sub-lens arrival point acquisition unit that acquires a sub-lens arrival point, which is a point where a light beam emitted from a portion of the subject that reaches the sub lens
The main table acquired by the main table acquisition unit has a small image weight of the sub image corresponding to the sub lens located at a predetermined distance or more from the sub lens arrival point acquired by the sub lens arrival point acquisition unit,
The reconstructed image generating apparatus according to Supplementary Note 3, wherein

(Appendix 5)
The predetermined distance related to the image weight of the main table acquired by the main table acquisition unit as the distance between the surface including the reconstructed image included in the reconstruction setting acquired by the reconstruction setting acquisition unit and the main lens is smaller Becomes larger,
The reconstructed image generating apparatus according to Supplementary Note 4, wherein

(Appendix 6)
The smaller the focal length of the main lens included in the imaging parameter acquired by the imaging parameter acquisition unit, the larger the predetermined distance related to the image weight of the main table acquired by the main table acquisition unit.
The reconstructed image generating apparatus according to appendix 4 or 5, characterized in that:

(Appendix 7)
The shooting parameter acquired by the shooting parameter acquisition unit further includes an effective diameter of the main lens,
The larger the effective diameter of the main lens included in the imaging parameter acquired by the imaging parameter acquisition unit, the larger the predetermined distance related to the image weight of the main table acquired by the main table acquisition unit.
The reconstructed image generating apparatus according to any one of appendices 4 to 6, characterized in that:

(Appendix 8)
Indicates the strength of correspondence between a reconstructed pixel that is a pixel constituting a reconstructed image defined by the template definition unit and at least one subpixel included in a target subimage that is one of the subimages A table that defines pixel weights that are coefficients, and further includes a sub-table acquisition unit that acquires a sub-table corresponding to the imaging parameter acquired by the imaging parameter acquisition unit;
The reconstructed image generation unit includes a pixel value of the reconstructed pixel, a pixel value of the corresponding pixel extracted by the corresponding pixel extraction unit, and the reconstructed pixel defined by the main table acquired by the main table acquisition unit, Determine based on the image weight corresponding to the sub-image to which the corresponding pixel belongs, and the pixel weight corresponding to the reconstructed pixel and the corresponding pixel defined by the sub-table acquired by the sub-table acquiring unit,
The reconstructed image generating apparatus according to any one of appendices 1 to 7, characterized in that:

(Appendix 9)
Obtaining a light field image composed of a plurality of sub-images obtained by photographing a subject from different viewpoints;
Obtaining a shooting parameter for obtaining the light field image;
Defining a model of a reconstructed image that is an image of a subject reconstructed from the light field image;
The acquired imaging parameter is a table that defines image weights that are coefficients indicating the strength of correspondence between the reconstructed pixels that constitute the defined reconstructed image and at least one of the sub-images. Obtaining a main table to respond;
Extracting corresponding pixels, which are pixels corresponding to the reconstructed pixels constituting the defined reconstructed image, from the sub-image pixels constituting the plurality of sub-images;
The pixel value of the reconstructed pixel is determined based on the pixel value of the extracted corresponding pixel and the image weight corresponding to the reconstructed pixel defined by the acquired main table and the sub-image to which the corresponding pixel belongs. Obtaining the reconstructed image by obtaining;
A reconstructed image generation method comprising:

(Appendix 10)
On the computer,
Processing to obtain a light field image composed of a plurality of sub-images obtained by photographing the subject from different viewpoints;
A process for acquiring a shooting parameter for acquiring the light field image;
Processing for defining a model of a reconstructed image that is an image of a subject reconstructed from the light field image;
The acquired imaging parameter is a table that defines image weights that are coefficients indicating the strength of correspondence between the reconstructed pixels that constitute the defined reconstructed image and at least one of the sub-images. Process to get the main table to respond,
A process of extracting corresponding pixels which are pixels corresponding to the reconstructed pixels constituting the defined reconstructed image from the sub-image pixels constituting the plurality of sub-images;
The pixel value of the reconstructed pixel is determined based on the pixel value of the extracted corresponding pixel and the image weight corresponding to the reconstructed pixel defined by the acquired main table and the sub-image to which the corresponding pixel belongs. A process for generating the reconstructed image by obtaining,
A program characterized by having executed.

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 10 ... Imaging part, 11 ... Imaging part, 110 ... Optical apparatus, 111 ... Shutter, 112 ... Aperture, 113 ... Optical apparatus, 120 ... Image sensor, 20 ... Information processing part, 30 ... Image reconstruction apparatus 31 ... Image reconstruction device, 31a ... Information processing unit, 32 ... Main storage unit, 33 ... External storage unit, 34 ... Operation unit, 35 ... Display unit, 36 ... Input / output unit, 37 ... Internal bus, 38 ... Program , 210 ... Image processing unit, 220 ... Imaging control unit, 300 ... I / O unit, 3010 ... Operation unit, 3020 ... Shooting setting acquisition unit, 3030 ... Image acquisition unit, 3040 ... Output unit, 310 ... Storage unit, 311 ... Storage unit, 3110 ... Reconfiguration setting storage unit, 3120 ... MLB table storage unit, 3130 ... Reconstructed image storage unit, 3140 ... SLB table storage unit, 320 ... Information processing unit, 321 ... Information processing unit 3210 ... Prototype definition unit, 3220 ... Attention reconstruction pixel selection unit, 3230 ... Reaching point acquisition unit, 3240 ... MLB weight table acquisition unit, 3250 ... Sub lens selection unit, 3260 ... Corresponding pixel specifying unit, 3261 ... Corresponding pixel specifying unit 3270 ... Pixel value addition unit, 3271 ... Pixel value addition unit, 3280 ... Imaging surface arrival point acquisition unit, 3290 ... SLB weight table acquisition unit, 40 ... Storage unit, 410 ... Imaging setting storage unit, 420 ... Image storage unit, 50: Interface unit (I / F unit), 510 ... I / O unit, 520 ... Display unit, 530 ... Operation unit, LFI ... Light field image, OA ... Optical axis, OB ... Subject, POB ... Subject part, P ... site of interest, PS ... imaging element, ML ... main lens, PF ... imaging point, MIP ... main lens imaging plane, SIP ... sub-lens imaging plane, PE ... arrival point, P ... arrival coordinates, IE ... imaging surface, SLA ... sub-lens array, SL ... sub-lens, SL1, SL5 ... sub-lens, MLB ... main lens blur, MLBC ... main lens blur center, SLB ... sub-lens _{blur, S 11} ~ S _MN ... Sub-image, RI ... Reconstructed image, RF ... Reconstructed surface

Claims (11)

Using a plurality of sub-lenses corresponding to the plurality of viewpoints further capture the light main lens and the main lens captures to capture light from an object, a plurality of sub-images taken the subject from a different said viewpoint A light field image acquisition unit for acquiring a light field image composed of:
A prototype image definition unit that defines the original image of the reconstructed image is an image of the subject reconstructed from the light-field image,
Image weight definition that defines an image weight that is a coefficient indicating the strength of correspondence between a prototype pixel that is a pixel constituting the prototype image defined by the prototype image definition unit and the sub lens corresponding to the sub image And
A corresponding pixel extracting unit that extracts a corresponding pixel that is a pixel corresponding to the original pixel constituting the original image defined by the original image defining unit from the sub image pixels constituting the plurality of sub images;
And the pixel value of the corresponding pixel before SL corresponding pixel extracting portion is extracted, the said image weight said image weight definition portion is defined in accordance with the sub-lenses corresponding to said sub-image belonging paired pixels, on the basis Obtaining a pixel value of the reconstructed pixel of the reconstructed image and generating the reconstructed image;
I have a,
When an area in which the light beam from the subject passes through the main lens and is projected onto the plurality of sub lenses is a main lens blur area, the image weight is determined by the main lens blur area on each sub lens. Defined as a percentage,
A reconstructed image generating apparatus characterized by that. A shooting parameter acquisition unit for acquiring a shooting parameter for acquiring the light field image;
The image weight definition unit includes:
Table defining said original image is a pixel constituting the original image in which the original image definition section is defined, and the sub-lenses corresponding to said sub-image, the image weighting a coefficient indicating the corresponding strength of the in it, it has a main table acquisition unit that acquires a main table responsive to the imaging parameters which the imaging parameter acquisition unit has acquired,
The reconstructed image generation unit
The pixel values of the reconstructed pixel and the pixel value of the corresponding pixel in which the corresponding pixel extracting portion is extracted, corresponding to the sub-image belonging of said corresponding pixels, wherein the main table in which the main table acquisition unit has acquired is defined Generating the reconstructed image by determining based on the image weight according to the sub-lens ,
The reconstructed image generating apparatus according to claim 1. Before SL imaging parameters include at least a focal length of the main lens,
The reconstructed image generating apparatus according to claim 2. A reconstruction setting acquisition unit that acquires a reconstruction setting including information indicating a distance between the surface including the reconstructed image and the main lens;
The main table acquisition unit acquires the main table responsive to said reconfiguration and that the imaging parameter and the reconfiguration acquiring unit which the imaging parameter acquisition unit has acquired is acquired,
The reconstructed image generating apparatus according to claim 2 , wherein the reconstructed image generating apparatus is a reconstructed image generating apparatus. Said imaging said imaging parameter parameter acquiring unit has acquired, wherein the reconfiguration acquiring unit said reconfiguration acquired, the reconstruction of the relative position of the said main lens which is previously stored sub-lens, further comprising a sub-lens reaches point acquiring unit, wherein the light beam emitted from a portion of the subject corresponding to the pixel to obtain a sub-lens reaches point is a point to reach the sub-lens,
The main table in which the main table acquisition unit acquires, the image weight of said sub-images corresponding to the said sub-lenses from the sub-lens reaches point sub-lens reaches point obtaining unit has obtained a predetermined distance or more far position small,
The reconstructed image generating apparatus according to claim 4. The shorter the distance between the main lens and the plane including the reconstructed image included in the reconfiguration of reconfiguration acquisition unit has acquired is smaller, the image weight of the main table, wherein the main table acquisition unit acquires wherein the predetermined distance is increased according,
The reconstructed image generating apparatus according to claim 5. The imaging as the focal length of the main lens parameter acquisition unit is included in the captured parameters acquired is smaller, the predetermined distance is increased according to the image weight of the main table in which the main table acquisition unit acquires,
The reconstructed image generating apparatus according to claim 5 or 6. The imaging parameter the imaging parameter acquisition unit acquires further includes an effective diameter of the main lens,
The photographing as the effective diameter of the main lens parameter acquisition unit is included in the imaging parameters acquired is larger, the predetermined distance is increased according to the image weight of the main table in which the main table acquisition unit acquires,
The reconstructed image generation apparatus according to claim 5, wherein the reconstructed image generation apparatus is a reconstructed image generation apparatus. Indicates the strength of correspondence between the original pixel that is a pixel constituting the original image defined by the original image definition unit and at least one of the sub-pixels included in the target sub-image that is one of the sub-images. a table that defines the pixel weighting coefficients, further comprising a sub-table acquisition unit that acquires a sub-table responding to the shooting parameters which the imaging parameter acquisition unit has acquired,
The reconstructed image generation unit, the corresponding pixel, wherein the pixel values of the reconstructed pixel, the pixel values of the corresponding pixels corresponding pixel extracting portion is extracted, the main table in which the main table acquisition unit has acquired is defined It said image weight according to the sub-lenses corresponding to said sub-image to the genus, and said pixel weighting the sub-table sub table acquisition unit has acquired in accordance with the corresponding pixel defining, determined based on,
Reconstructed image generating apparatus according to any one of claims 2 to 8, characterized in that. Using a plurality of sub-lenses corresponding to the plurality of viewpoints further capture the light main lens and the main lens captures to capture light from an object, a plurality of sub-images taken the subject from a different said viewpoint Obtaining a light field image comprising:
A step of defining a prototype image of a reconstructed image which is an image of the subject reconstructed from the light-field image,
A step of defining the original pixel is a pixel constituting the original image defined, and the sub-lenses corresponding to said sub-image, the image weight is a factor that indicates the strength of the corresponding,
Sub image pixels constituting the plurality of sub-images, a step of extracting the corresponding pixel is a pixel corresponding to the original pixels constituting the original image defined,
And pixel values of the corresponding pixels out extraction, the said image weight as defined according to the sub-lenses corresponding to said sub-image belongs corresponding pixel, the pixel value of the reconstructed pixels of the reconstructed image based on Determining and generating the reconstructed image; and
Only including,
When an area in which the light beam from the subject passes through the main lens and is projected onto the plurality of sub lenses is a main lens blur area, the image weight is determined by the main lens blur area on each sub lens. Defined as a percentage,
A reconstructed image generation method characterized by the above. On the computer,
Using a plurality of sub-lenses corresponding to the plurality of viewpoints further capture the light main lens and the main lens captures to capture light from an object, a plurality of sub-images taken the subject from a different said viewpoint Processing to obtain a light field image consisting of
The procedure for defining original image of a reconstructed image which is an image of the subject reconstructed from the light-field image,
Process that defines the original pixel is a pixel constituting the original image defined, and the sub-lenses corresponding to said sub-image, the image weighting a coefficient indicating the corresponding strength of,
Wherein the plurality of sub-images from the sub-image pixels constituting the process of extracting the corresponding pixel is a pixel corresponding to the original pixels constituting the original image defined,
And pixel values of the corresponding pixels out extraction, the said image weight as defined according to the sub-lenses corresponding to said sub-image belongs corresponding pixel, the pixel value of the reconstructed pixels of the reconstructed image based on Determining and generating the reconstructed image;
Was executed,
When an area in which the light beam from the subject passes through the main lens and is projected onto the plurality of sub lenses is a main lens blur area, the image weight is determined by the main lens blur area on each sub lens. Defined as a percentage,
A program characterized by that.

Images