JP2014160912A - Image processor, imaging apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress a data amount when a plurality of light field images are stored.SOLUTION: A reconstitution section 150 of a digital camera 100 generates a plurality of reconstitution images obtained by reconstituting a light field image including a plurality of sub-images obtained by photographing a subject from different view points from the plurality of light field images different in exposure. An HDR synthesis section 160 determines a synthesis ratio of the plurality of generated reconstitution images in accordance with pixel values of respective corresponding pixels in the plurality of reconstitution images for each pixel. A specification section 170 specifies the pixel whose determined synthesis ratio is a prescribed ratio or more among the pixels constituting the plurality of reconstitution images. A reduction section 180 reduces a data amount of the sub-images including the pixel in which a relation with the specified pixel satisfies a prescribed condition. A storage section 190 stores the plurality of light field images whose data amount is reduced by the reduction section 180 in SD.

Description

  The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

In recent years, various digital cameras are known.
For example, a light field camera is known in which a user can change the focus and depth of field after shooting. As a technique related to this light field camera, Patent Document 1 discloses a technique for reconstructing a subject image at an arbitrary focal length from a light field image including a plurality of sub-images.

  On the other hand, there is known an HDR camera capable of obtaining a high dynamic range (HDR) image without overexposure and underexposure. As a technique related to this HDR camera, Patent Document 2 discloses a technique for obtaining an HDR image by synthesizing a plurality of high-speed continuous images with different exposures at an appropriate composition ratio.

Special table 2008-515110 gazette JP 2012-034340 A

  By the way, an HDR image can be obtained by synthesizing a plurality of reconstructed images respectively reconstructed from a plurality of light field images that are continuously shot at a high speed by changing the exposure with the light field camera described above. In order to enable adjustment of the focus and depth of field of the HDR image, it is necessary to store a plurality of light field images. For this reason, there is a problem that the amount of data to be stored becomes large.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to provide an apparatus, a method, and a program for reducing the amount of data when storing a plurality of light field images.

In order to achieve the above object, an image processing apparatus according to the present invention provides:
Generating means for generating a plurality of reconstructed images obtained by reconstructing a light field image including a plurality of sub-images obtained by photographing subjects from different viewpoints, from a plurality of the light field images having different exposures;
Determining means for determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying unit for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction means for reducing a data amount of a sub image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
Storage means for storing a plurality of light field images whose data amount has been reduced by the reduction means.

  According to the present invention, it is possible to reduce the amount of data when storing a plurality of light field images.

It is a block diagram which shows the hardware constitutions of the digital camera which concerns on embodiment of this invention. It is a schematic diagram which shows the structural example of the optical system of the digital camera shown in FIG. It is a conceptual diagram of a light field image. It is a photograph which shows an example of a light field image and a reconstruction image. It is a block diagram which shows the function structure of a digital camera. It is a flowchart which shows an example of the flow of the reconstruction process which concerns on embodiment. It is a figure which shows an example of a partial image definition table. It is a figure which shows an example of an arrangement | positioning space | interval table. It is a figure which shows the outline | summary of the process which produces | generates a reconstruction image from a light field image. It is a figure which shows an example of a pixel correspondence table. It is a figure for demonstrating the synthesis | combination of a reconstruction image, (a) shows the reconstruction image from which exposure differs, (b) shows the image for determination, (c) shows an HDR image, respectively. It is a figure which shows an example of a synthetic | combination ratio setting graph. It is a figure which shows an example of a synthetic | combination ratio correspondence table. It is a flowchart which shows an example of the flow of the data amount reduction process which concerns on embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
First, the hardware configuration of the digital camera 100 will be described with reference to FIG. As illustrated in FIG. 1, the digital camera 100 is an imaging device that includes a photographing unit 110, an image processing unit 120, an operation unit 130, and a display unit 140.
The imaging unit 110 includes a main lens 111, a microlens array 112, and an image sensor 113. Details of the photographing unit 110 will be described later.
Next, the image processing unit 120 includes a CPU (Central Processing Unit) 121, a ROM (Read Only Memory) 122, a RAM (Random Access Memory) 123, and an SD (Secure Digital Memory Card) 124.

Among these, the CPU 121 is a central processing unit that controls the image processing unit 120 and the digital camera 100.
The ROM 122 is a non-volatile memory that stores a program executed by the CPU 121.
The RAM 123 is a volatile memory that temporarily develops a program executed by the CPU 121 and is used as a work area when the CPU 121 performs various processes.

The SD 124 is a storage unit that stores various image data, various tables, and various setting information. The various image data includes, for example, light field image data and HDR image data, which will be described later. Examples of the various tables include a partial image definition table, an arrangement interval table, and a composition ratio setting graph described later. The various setting information includes, for example, the focal length f _ML of the main lens 111, a diaphragm (F value), position information between the microlens array 112 and the image sensor 113, and the like.

Next, the operation unit 130 includes a plurality of switches such as a shutter key that triggers a photographing process by the photographing unit 110 and a power switch for turning on the digital camera 100.
The display unit 140 is a display unit such as a liquid crystal display or an organic EL (Electroluminescence) display. The display unit 140 displays the various images based on the various image data stored in the SD 124.

Here, the details of the photographing unit 110 of the digital camera 100 will be described with reference to FIG. As shown in FIG. 2, in the optical system of the digital camera, a main lens 111, a microlens array 112, and an image sensor 113 are arranged in that order when viewed from the subject OB.
The main lens 111 is composed of one or a plurality of convex lenses, concave lenses, aspherical lenses, or the like. The main lens 111 condenses the light beam emitted from a point on the subject OB and forms an image on the imaging surface MA.

The microlens array 112 includes N × M microlenses 112-1 to 112 -N × M (N and M are arbitrary integer values of 2 or more) arranged in a grid pattern on a plane. FIG. 3 shows a vertical row (M pieces) of microlenses. Note that the microlenses 112-i (i is an integer value in the range of 1 to N × M) have the same diameter, and are arranged in a lattice pattern in the microlens array 112 at the same interval.
The micro lens 112-i condenses the light beam incident from the subject OB through the main lens 111 for each incident direction and forms an image on the image sensor 113 as a sub image.

The image sensor 113 is an image sensor that generates a light field image (Light Field Image: LFI) including a plurality of sub-images formed by the micro lenses 112-i.
This light field image includes a plurality of sub-images when the subject OB is viewed from a plurality of viewpoints at the same time. A conceptual diagram of this light field image is shown in FIG. FIG. 3 is an example of a light field image obtained by photographing a block-like subject OB.
This light field image is composed of sub-images (S _{11 to} S _MN ) corresponding to each of N × M microlenses 112-i arranged in a grid pattern. For example, the upper left sub-image S ₁₁ corresponds to an image obtained by photographing the object OB from the upper left, the sub-image S _MN the lower right corresponds to an image obtained by photographing the object OB from the lower right.

An example of a photograph of this light field image is shown in FIG. Since the light field image includes a plurality of sub-images when the subject OB is viewed from a plurality of viewpoints at the same time, the image processing unit 120 of the digital camera 100 appropriately synthesizes the sub-images so that a predetermined distance from the main lens 111 is obtained. It is possible to reconstruct an image that is focused on a plane that is farther forward (FIG. 4B). A reconstruction process, which is a process for generating a reconstructed image (RI) as shown in FIG. 4B from the light field image, will be described later.
In this embodiment, each sub image is a grayscale image, and each pixel constituting the sub image has a pixel value.

  Here, the CPU 121 reads out various programs in the ROM 122, develops them in the RAM 123, and then controls the image processing unit 120 according to the various programs, thereby exhibiting the functions of the respective units as shown in FIG. As functions, as shown in FIG. 5, a reconfiguration unit 150, HDR synthesis unit 160, identification unit 170, reduction unit 180, and storage unit 190 are provided. It should be noted that although the execution subject of the functions of each unit is the CPU 121, for convenience of explanation, the function of each unit will be described as the execution subject.

First, the reconstruction unit 150 is a generation unit that generates a reconstructed image from a light field image including a plurality of sub-images obtained by capturing the subject OB from different viewpoints.
Specifically, the reconstruction unit 150 arranges the partial image on the reconstruction plane and the size when the partial image is cut out from each sub image based on the image shift coefficient of each sub image included in the light field image. The arrangement interval at the time is specified. Here, to cut out the partial image from the sub image means to extract the partial image from the sub image while determining the boundary of the partial region and leaving all the image data of the sub image.

  Next, the reconstructed image is generated by sequentially arranging the partial images of the specified size on the reconstruction surface according to the arrangement interval. The reconstruction plane is a virtual plane at a predetermined reconstruction distance within a reconstruction distance range in which reconstruction is possible. This reconstruction distance range is determined by the performance of the optical system such as the main lens 111. Specifically, this reconstruction distance range is the closest distance from the main lens 111 where the pixel shift (parallax) of the subject between sub-images is the largest, the farthest distance from the main lens 111 where the parallax is the smallest, The range is between. In this embodiment, as an example, it is assumed that a reconstruction distance range in which reconstruction is possible is within a range of 20 cm to 10 m before the main lens 111, and a predetermined reconstruction distance is 2 m. Hereinafter, the reconstruction process will be described in detail with reference to the flowchart of FIG.

When the reconstruction unit 150 acquires the light field image generated by the image sensor 113 and temporarily stores it in the buffer memory set in a predetermined area of the RAM 123, the reconstruction unit 150 starts the reconstruction process of FIG.
First, the reconstruction unit 150 pays attention to the k1th sub-image of the light field image using k1 as a counter variable (step S11). Next, the reconstruction unit 150 calculates an image shift coefficient of the target sub-image that is the sub-image focused in step S11 (step S12). Specifically, the reconstruction unit 150 is an image shift coefficient that is a coefficient indicating how much the subject that appears in the target sub-image appears in a sub-image around a predetermined range centered on the target sub-image. Is calculated. The image shift coefficient is calculated by the following method, for example.

  A predetermined area at the center of the target sub-image (for example, the center 10 × 10 pixel area) is the central area, and the part corresponding to the central area is located on the right side (or the left side if there is none) of the sub-image. Ask for. Specifically, the area of the 10 × 10 pixel at the center on the right is set as the calculation target area. Then, the sum of absolute differences between the pixel values of the center area and the calculation target area is calculated. Next, the sum of absolute differences is similarly obtained for the calculation target region shifted to the right by one pixel. This is repeated for a range of possible pixel shifts (parallax). The area to be calculated having the smallest sum of absolute differences obtained is the area corresponding to the central area. In addition, the number of pixel shifts for which the minimum sum of absolute differences is obtained is defined as an image shift coefficient.

  Next, the reconstruction unit 150 identifies the size of the partial image (step S13). Specifically, the reconstruction unit 150 refers to the partial image definition table illustrated in FIG. 7 stored in the SD 124 and specifies the size of the partial image corresponding to the calculated image shift coefficient. For example, when the image shift coefficient is 8 pixels, the size of the partial image is a region of 6 × 6 pixels in the center of the target sub-image. Note that this partial image definition table is stored in the SD 124 in advance. The size of the partial image in the partial image definition table increases as the image shift coefficient increases. This means that if the image misalignment coefficient is large, the position of the subject OB is greatly shifted between the sub-images, and information is not lost when the reconstructed images are generated by connecting the partial images of the adjacent sub-images. This is because a larger partial image is preferable. On the other hand, for an image with a small image deviation coefficient, the partial image is made small so that the duplication of information appearing between adjacent sub-images does not appear excessively on the reconstructed image.

  Next, the reconfiguration unit 150 identifies the arrangement interval (step S14). Specifically, the reconstruction unit 150 refers to the arrangement interval of the partial images with reference to the arrangement interval table illustrated in FIG. 8 in which the reconstruction distance, the image deviation coefficient, the focal length of the main lens 111, and the arrangement interval are associated with each other. Is identified. In this arrangement interval table, suitable arrangement intervals obtained in advance through experiments are set. This arrangement interval table is stored in advance in the SD 124. For example, when the reconstruction distance is 2 m, the calculated image shift coefficient is 3 pixels, and the focal length of the main lens 111 as setting information is 5 m, the reconstruction unit 150 identifies an arrangement interval of 2 pixels.

  Next, the reconstruction unit 150 superimposes and arranges the partial images on the reconstruction surface (step S15). Specifically, the reconstruction unit 150 cuts out a partial image having a specified size from the target sub-image, and places the cut-out partial image on the configuration surface according to the arrangement interval. At this time, since the size of one side of the partial images is larger than the arrangement interval, the partial images arranged on the reconstruction plane overlap each other. For the overlapped portion, the pixel values of the pixels are added and averaged to obtain the pixel value of the reconstructed image.

  Next, the reconstruction unit 150 determines whether or not all sub-images have been processed (step S16). If not processed (step S16; No), the counter is incremented (step S17) to process all sub-images. Steps S11 to S15 are repeated until When the reconstruction unit 150 determines that all the sub-images have been processed (step S16; Yes), the generated reconstruction image is stored in the SD 124, and the process ends. Further, the reconstruction unit 150 deletes the light field image temporarily stored in the RAM 123.

Here, an outline of processing for generating a reconstructed image from a light field image will be described with reference to FIG. For convenience of explanation, a case where one side of the size of the specified partial image is twice the arrangement interval will be described as an example. In this example, the partial images are overlapped four by four except for the end portion of the reconstruction surface.
First, the reconstruction unit 150 will focus the upper left sub-image _{S 11} of the light field image as a sub image (step S11). The reconstruction unit 150, after the process of step S12 through 14, cut out partial image _{PI 11} from the target sub-images _{S 11,} arranged at the upper left of the reconstruction plane partial image _{PI 11} thereof cut out (step S15) .

Returning to step S11, reconstruction unit 150, then focuses on the sub-image _{S 12} to the right (step S11). Reconstruction unit 150, target sub-image is cut out _{S 12} partial image _{PI 12} from placing the clipped partial image _{PI 12} on the right of the partial image _{PI 11} disposed in the earlier reconstruction plane (step S15). Since one side of the cut out partial image PI ₁₂ is twice the arrangement interval, it is arranged with overlap in the horizontal direction. Similarly, the reconstruction unit 150 cuts out the partial image PI ₁₃ from the sub-image S _{13 on} the right side of the sub-image S ₁₂ and arranges the partial image PI ₁₃ with overlapping in the horizontal direction (see FIG. 9A).

The reconstruction unit 150 repeats this process up to the right end of the light field image. The reconstruction unit 150 returns to the left end of the light field image after the processing of the rightmost sub-image _S1N is completed. The reconstruction unit 150 may focus on the sub-image S ₂₁ next to the uppermost (step S11), and noted the partial image and the partial image PI ₂₁ cut out from the sub-image S ₂₁ disposed in the first reconstruction plane placed below the P ₁₁ (step S15). Since one side of the cut out partial image PI ₂₁ is twice the arrangement interval, the reconstruction unit 150 arranges the overlapping partial image with overlap in the vertical direction (see FIG. 9B).

The reconstructing unit 150 processes the sub-image in the next stage of the light field image after all the sub-images S _{21 to} S _2N in the second stage have been processed. Reconstruction unit 150, when processing to sub-image _{S MN} bottom right of the light field image (Step S16; Yes), the reconstructed image generated ends the reconstruction process and stored in the SD124.

Note that, after the reconstruction processing described with reference to FIGS. 6 to 9, the reconstruction unit 150 uses coordinates indicating each pixel constituting the generated reconstruction image and a sub image for data reduction processing described later. A pixel correspondence table shown in FIG. 10 in which coordinates indicating the respective pixels constituting the image are associated with each other is created and stored in the SD 124.
In this pixel correspondence table, the coordinates of each pixel of the reconstructed image are represented by the coordinate origin (0, 0) at the upper left of the reconstructed plane, the x direction in the horizontal direction, and the y coordinate in the vertical direction. The coordinates of each pixel of the sub image, of the upper left sub-image S ₁₁ constituting the light field image coordinates of each pixel of the sub-image S _MN lower right corresponds to the each pixel of the reconstructed image coordinates The coordinates are shown. For example, the coordinates (x ₁ , y ₁ ) of the upper left sub-image S ₁₁ constituting the light field image correspond to the coordinates (0, 0) of the pixel of the reconstructed image. On the other hand, the coordinates (1, 1) of the pixels of the reconstructed image correspond to S ₁₁ (x ₃ , y ₃ ), S ₁₂ (x ₂ , y ₂ ), S ₂₁ (x ₁ , y ₁ ) and S ₂₂ (x ₁ , y ₁ ). This indicates that the pixels of the reconstructed image at the coordinates (1, 1) are overlapped with the pixels of the partial image at the coordinates of S ₁₁ , S ₁₂ , S ₂₁ and S ₂₂ .

Returning to FIG. 5, the HDR synthesizing unit 160 is a synthesizing unit that synthesizes a plurality of reconstructed images generated from a plurality of light field images with different exposures by the reconstructing unit 150 using a predetermined composition ratio for each pixel. is there.
Specifically, the HDR synthesizing unit 160 acquires, from the SD 124, three reconstructed images reconstructed by the reconstructing unit 150 from three light field images that have been subjected to three high-speed continuous shooting with different exposures. Next, the HDR synthesizing unit 160 synthesizes the acquired three reconstructed images using a predetermined synthesis ratio for each pixel, generates an HDR image, and stores the HDR image in the SD 124. Hereinafter, the HDR synthesizing process for generating the HDR image will be described in detail with reference to FIG.

  First, the HDR synthesizing unit 160 acquires three reconstructed images (see FIG. 11A) reconstructed by the reconstructing unit 150 from the SD 124. The three reconstructed images shown in FIG. 11A are a reconstructed image reconstructed from an underexposed light field image in order from the left (hereinafter referred to as “underexposed reconstructed image”), a light field with proper exposure. A reconstructed image reconstructed from an image (hereinafter referred to as “appropriate exposure reconstructed image”), and a reconstructed image reconstructed from an overexposed light field image (hereinafter referred to as “overexposed reconstructed image”). . Underexposure means exposure below proper exposure, and overexposure means exposure beyond proper exposure. As shown in the drawing, the underexposed reconstructed image is dark over the entire image because the exposure is less than the appropriate exposure, and the overexposed reconstructed image is bright over the entire image because the overexposed reconstructed image is exposed beyond the proper exposure.

  Next, the HDR synthesizing unit 160 averages the pixel values of each pixel constituting the underexposure reconstructed image, the proper exposure reconstructed image, and the overexposed reconstructed image, and smoothes the averaged image for determination. An image is created (FIG. 11B). As a result, the pixel values (0: black to 255: white) of the three reconstructed images are averaged, so that the brightness is averaged and a determination image from which random noise is removed by smoothing is obtained. It is done. The smoothing can be performed by a known method (for example, by applying an ε filter to the averaged image).

Next, the HDR synthesizing unit 160 refers to the synthesis ratio setting graph shown in FIG. 12 stored in the SD 124, and based on the pixel value of the image for determination, the under-exposure synthesis ratio Ru and the appropriate exposure composition. The ratio Rm and the overexposure composition ratio Ro are respectively determined. Note that the pixel value of each pixel of the determination image is obtained by averaging and smoothing the pixel value of each pixel of the three reconstructed images as described above. For this reason, it can be said that the combination ratio of each of Ru, Rm, and Ro determined based on the pixel value of each pixel of the determination image is determined according to the pixel value of each pixel of the three reconstructed images. That is, it can be said that the HDR synthesizing unit 160 is a determining unit that determines a composition ratio of a plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images.
In the combination ratio setting graph shown in FIG. 12, when the pixel value of the determination image is low (when dark), Ro is set high so that the combination ratio of the pixels of the bright overexposed reconstructed image is high. On the other hand, in the combination ratio setting graph, when the pixel value of the determination image is high (when it is bright), Ru is set high so that the combination ratio of the pixels of the dark underexposed reconstructed image is high.

For example, when the pixel value of the determination image is high (when the upper right of the determination image in FIG. 11B is the target pixel), the combination ratios of Ru, Rm, and Ro are each based on the combination ratio setting graph. They are generally 70%, 30% and 0%.
Here, the pixel value of the target pixel of the proper-exposure reconstructed image is “RIm”, the pixel value of the target pixel of the under-exposure reconstructed image is “RIu”, and the pixel value of the target pixel of the over-exposed reconstructed image is “ RIo ". In this case, assuming that the pixel value of the pixel of the HDR image is “HDRmix”, HDRmix is expressed by the following equation (1).
HDRmix = Ru × RIu + Rm × RIm + Ro × RIo (1)

  The HDR synthesizing unit 160 sets a pixel having the same coordinate in each of the under-exposure reconstructed image, the proper exposure reconstructed image, and the over-exposure reconstructed image as a target pixel, and sets the pixel of the determination image at the same coordinate as the target pixel. Ru, Rm, and Ro are determined based on the pixel values. Then, the HDR synthesizing unit 160 calculates HDRmix that is a pixel value of a pixel of the HDR image based on the formula (1). The HDR synthesizing unit 160 calculates HDRmix for all pixels of the underexposure reconstructed image, the proper exposure reconstructed image, and the overexposed reconstructed image, generates the HDR image shown in FIG. 11C, and stores it in the SD 124. .

In addition, the coordinate position which shows each pixel of an HDR image, an underexposure reconstruction image, a proper exposure reconstruction image, and an overexposure reconstruction image is the same. The HDR synthesizing unit 160 creates a synthesis ratio correspondence table shown in FIG. 13 in which coordinates indicating each pixel of the HDR image are associated with the synthesis ratios Ru, Rm, and Ro for data amount reduction processing to be described later, and stores them in the SD 124. Remember.
In the combination ratio correspondence table, the coordinates of each pixel of the HDR image are represented by the coordinate origin (0, 0) at the upper left of the HDR image, the x direction in the horizontal direction, and the y coordinate in the vertical direction. In addition, the composition ratio indicates how much composition ratio (Ru,) each pixel of each of the underexposure reconstructed image, the proper exposure reconstructed image, and the overexposed reconstructed image at the same coordinate position as the coordinates indicating each pixel of the HDR image. Rm and Ro). A to I shown in the figure indicate respective composition ratios determined based on the composition ratio setting graph of FIG.

Returning to FIG. 5, the specifying unit 170 is a specifying unit that specifies a pixel having a combination ratio determined by the HDR combining unit 160 larger than 0 among the pixels constituting the plurality of reconstructed images.
Specifically, the specifying unit 170 refers to the combination ratio correspondence table in FIG. 13 and selects pixels with a combination ratio larger than 0 for each of the underexposure reconstructed image, the proper exposure reconstructed image, and the overexposure reconstructed image. Specify more than one. This is because the HDR image and each reconstructed image have the same coordinate position indicating each pixel. Therefore, referring to the composition ratio correspondence table in FIG. 13, the composition ratio of each reconstructed image is determined from the coordinates of the HDR image pixel. This is because a pixel larger than 0 can be specified. A specific specifying method of the specifying unit 170 will be described later.

Next, the reduction unit 180 is a reduction unit that reduces the data amount of a sub-image that includes a plurality of pixels specified by the specifying unit 170 (a plurality of pixels having a combination ratio greater than 0) and pixels that do not correspond to any of them. .
Specifically, the reduction unit 180 changes the pixel value of the pixel of the sub-image to a specific value for the sub-image including a plurality of pixels specified by the specifying unit 170 and none of the pixels. The data amount is reduced by compressing the light field image including the subsequent sub-image. A specific method for reducing the amount of data by the reduction unit 180 will be described later.

Next, the storage unit 190 is means for storing a plurality of light field images whose data amount has been reduced by the reduction unit 180 in the SD 124.
As described above, one characteristic point of the digital camera 100 described with reference to FIGS. 1 to 11 is that pixels corresponding to pixels of a reconstructed image having a combination ratio of 0 that is not used for combining in generating an HDR image. The light field image is stored in the SD 124 by reducing the data amount of the included sub image. Therefore, data amount reduction processing related to this point will be described below with reference to the flowchart of FIG.

The flowchart of FIG. 14 is implemented by the CPU 121 executing various programs stored in the ROM 122 to exhibit the functions of the units shown in FIG. Further, at the start of the process of the flowchart of FIG. 14, the CPU 121 executes an initial setting process in which a flag 0 indicating that the light field image is not used for composition is set in advance in the coordinates of the sub-image pixels of each light field image.
When the CPU 121 obtains three light field images (LFI) obtained by performing three high-speed continuous shootings with different exposures from the image sensor 113 and temporarily stores them in a buffer memory set in a predetermined area of the RAM 123, the data amount Start reduction processing.

  First, the CPU 121 reconstructs three light field images with different exposures to obtain three reconstructed images (step S21). Specifically, the CPU 121 obtains three reconstructed images by performing the reconstructing process shown in FIG. 6 on each light field image at a predetermined reconstruction distance (2 m in this embodiment). Further, the CPU 121 creates the pixel correspondence table in FIG. 10 and stores it in the SD 124.

  Next, the CPU 121 performs HDR composition on the three reconstructed images to obtain an HDR image (step S22). Specifically, the CPU 121 performs the HDR combining process described with reference to FIG. 11 and combines the three reconstructed images using a predetermined combining ratio for each pixel to obtain an HDR image. Further, the CPU 121 creates a composition ratio correspondence table in FIG. 13 and stores it in the SD 124.

  Next, the CPU 121 pays attention to the k2th pixel of the HDR image using k2 as a counter variable (step S23). Next, the CPU 121 specifies the coordinates of the pixel of the sub-image corresponding to the coordinates of the pixel of interest for the k3th light field image using k3 as a counter variable (step S24). Specifically, when the coordinate of the target pixel of the HDR image is HDR (X, Y), the CPU 121 uses that the coordinate HDR (X, Y) and the coordinates of the three reconstructed images are the same. Then, the coordinates of the pixels of the sub-image are specified.

First, the CPU 121 selects one of the three reconstructed images. Then, the CPU 121 displays the coordinates of the sub-image pixels corresponding to HDR (X, Y) in the light field image (k3 = 1) that is the generation source of the selected reconstructed image, and the pixel correspondence table in FIG. Identify by reference. For example, when the coordinate HDR (X, Y) of the pixel of the selected overexposed reconstructed image is (1, 1), the partial images cut out from the four sub-images S ₁₁ , S ₁₂ , S ₂₁ and S ₂₂ are overlapped. The coordinates of the four sub-images are S ₁₁ (x ₃ , y ₃ ), S ₁₂ (x ₂ , y ₂ ), S ₂₁ (x ₁ , y ₁ ), and S ₂₂ (x ₁ , y _{1), respectively.} ).
Here, for convenience of explanation, the coordinates of the four sub-images corresponding to the coordinates HDR (X, Y) are respectively Sb1 (x, y), Sb2 (x + α, y), Sb3 (x, y + α), and Sb4 (x + α, This will be described as y + α). The sub-images Sb1 to Sb4 are sub-images including pixels corresponding to the pixel at the coordinate HDR (X, Y) position of the reconstructed image. Further, how much each coordinate of the sub-images Sb2 to Sb4 is deviated from the reference coordinate (x, y) with respect to the coordinate (x, y) of the sub-image Sb1 corresponding to the coordinate HDR (X, Y). Is expressed using a coefficient α. Here, α is a coefficient that varies depending on the reconstruction distance. In this embodiment, since the reconstruction distance is 2 m, the amount of pixel shift (parallax) occurring at the reconstruction distance 2 m when Sb1 is used as a reference is α.

  Next, the CPU 121 determines whether or not the composition ratio of the reconstructed image generated from the k3rd light field image among the composition ratios of the target pixel is 0 (step S25). Specifically, the CPU 121 identifies the composition ratios Ru, Rm, and Ro corresponding to the coordinates HDR (X, Y) of the target pixel of the HDR image with reference to the composition ratio correspondence table of FIG. Next, the CPU 121 determines whether or not the composition ratio of the reconstructed image generated from the first (k3 = 1) light field image among the composition ratios Ru, Rm, and Ro of the target pixel is zero. For example, when the first light field image is an overexposed light field image, it is determined whether the composition ratio Ro of the target pixel is 0 with reference to the composition ratio Ro of the overexposed reconstructed image.

  Next, when the composition ratio of the reconstructed image generated from the k3th light field image is not 0 (used for composition) among the composition ratios of the target pixel, the CPU 121 specifies in step S24. A flag 1 is set to the coordinates of the pixel of the sub-image of the k3rd light field image (step S26). That is, the coordinates of the pixel of the sub-image having flag 0 indicating that it is not used for composition are overwritten on flag 1. Here, for example, when the composition ratio Ro of the over-exposure reconstructed image is not 0, the coordinates Sb1 (x, y), Sb2 (x + α, y), Sb3 ( x, y + α) and Sb4 (x + α, y + α) are each set to 1.

  On the other hand, when the composition ratio of the reconstructed image generated from the k3th light field image is 0 (not used for composition) among the composition ratio of the target pixel, the CPU 121 proceeds to step S27. move on.

  Next, the CPU 121 determines whether or not the coordinates of the sub-images of all light field images have been specified (step S27). Here, when the coordinates of the sub-images of all the light field images have not been specified (step S27; No), the CPU 121 increments the counter k3 to perform the processing of steps S24 to S26 for the remaining light field images. (Step S28), the process returns to Step S24. In the above example, the first light field image is an overexposed light field image, so the second light field image is selected from a properly exposed or underexposed light field image.

  When the processing of steps S24 to S26 is completed for all the light field images, the CPU 121 determines that the coordinates of the sub-images of all the light field images have been identified (step S27; Yes), and whether all the pixels of the HDR image have been noticed. (Step S29). If all the pixels of the HDR image have not been noticed (step S29; No), the CPU 121 increments the counter k2 (step S30), returns to step S23, and steps S23 to S27 until all the pixels of the HDR image have been noticed. Repeat the process.

  Next, when the CPU 121 determines that all the pixels of the HDR image have been noticed (step S29; Yes), the pixel used for composition among the coordinates of the pixels of the sub image respectively corresponding to the coordinates of all the pixels of the HDR image. As for all of the coordinates, the flag 1 is left (assuming that the pixels of the sub-image where the flag remains 0 are not used for HDR synthesis), and the process proceeds to step S31. In step S <b> 31, the CPU 121 specifies a sub image in which all of the pixel coordinate flags in the sub image are 0 (flag 1 has not been set) from among the sub images included in each light field image. Next, the CPU 121 sets all the pixel values in the specified sub-image to 0 (step S32).

  In the processes of steps S31 and S32, the CPU 121 identifies all sub-images that are not used for HDR synthesis from among the plurality of sub-images included in each light field image, and all the pixel values in the identified sub-images. The specific pixel value is changed to 0.

  Next, the CPU 121 compresses and saves each light field image. Specifically, the CPU 121 compresses a plurality of light field images including sub-images whose pixel values are all changed to 0 and stores them in the SD 124 (step S33). For example, when storing in the bitmap format, run length compression such as LZH is executed. Even in the case of storing in the JPEG encoding format, since the run length encoding (two-dimensional Huffman encoding) is used for the coefficient compression after the DCT coefficient is quantized, the pixel value of the sub-image is set to 0. If so, the amount of data after encoding can be reduced. As a result, the SD 124 stores a plurality of light field images with a reduced data amount. The CPU 121 deletes each light field image temporarily stored in the RAM 123 after compressing and saving each light field image.

  Further, among the processes performed by the CPU 121 in FIG. 14, step S21 is a function of the reconstruction unit 150, step S22 is a function of the HDR synthesis unit 160, steps S23 to S30 are functions of the specifying unit 170, and step S31. And S32 correspond to the function of the reduction unit 180, and step S33 corresponds to the function of the storage unit 190, respectively.

  As described above, in the processing of FIG. 14, the digital camera 100 includes the functions of the reconstruction unit 150, the HDR composition unit 160, the identification unit 170, the reduction unit 180, and the storage unit 190, thereby reconstructing image pixels used for HDR composition. When a plurality of pixels are specified and pixels that do not correspond to any of the specified plurality of pixels (pixels that are not used for composition) are all the pixels constituting the sub image, the pixel values of all the sub images are specified. The pixel value is changed to 0. Thereafter, a plurality of light field images including the sub-image are compressed and stored. As a result, the compression efficiency can be increased, so that the amount of data can be suppressed even when a plurality of light field images are stored in the SD 124 for HDR images.

  Note that the compression efficiency is increased because, for example, when reversibly compressing a bitmap light field image into a ZIP or LHA light field image, run-length compression is generally used, so that the pixel values are all zero. This is because the data amount of the sub image can be greatly reduced by compression. For example, when a bitmap light field image is irreversibly compressed into a JPEG light field image, the high-frequency component of the light field image is reduced by setting all pixel values to 0 for at least some sub-images. Therefore, the compression rate can be increased while ensuring the image quality. Furthermore, the quantization rate may be set low in a block including a sub-image having at least a pixel value of 0. Thereby, since the data amount can be greatly reduced by compression for the sub-images whose pixel values are all 0, the compression efficiency is increased.

  Further, the user can obtain an HDR image in which the reconstruction distance is changed based on each light field image stored in the SD 124 after the processing of FIG. In this case, the CPU 121 generates a reconstructed image with a reconstruction distance set by the user (for example, in the case of a reconstruction distance of 1.5 m, a reconstructed image obtained by focusing on a distance of 1.5 m from the main lens 111). HDR synthesis is performed), and the same composition ratio as in the case of the reconstruction distance of 2 m is used as the composition ratio of each pixel in the HDR composition. This is because the composition ratio changes according to the brightness of the judgment image obtained by averaging and smoothing the reconstructed images, but this brightness can be changed even if the reconstruction distance is changed (for example, from 2 m to 1.5 m). Because it never changes drastically. For this reason, even if the user adjusts the focus later, it is sufficient to perform the process of FIG. 14 once at a predetermined reconstruction distance, and the process of FIG. 14 is performed over all reconstruction distances in advance. Since it is not necessary to repeat the calculation, the amount of calculation can be greatly reduced. However, the process of FIG. 14 may be repeatedly performed over all reconstruction distances in advance. For example, when the reconstruction distance changes after performing the process of FIG. 14 a predetermined number of times within a reconstruction distance range. You may raise the precision of the composition ratio to be used.

Although the description of the embodiment has been completed, the specific configuration of the digital camera 100, the contents of the processing illustrated in FIG. 14, and the like are of course not limited to those described in the above embodiment.
For example, in the above-described embodiment, the CPU 121 in FIG. 1 controls the image processing unit 120 and the digital camera 100, but the digital camera 100 may be provided with a separate CPU. In this case, the CPU 121 of the image processing unit 120 specializes in the control of the image processing unit 120 and does not perform display control on the display unit 140, operation detection of the operation unit 130, or the like.

  Further, instead of the SD 124 of the image processing unit 120, other removable recording media (for example, CD, DVD, etc.) may be used. In this case, the above-described various image data are recorded on the removable recording medium.

  Further, in the reconstruction processing of FIG. 6 performed by the reconstruction unit 150 illustrated in FIG. 5, the distance from the main lens 111 to the subject OB is estimated based on the pixel shift of the subject OB between the sub-images, and the estimation is performed. A reconstructed image in which the subject OB is blurred may be generated according to the difference between the distance and the reconstruction distance. Note that the distance to the subject OB is estimated using a table in which the pixel shift is associated with the distance from the main lens 111. For example, the distance of the subject indicated by the pixel shift value is estimated for each pixel using this table. Then, as the difference between the estimated subject distance and the reconstruction distance is larger, blurring may be performed using a stronger Gaussian filter. At this time, the filter to be applied to the pixel value is also applied to the synthesis ratio, and the processing of steps S25 to S26 is performed using the synthesis ratio after the filter, and is referred to when generating an HDR image when the filter is applied. Flag 1 can be set at the coordinates of the pixel to be processed.

  Further, the reconstruction process performed in this embodiment is not limited to the reconstruction process shown in FIG. That is, if the reconstruction processing can associate the coordinates of each pixel of the reconstructed image with the coordinates of each pixel of the sub-image of the light field image, a method of generating a reconstructed image by superimposing the partial images of FIG. Not necessarily. For example, as another technique, a reconstructed image may be generated using the ray tracing technique disclosed in Patent Document 1.

Further, the HDR synthesizing process performed by the HDR synthesizing unit 160 illustrated in FIG. 5 may be another method as long as it can associate the coordinates of each pixel of the reconstructed image with the synthesis ratio. For example, the image may be synthesized by tone mapping or exposure enhancement by detail emphasis or tone compression. Furthermore, in this embodiment, the case where the reconstructed images respectively generated from a plurality of light field images with different exposures are HDR-combined has been described as an example, but the combining is not limited to HDR combining. For example, reconstructed images generated from a plurality of light field images having different transparency levels may be combined. In this case, in the reconstruction process in step S22 of FIG. 14, a composite image corresponding to the purpose may be generated from a plurality of reconstructed images, and the processes in and after step S22 may be executed using the composite image.
In the processing of FIG. 14, in steps S25 and S26, if the composition ratio of the target pixel of the HDR image is larger than 0 (used for composition), the flag 1 is set to the pixel coordinate of the sub-image corresponding to the coordinate of the target pixel. However, it is not limited to this. For example, a flag may be set in the same manner as described above for pixels of a reconstructed image whose composition ratio is equal to or higher than a predetermined ratio (for example, 5% or higher).

  Further, in the processing of FIG. 14, in step S26, a flag is set on the coordinates of the pixel of the sub-image corresponding to the target pixel, but the present invention is not limited to this. For example, when the coordinates of the pixel of the specified sub-image are Sb1 (x, y), Sb2 (x + α, y), Sb3 (x, y + α), and Sb4 (x + α, y + α), α is a change in reconstruction distance. It is also possible to change the range of pixel shift (parallax) that can occur according to the above, and set flags for all the coordinates determined by the changed α. As a result, even when the user changes the reconstruction distance later, it is possible to prevent a situation in which pixels necessary for the synthesis process are lost.

  In the data amount reduction process shown in FIG. 14, in steps S24 to S27, the coordinates of the target pixel of the HDR image and the coordinates of the sub-image corresponding to each light field image are separately specified and flagged. However, another method may be used. For example, since the coordinates of the pixel of interest and the coordinates of the corresponding sub-image are the same in each light field image, after specifying the coordinates of the sub-image of one light field image, each light field image You may perform the process which sets a flag to the coordinate of a subimage simultaneously. As a result, the processing of steps S27 and S28 can be omitted, and the amount of calculation for the data amount reduction processing can be suppressed.

Further, in the processing of FIG. 14, in steps S31 and S32, sub-images in which the pixel coordinate flags in the sub-images are all 0 out of the sub-images included in each light field image (flag 1 was not set). Sub-image) is specified, and all the pixel values in the specified sub-image are set to zero. In other words, if even one flag is set at the coordinates of the pixels in the sub-image, the value of the pixel value of each pixel of the sub-image is not changed. This is because, when only one pixel is used for composition at a predetermined reconstruction distance in the sub-image, if all pixel values are changed to a specific value, it is used for composition at a different reconstruction distance. This is because the pixel value of this pixel is changed. That is, even if only one pixel is used for composition in the sub-pixel, the pixel values of other pixels that may be used when the reconstruction distance changes by not changing all the pixel values in the sub-image. Can be secured.
However, the pixel value may be changed even if the above-described pixel coordinate flags in the sub-image are not all zero. For example, if the coordinates flags of pixels included in a certain area in the sub-image are all 0, the pixel values in that area may all be 0. This is because high compression can be expected for that region. Further, the pixel value 0 may be set to another specific pixel value. This is because if the same pixel value continues, the compression ratio of the sub-image increases.

In this embodiment, as a technique for reducing the data amount of the light field image, the pixel value in the sub-image is changed to a specific pixel value so as to be the same continuous pixel value, and the compression is performed after increasing the compression rate. The amount of data was reduced by doing this, but another method may be used. For example, the data amount of the light field image may be reduced by deleting the sub image data for the sub image in which all the pixel coordinate flags in the sub image are 0 or less than a predetermined composition ratio.
Alternatively, if the coordinates of pixels included in a region having an area equal to or larger than a predetermined threshold in the sub-image are all 0 or less than a predetermined composition ratio, the region is deleted to reduce the data amount of the light field image May be. In addition, since a region that does not constitute a sub-image in the light field image is not used for composition, this region may be deleted. These means that, in the light field image, when all the pixels constituting an area having an area equal to or larger than a predetermined threshold value are not used for synthesis, the area is deleted to reduce the data amount.

  In this embodiment, the description has been made on the assumption that the various images are grayscale images, but the various images may be color images. In this case, for example, an arbitrary color space such as RGB or YUV can be used.

In addition, each process (reconstruction process, HDR synthesis process, and data amount reduction process) of the image processing unit 120 of the present invention can also be performed by a computer such as a normal PC.
Specifically, in the above-described embodiment, it has been described that a program for exhibiting a function related to each process is stored in the ROM 122 in advance. However, a program for executing the functions of the respective units in FIG. 5 is stored and distributed on a computer-readable recording medium such as a flexible disk, CD-ROM, DVD, and MO, and the program is installed in the computer. A computer that can exhibit the functions of the above-described units may be configured.
Further, the program may be stored in a disk device or the like of a server device on a communication network such as the Internet so that, for example, the computer can download it.

  As mentioned above, although embodiment of this invention was described, this embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take various other embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope of the invention described in the claims and equivalents thereof. The invention described in the scope of claims at the beginning of the present application will be appended.

(Appendix 1)
Generating means for generating a plurality of reconstructed images obtained by reconstructing a light field image including a plurality of sub-images obtained by photographing subjects from different viewpoints, from a plurality of the light field images having different exposures;
Determining means for determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying unit for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction means for reducing a data amount of a sub image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
An image processing apparatus comprising: storage means for storing a plurality of light field images whose data amount has been reduced by the reduction means.

(Appendix 2)
The image processing apparatus according to appendix 1, wherein the reduction unit sets a pixel that does not correspond to any of the identified pixels as a pixel that satisfies the predetermined condition.

(Appendix 3)
The reduction means changes a pixel value of a pixel of the sub-image to a specific value for a sub-image including a pixel satisfying the predetermined condition, and compresses the light field image including the changed sub-image. ,
The image processing apparatus according to appendix 1 or 2, wherein the storage unit stores a plurality of the compressed light field images.

(Appendix 4)
The reduction means changes the pixel values of all the pixels constituting the sub-pixel to a specific pixel value when all the pixels constituting the sub-image satisfy the predetermined condition. The image processing apparatus according to appendix 3.

(Appendix 5)
In any of the sub-images included in each of the plurality of light field images, the reduction means includes, when all pixels constituting an area having an area equal to or larger than a predetermined threshold value are pixels that satisfy the predetermined condition, The image processing apparatus according to appendix 1 or 2, wherein the image processing apparatus is deleted to reduce the data amount.

(Appendix 6)
A plurality of sub-images obtained by photographing the subject from different viewpoints using a main lens that captures light from the subject and a plurality of microlenses that respectively correspond to a plurality of viewpoints that further capture the light captured by the main lens Photographing means for photographing a light field image including
Generation means for generating a plurality of reconstructed images by reconstructing each of the plurality of light field images photographed with different exposures by the photographing means;
Determining means for determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying unit for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction means for reducing a data amount of a sub image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
An image pickup apparatus comprising: storage means for storing a plurality of light field images in which the data amount is reduced by the reduction means.

(Appendix 7)
Generating a plurality of reconstructed images obtained by reconstructing a light field image including a plurality of sub-images obtained by photographing subjects from different viewpoints, from a plurality of the light field images having different exposures;
A determining step of determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying step for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction step of reducing a data amount of a sub-image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
A storage step of storing a plurality of light field images whose data amount has been reduced by the reduction step.

(Appendix 8)
Computer
Generating means for generating a plurality of reconstructed images obtained by reconstructing a light field image including a plurality of sub-images obtained by photographing subjects from different viewpoints, from a plurality of the light field images having different exposures;
Determining means for determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying unit for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction means for reducing a data amount of a sub-image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
A program for causing a storage unit to store a plurality of light field images whose data amount has been reduced by the reduction unit.

DESCRIPTION OF SYMBOLS 100 ... Digital camera, 110 ... Image pick-up part, 111 ... Main lens, 112 ... Micro lens array, 113 ... Image sensor, 120 ... Image processing part, 121 ... CPU, ROM ... 122, RAM ... 123, SD ... 124, 130 ... Operation unit 140 ... Display unit 150 ... Reconstruction unit 160 ... HDR synthesis unit 170 ... Identification unit 180 ... Reduction unit 190 ... Storage unit

Claims (8)

Generating means for generating a plurality of reconstructed images obtained by reconstructing a light field image including a plurality of sub-images obtained by photographing subjects from different viewpoints, from a plurality of the light field images having different exposures;
Determining means for determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying unit for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction means for reducing a data amount of a sub image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
An image processing apparatus comprising: storage means for storing a plurality of light field images whose data amount has been reduced by the reduction means.   The image processing apparatus according to claim 1, wherein the reduction unit sets a pixel that does not correspond to any of the identified pixels as a pixel that satisfies the predetermined condition. The reduction means changes a pixel value of a pixel of the sub-image to a specific value for a sub-image including a pixel satisfying the predetermined condition, and compresses the light field image including the changed sub-image. ,
The image processing apparatus according to claim 1, wherein the storage unit stores a plurality of the compressed light field images.   The reduction means changes the pixel values of all the pixels constituting the sub-pixel to a specific pixel value when all the pixels constituting the sub-image satisfy the predetermined condition. The image processing apparatus according to claim 3.   In any of the sub-images included in each of the plurality of light field images, the reduction means includes, when all pixels constituting an area having an area equal to or larger than a predetermined threshold value are pixels that satisfy the predetermined condition, The image processing apparatus according to claim 1, wherein the image processing apparatus is deleted to reduce a data amount. A plurality of sub-images obtained by photographing the subject from different viewpoints using a main lens that captures light from the subject and a plurality of microlenses that respectively correspond to a plurality of viewpoints that further capture the light captured by the main lens Photographing means for photographing a light field image including
Generation means for generating a plurality of reconstructed images by reconstructing each of the plurality of light field images photographed with different exposures by the photographing means;
Determining means for determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying unit for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction means for reducing a data amount of a sub image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
An image pickup apparatus comprising: storage means for storing a plurality of light field images in which the data amount is reduced by the reduction means. Generating a plurality of reconstructed images obtained by reconstructing a light field image including a plurality of sub-images obtained by photographing subjects from different viewpoints, from a plurality of the light field images having different exposures;
A determining step of determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying step for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction step of reducing a data amount of a sub-image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
A storage step of storing a plurality of light field images whose data amount has been reduced by the reduction step. Computer
Generating means for generating a plurality of reconstructed images obtained by reconstructing a light field image including a plurality of sub-images obtained by photographing subjects from different viewpoints, from a plurality of the light field images having different exposures;
Determining means for determining a composition ratio of the plurality of reconstructed images for each pixel according to a pixel value of each corresponding pixel in the plurality of reconstructed images;
A specifying unit for specifying a pixel having a combination ratio equal to or higher than a predetermined ratio among the pixels constituting the plurality of reconstructed images;
In the light field image, a reduction means for reducing a data amount of a sub-image including a pixel whose relationship with the identified pixel satisfies a predetermined condition;
A program for causing a storage unit to store a plurality of light field images whose data amount has been reduced by the reduction unit.