JP2013176073A - Imaging apparatus, image display method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a reconstructed image with a small calculation amount from images obtained by photographing a subject from a plurality of view points.SOLUTION: A sub-image extraction section 711 extracts a target sub-image from a light field image. A partial region definition section 712 defines a prescribed region on the target sub-image as a partial region. A pixel extracting section 713 extracts the number of pixels, which is matched with corresponding regions of a generated image from the partial region. A pixel arranging section 714 arranges the extracted pixels in the corresponding regions of the generated image in arrangement corresponding to an optical path of an optical system which photographs the light field image. The pixels of all the sub-images of the light field image are extracted and are arranged in the generated image, and a reconstructed image is generated.

Description

  The present invention relates to a technique for reconstructing an image.

2. Description of the Related Art In recent years, an imaging apparatus that captures information about a direction distribution of incident light rays, that is, an imaging apparatus called a “plenoptic camera” is known (see Patent Document 1).
In the optical system of a plenoptic camera, a very small lens (hereinafter referred to as a “micro lens”) is repeatedly arranged between a conventional imaging lens (hereinafter referred to as a “main lens”) and an image sensor. A compound eye lens (hereinafter referred to as “microlens array”) is inserted.

The individual microlenses constituting the microlens array distribute the light collected by the main lens to a plurality of pixel groups in the image sensor according to the angle reached.
In other words, if the image focused on the image sensor by each microlens is hereinafter referred to as a “sub-image”, image data consisting of a collection of a plurality of sub-images is output from the image sensor as image data. The
Such a captured image of the plenoptic camera is hereinafter referred to as a “light field image”.

  The light field image is generated not only by a conventional main lens but also by light incident through a microlens array. For this reason, the light field image has two-dimensional spatial information indicating the light ray that has reached from which part, which is included in the conventional captured image, as well as the conventional captured image. As information that is not included in, there is two-dimensional direction information indicating from which direction the light beam has arrived as viewed from the image sensor.

After capturing a light field image, the plenoptic camera can reconstruct an image of a surface that has been separated forward by an arbitrary distance using the data of the light field image.
In other words, even if the plenoptic camera captures a light field image without focusing at a predetermined distance, it uses the data of the light field image after the imaging to match the light field image. Data of an image that is captured in focus (hereinafter referred to as “reconstructed image”) can be freely created.

Special table 2008-515110 gazette

  In the technique described in Patent Document 1, the amount of calculation for generating a reconstructed image from a light field image is large. Therefore, even when it is required to generate and display an image early, it takes time to display the image, and the convenience for the user is low.

  The present invention has been made in view of such a situation, and an object thereof is to generate a reconstructed image with a small amount of calculation.

In order to achieve the above object, an image processing apparatus according to the present invention provides:
An acquisition unit for acquiring a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned;
An extraction unit that extracts a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
A generation unit configured to generate the reconstructed image by arranging the partial images in the arrangement order of the sub-images corresponding to the partial images;
It is provided with.

  According to the present invention, a reconstructed image can be generated with a small amount of calculation.

1A is a front view of a digital camera according to Embodiment 1 of the present invention. FIG. FIG. 2B is a rear view of the digital camera according to the first embodiment. 2 is a block diagram illustrating a hardware configuration of the digital camera according to Embodiment 1. FIG. FIG. 3 is a schematic diagram illustrating a configuration example of an optical system in the digital camera according to the first embodiment. FIG. 4A is a conceptual diagram of a light field image according to the first embodiment. (B) is a diagram showing an example of a photographed subject. (C) is a figure which shows the light field image which image | photographed (b). (D) is a partially enlarged view of (c). 3 is a schematic diagram illustrating a functional configuration of the digital camera according to Embodiment 1. FIG. 3 is a flowchart illustrating image output processing according to the first embodiment. 3 is a schematic diagram illustrating a functional configuration of a live view image generation unit according to Embodiment 1. FIG. 6 is a flowchart illustrating a flow of image generation processing executed by the digital camera according to the first embodiment. (A) is a figure which shows the light field image which concerns on Embodiment 1. FIG. (B) is a figure which shows the partial area | region which concerns on Embodiment 1. FIG. (C) is a figure which shows the 1st reconstruction image which concerns on Embodiment 1. FIG. (A) is a schematic diagram which shows the function structure of the confirmation image generation part which concerns on Embodiment 1. FIG. (B) is a figure which shows the example of the area | region definition table which concerns on Embodiment 1. FIG. 6 is a flowchart illustrating a flow of image generation processing executed by the digital camera according to the first embodiment. (A) is a figure which shows the light field image which concerns on Embodiment 1. FIG. (B) is a figure which shows the partial area | region which concerns on Embodiment 1. FIG. (C) is a figure which shows the 2nd reconstruction image which concerns on Embodiment 1. FIG. FIG. 2 is a schematic diagram illustrating a functional configuration of a main image generation unit according to the first embodiment. 6 is a flowchart illustrating a flow of image generation processing executed by the digital camera according to the first embodiment. (A) is a figure which shows the light field image which concerns on Embodiment 1. FIG. (B) is a figure which shows the partial area | region which concerns on Embodiment 1. FIG. (C) is a figure which shows the 3rd reconstruction image which concerns on Embodiment 1. FIG. It is a flowchart which shows the image output process which concerns on Embodiment 2 of this invention. It is a flowchart which shows the image output process which concerns on Embodiment 3 of this invention. 10 is a schematic diagram illustrating a functional configuration of a live view image generation unit according to Embodiment 3. FIG. 10 is a flowchart illustrating image generation processing executed by the digital camera according to the third embodiment. It is a schematic__ figure which shows the function structure of the live view image generation part which concerns on Embodiment 4 of this invention. 10 is a flowchart illustrating image output processing according to the fourth embodiment. 10 is a flowchart illustrating a deviation coefficient estimation process according to the fourth embodiment. It is a figure which shows the example of grouping of the estimated image and calculation image which concern on Embodiment 4. FIG. It is a figure which shows the example of grouping of the estimated image and calculation image which concern on Embodiment 4. FIG. FIG. 10 is a diagram illustrating an overview of an example of an image shift coefficient estimation process according to a fourth embodiment.

  Hereinafter, an imaging device (digital camera) according to an embodiment for carrying out the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

(Embodiment 1)
The appearance of a digital camera 1 as an embodiment of an imaging apparatus according to the present invention will be described with reference to FIG. FIG. 1A shows the appearance from the front, and FIG. 1B shows the appearance from the back. As shown in FIG. 1A, the digital camera 1 has an imaging lens (lens group) 12 on the front side. On the back of the digital camera 1, as shown in FIG. 1B, a liquid crystal monitor screen 13 as a display device, a mode dial, a cursor key, a SET key, a zoom button (W button, T button), a menu key Etc., and an operation unit 14 made up of and the like. A shutter key 10 and a power button 11 are provided on the top surface. Although not shown in the figure, a USB terminal connection portion used when connecting to an external device such as a personal computer (hereinafter referred to as a personal computer) or a modem and a USB cable is provided on the side portion.

  Next, the hardware configuration of the digital camera 1 will be described with reference to a block diagram (FIG. 2).

  The digital camera 1 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, and an internal bus 20. The digital camera 1 also includes an input / output interface 30, an imaging unit 31, an input unit 32, an output unit 33, a storage unit 34, a display unit 35, a communication unit 36, and a media drive 37. ing.

  The CPU 21 uses the RAM 23 as a work area, and executes later-described processing for outputting an image by a program stored in the ROM 22 or the storage unit 34.

  The ROM 22 and / or the storage unit 34 stores data necessary for the CPU 21 to execute various processes described later. The stored data is loaded into the RAM 23 as appropriate. Further, the RAM 23 stores intermediate data of processing to be described later in a timely manner.

  The CPU 21, ROM 22, RAM 23, and input / output interface 30 are connected to each other via the internal bus 20. In addition, an imaging unit 31, an input unit 32, an output unit 33, a storage unit 34, a display unit 35, a communication unit 36, and a media drive 37 are connected to the input / output interface 30.

  The imaging unit 31 includes a main lens 311, a microlens array 312, and an imaging element 313. Further details of the imaging unit 31 will be described later with reference to FIG.

  The input unit 32 receives input information such as the shutter key 10, various buttons of the operation unit 14, and a touch panel set in the display unit 35, and information on operations performed by the user using these input devices. And a transmission unit for transmitting to the network. The user can input commands to the digital camera 1 using the input unit 32 and can input various information.

The output unit 33 includes a monitor, a speaker, and the like, and outputs various images and various sounds generated by the processing of the CPU 21.
The storage unit 34 is composed of a hard disk, DRAM (Dynamic Random Access Memory), etc., and is transmitted from the CPU 21 or input from other devices, such as light field images and reconstructed images described later, Store setting information.
The communication unit 36 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 38 (a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like) is appropriately attached to the media drive 37. The program read from the removable medium 38 by the media drive 37 is installed in the storage unit 34 as necessary. The removable medium 38 can store various data such as image data stored in the storage unit 34 in the same manner as the storage unit 34. The display unit 35 includes a liquid crystal display, an organic EL (Electro-Luminescence) display, and the like. The display unit 35 displays a live view image in a shooting preparation state and a confirmation image after shooting, which are generated by processing described later and transmitted from the CPU 21. Hereinafter, the display unit 35 will be described as having a resolution of 320 × 240 pixels.

  A configuration example of the optical system in the digital camera 1 having such a configuration will be described with reference to FIG.

In the optical system of the digital camera 1, a main lens 311, a microlens array 312, and an image sensor 313 are arranged in that order when viewed from the subject OB.
In the microlens array 312, N × M (N and M are arbitrary integer values of 2 or more) microlenses 312-1 to 312-N × M are arranged. FIG. 3 shows one horizontal row (N pieces) of microlenses. In the following description, it is assumed that the microlenses 312-i (i is an integer value in the range of 1 to N × M) have the same size and are arranged in a grid pattern on the microlens array 312 at the same interval. To do.

  The main lens 311 condenses the light beam emitted from a point on the subject OB, forms an image on a predetermined surface MA, and enters the micro lens array 312. Hereinafter, the surface imaged by the main lens 311 is referred to as a “main lens imaging surface MA”. In the present embodiment, it is assumed that the main lens imaging surface MA is between the main lens 311 and the image sensor 313.

The micro lens 312-i condenses the light beam incident from the subject OB through the main lens 311 for each incident direction and forms an image on the image sensor 313 as a sub image.
That is, in the image sensor 313, a plurality of sub-images are formed by each of the plurality of microlenses 312-1 to 312-N × M, and a light field image that is an aggregate of the plurality of sub-images is formed. Generated.
In this way, images of the same subject viewed from different angles are recorded in a plurality of corresponding sub-images. In other words, an image in which a plurality of sub-images in which the subject is viewed from a plurality of viewpoints at the same time is aligned is obtained.

  The sub-images thus obtained include images viewed from different angles with respect to the same subject. Therefore, by selecting and synthesizing appropriate pixels from multiple sub-images, you can arbitrarily change the depth of field or the image focused on the surface that was at the front at an arbitrary distance during imaging. The reconstructed image can be reconstructed.

An example of a light field image LFI obtained by photographing a block-like subject OB is shown in FIG.
This light field image (LFI) is composed of images (sub-images, sub-images S _{11 to} S _MN ) corresponding to each of N × M microlenses 312-i arranged in a grid pattern. For example, the upper left sub-image S ₁₁ corresponds to an image obtained by photographing the object OB from the upper left, the sub-image S _MN the lower right corresponds to an image obtained by photographing the object OB from the lower right.

The i-th row sub-images (horizontal row sub-images) Si <b> 1 to SiN are images formed by the main lens 311 and formed by the microlenses 312-i arranged next to the i-th row of the microlens array 312. It corresponds to a stereo image. Similarly, the j-th row sub-images (vertical one-row sub-images) S1j to SMj are images of the main lens 311 formed into an image formed by microlenses 312-i arranged vertically in the jth row of the microlens array 312. Corresponds to the stereo image taken in.
In the present embodiment, each sub image S is a gray scale image, and each pixel constituting the sub image has a pixel value (scalar value).

  In the present embodiment, the microlens array 312 includes 80 microlenses arranged horizontally (x axis) and 60 microlenses arranged vertically (y axis). The sub-images corresponding to each microlens fall within a 60 × 60 pixel area to form an LFI having a resolution of 4800 × 3600 pixels as a whole.

  In LFI, a subject is reflected in a point object in units of sub-images. FIG. 4C shows an example of LFI obtained by photographing the subject shown in FIG. When this part (the part surrounded by the gray square) is enlarged (FIG. 4 (d)), it can be seen that the subject (the tip of FIG. 4 (b)) is inverted vertically and horizontally. The digital camera 1 generates and outputs a reconstructed image RI (an image as shown in FIG. 4B) from such an LFI.

  As shown in FIG. 5, the digital camera 1 includes an image capturing unit 40, an input unit 50, an LFI generation unit 610, a live view image generation unit 620, a confirmation image generation unit 630, and a main image generation unit 640 as shown in FIG. 5. It functions as the generation processing unit 60, the display unit 70, and the storage unit 80.

  The image capturing unit 40 captures the subject with the main lens 311 and the microlens array 312 and transmits the information to the LFI generating unit 610. The input unit 50 acquires shooting settings stored in the ROM 22 and the RAM 23, user operations received by the input unit 32, and reconstruction settings stored in the ROM 22 or RAM 23, and transmits them to each unit of the image generation processing unit 60.

  The LFI generation unit 610 generates an LFI as illustrated in FIG. 4A from the information transmitted from the image capturing unit 40. The LFI generation unit 610 transmits the generated LFI to the live view image generation unit 620, the confirmation image generation unit 630, and the main image generation unit 640.

  The live view image generation unit 620, the confirmation image generation unit 630, and the main image generation unit 640 generate an image (reconstructed image) from the sub image based on the position of each microlens from the LFI.

Processing for generating an image (reconstructed image) from the sub-image based on the position of each microlens from the light field image is hereinafter referred to as “reconstruction processing”. In the reconstruction process, a surface on which a subject to be reconstructed virtually exists is referred to as a “reconstructed surface”.
The live view image generation unit 620 generates an image for live view (live view image) among the reconstructed images. Processing for generating a live view image is referred to as “live view image generation processing”. In live view, in order to display the preparation and shooting status at the time of shooting to the user, it is desirable to display the subject image by switching at high speed. Therefore, in the present embodiment, the live view image generation process is set with a small amount of calculation. The live view image generation unit 620 outputs the generated live view image to the display unit 70. The live view image generation unit 620 is also referred to as a first generation unit.

  In addition, the confirmation image generation unit 630 generates an image (confirmation image) for confirmation after image capturing among the reconstructed images. The process for generating the confirmation image is referred to as “confirmation image generation process”. In the confirmation after shooting, the image quality is not as high as that for viewing images, but high image quality is desired so that what kind of image is obtained can be confirmed. Further, although it is necessary to display promptly after shooting, it is not necessary to generate as quickly as the live view image. Therefore, the necessary amount of calculation is set more in the confirmation image generation process than in the live view image generation process so that an image with higher image quality can be generated. The confirmation image generation unit 630 outputs the generated confirmation image to the display unit 70. Note that high image quality here means that the resolution is high, the reconstruction accuracy is high, the noise is low, the proper blur is applied, and the characteristics of this confirmation are reflected more accurately. This is an image that is convenient for all users. The confirmation image generation unit 630 is also referred to as a second generation unit.

  The main image generation unit 640 generates an ornamental image (main image) among the reconstructed images. The process of generating the main image is called “main image generation process”. Since this image can withstand viewing, it is desired that the image matches the reconstruction setting desired by the user and is a high-quality image. Therefore, a large amount of calculation is required for the image generation process. The main image generation unit 640 stores the generated main image in the storage unit 80. This image generation unit 640 is also referred to as a third generation unit. The live view image generation unit 620, the confirmation image generation unit 630, and the main image generation unit 640 may be simply referred to as a generation unit without being distinguished from each other.

The display unit 70 displays the transmitted reconstructed image. The display unit 70 has a resolution of 320 × 240 pixels.
The storage unit 80 stores the transmitted main image. The storage unit 80 may store the main image together with the corresponding confirmation image.

Processing for acquiring an LFI executed by the digital camera 1 and outputting a reconstructed image (image output processing 1) will be described with reference to FIG.
When the digital camera 1 is turned on and the input unit 50 receives an operation for preparing for shooting, the digital camera 1 starts an image output process 1 shown in FIG.

  In the image output process 1, first, the image capturing unit 40 and the LFI generation unit 610 generate an LFI (step S101).

Next, the input unit 50 acquires the current imaging setting information stored in the RAM 23 and the ROM 22 (step S102). The imaging settings include the current focal length f _ML of the main lens 311, aperture (F value), position information of each microlens in the microlens array 312, position information between the microlens array 312 and the image sensor 313, and the like. It is.

  When the imaging setting information is acquired, the live view image generation unit 620 executes live view image generation processing (step S103). In the present embodiment, the live view image generation process is an image generation process 1 described later.

  A functional configuration of the live view image generation unit 620 that executes the live view image generation process will be described with reference to FIG. In the present embodiment, the live view image generation unit 620 is the first reconstructed image generation unit 710 illustrated in FIG.

  The first reconstructed image generation unit 710 includes a sub image extraction unit 711, a partial region definition unit 712, a pixel extraction unit 713, a pixel arrangement unit 714, and an output unit 715. The first reconstructed image generation unit 710 generates a first reconstructed image (live view image of the present application) as an output image from the LFI using these units. The first reconstructed image is a 320 × 240 pixel image.

  The sub-image extraction unit 711 sequentially extracts the sub-images constituting the LFI as the attention sub-images and transmits them to the partial region definition unit 712.

  The partial region definition unit 712 defines a predetermined region of the target sub-image transmitted from the sub-image extraction unit 711 as a partial region. Details of the partial area will be described later. The partial region definition unit 712 transmits information indicating the partial region to the pixel extraction unit 713.

  The pixel extraction unit 713 extracts pixels under a predetermined condition from the partial region on the target sub-image indicated by the information transmitted from the partial region definition unit 712. The specific contents of the pixel extracting process will be described later. The pixel extraction unit 713 transmits the extracted pixel to the pixel arrangement unit 714.

The pixel placement unit 714 places the pixels extracted by the pixel extraction unit 713 on the first reconstructed image.
Specific contents of the process of arranging the pixels will be described later.

The pixel arrangement unit 714 from the sub image extraction unit 711 generates a first reconstructed image by sequentially extracting and arranging the pixels on the first reconstructed image using all the sub images as the target image.
The output unit 715 outputs the first reconstructed image generated in this way to the outside (display unit 70).

Next, reconstructed image generation processing (image generation processing 1) executed by the first reconstructed image generation unit 710 will be described with reference to FIGS.
When the live view image generation unit 620 (first reconstructed image generation unit 710) reaches step S103 of the image output process 1 (FIG. 6), it starts the image generation process 1 shown in FIG.

  In the image generation processing 1, first, the sub image extraction unit 711 extracts the k1 th sub image of the LFI as a target sub image using k1 as a counter variable (step S201).

  Next, the partial region defining unit 712 defines a predetermined region of the target sub-image (here, a square region of 8 × 8 pixels located at the center of the sub-image) as a partial region (step S202). The size of the predetermined area is set based on the size of pixel shift (parallax) that occurs with respect to a subject that can be photographed in the design of the digital camera 1. For example, when the pixel shift that occurs between the closest distance and the far distance that can be photographed by the camera is 4 to 12 pixels, 8 pixels near the middle are employed.

  When the partial area is defined, the pixel extracting unit 713 extracts the pixels of the partial area by the number of pixels of the area (corresponding area) corresponding to the target sub-image in the first reconstructed image described later (step S203).

And the pixel arrangement | positioning part 714 arrange | positions the extracted pixel to the corresponding | compatible part of the corresponding | compatible area | region of a 1st reconstruction image (generated image) (step S204).
The processing from step S201 to step S204 will be described with reference to FIG. 9 using a specific example.

In the LFI, as shown in FIG. 9A, substantially circular sub-images S _{11 to} S _MN are arranged in a lattice pattern. Extracting from the sub-image _{S 11} as the target sub image sequentially (Step S201), and extracts the square portion of the 8 × 8 pixels of the central portion as a partial area (step S202).

An enlarged view of the partial area is shown in FIG. Left side of FIG. 9 (b) sub-image _{S 11,} the right side is a partial area of the sub-image _{S 23.} The pixels in each partial area are represented by a grid and represented by coordinates (1, 1) to (8, 8).

An example of the first reconstructed image is shown in FIG. In this example, the first reconstructed image is an image of 320 × 240 pixels, and each of 80 × 60 sub-images is divided into corresponding regions (regions indicated by bold lines) of 4 × 4 pixels corresponding to the coordinates. Has been.
Corresponding region of the S ₁₁ is square part of the coordinates (1,1) to (4,4) in FIG. 9 _(c), the corresponding region of the _{S 23} coordinates (9,5) through (12,8) It is a square part.

In step S203, every other 4 × 4 pixels are extracted from the partial area. In step S204, the extracted pixels are inverted vertically and horizontally (point symmetry) for each sub-image and arranged in the corresponding region. This is because the main lens image formation surface MA exists in front of the image sensor 313, so that the image of the subject OB is inverted for each sub image.
In the example of FIG. _9, the pixels of the partial region of the _{S 11} (1, 1) is arranged at coordinates (4,4) in the first reconstructed image. _The pixel portion region of the _{S 11} (1, 3) is arranged at coordinates (3,4) in the first reconstructed image. Similarly, every other pixel is extracted and arranged from the partial area, and the pixel (7, 7) of the partial area is arranged at the coordinates (1, 1) of the first reconstructed image.

  Returning to FIG. 8, when the extracted pixels are arranged in all the pixels of the corresponding area in the current target sub-image in step S204, it is next determined whether or not the above processing has been executed for all the sub-images (step S205). If there is an unprocessed sub-image (step S205; NO), the counter variable k1 is incremented (step S206), and the processing from step S201 is repeated for the next target sub-image.

  On the other hand, when all the sub-images have been processed (step S205; YES), the image generation process 1 ends with the completed first reconstructed image as a live view image.

If the number of pixels in the partial area and the corresponding area is the same, the extraction process may be omitted, and all the pixels in the partial area may be inverted in a point-symmetric manner and arranged in the corresponding area.
If the partial area is 11 × 11 pixels and the corresponding area is 4 × 4 pixels, etc., and the pixels cannot be extracted at equal intervals, the first pixel, the fourth pixel, the seventh pixel, and the eleventh pixel The pixels are extracted by an extraction method that is close to regular intervals, such as extracting the pixels. Or you may resize to a corresponding area | region (4x4 pixel), interpolating a partial area | region (11x11 pixel).

Returning to FIG. 6, when the generation of the live view image is completed, the display unit 70 displays the generated live view image (step S104).
And it is discriminate | determined whether the input part 50 detected imaging operation (operation which presses the shutter key 10) (step S105). If it is determined that the imaging operation has not been detected (step S105; NO), the digital camera 1 returns to step S101 and repeats the process for displaying the live view image.

  On the other hand, if it is determined that an imaging operation has been detected (step S105; YES), the confirmation image generation unit 630 next executes a confirmation image generation process (step S106). In this embodiment, the confirmation image generation process is an image generation process 2 described later.

  A functional configuration of the confirmation image generation unit 630 that executes the confirmation image generation process will be described with reference to FIG. In the present embodiment, the confirmation image generation unit 630 is the second reconstructed image generation unit 720 illustrated in FIG.

  The second reconstructed image generation unit 720 includes a sub image extraction unit 721, a deviation value calculation unit 722, a partial region definition unit 723, an arrangement image generation unit 724, an image arrangement unit 725, and an output unit 726. Composed. The second reconstructed image generation unit 720 generates a second reconstructed image (confirmation image of the present embodiment) as an output image from the LFI by these units. The second reconstructed image is a 320 × 240 pixel image.

  The sub-image extraction unit 721 sequentially extracts the sub-images constituting the LFI as the attention sub-image and transmits the extracted sub-images to the deviation value calculation unit 722.

The deviation value calculation unit 722 calculates a coefficient (image deviation coefficient) indicating how much the target sub-image is deviated from the surrounding sub-images (peripheral sub-images) around the predetermined range centering on the target sub-image. . The image shift coefficient is calculated by the following method, for example.
A predetermined area at the center of the target sub-image determined by design (for example, a 10 × 10 area at the center) is set as the central area, and a portion corresponding to the central sub-image is adjacent to the right (if not present, left adjacent; Find where the sub-image is. For this, any known method for determining where an image part is located in another image can be used. In the present embodiment, the following method is used.
First, a center 10 × 10 region on the right side is set as a calculation target part. Then, the sum of absolute differences of the pixel values of the central part and the calculation target part is taken. Next, the sum of absolute differences is similarly taken for the area shifted by one pixel to the right. This is repeated for the range of possible parallax. The obtained position shift with the smallest sum of absolute difference values is defined as an image shift coefficient.

  In addition, when acquiring an image shift coefficient from two right-side sub-images, a difference absolute value sum is obtained at a position shifted by d pixels for the right-side sub-image, and a position shifted by 2d pixels for the next sub-image. Then, the sum of the absolute difference sums is obtained, and d that minimizes the evaluation function may be obtained using the sum of the two values as the evaluation function. When the number of sub-images increases, the same processing can be performed thereafter.

The partial area definition unit 723 acquires the size of the partial area corresponding to the image deviation coefficient calculated by the deviation value calculation unit 722 with reference to the area table stored in the ROM 22.
As shown in FIG. 10B, the region definition table records the range of the image shift coefficient and the value indicating the size of the region in association with each other. For example, when the image deviation coefficient is 8, the partial area is an area of the center 6 × 6 pixels of the target sub-image. The partial area definition unit 723 transmits information indicating the defined partial area to the arrangement image generation unit 724.

Note that the area definition table is desirably a table in which the size of the partial area increases approximately in proportion to a predetermined offset value when the image shift coefficient increases.
The method for obtaining the size of the area is not limited to this, and the area size may be calculated using an arbitrary table or calculation formula that increases the size (natural number) of the partial area within a predetermined range as the image shift coefficient increases.

  When the arrangement image generation unit 724 receives the information defining the partial area, the arrangement image generation unit 724 resizes the image included in the area by interpolation, and generates an arrangement image for arrangement in the generation image (second reconstructed image). Specifically, the image is resized to a size (4 × 4 pixels here) that matches the size of the corresponding region of the generated image (second reconstructed image).

  The image placement unit 725 places the placement image generated by the placement image generation unit 724 on the second reconstructed image.

The sub image extraction unit 721 to the image arrangement unit 725 generate arrangement images for all sub images, and the image arrangement unit 725 arranges them on the second reconstructed image to generate a second reconstructed image.
The output unit 726 outputs the second reconstructed image generated in this way to the outside (display unit 70).

Next, processing (image generation processing 2) executed by the second reconstructed image generation unit 720 will be described with reference to FIGS.
When the confirmation image generation unit 630 (second reconstructed image generation unit 720) reaches step S106 of the image output process 1 (FIG. 6), it starts the image generation process 2 shown in FIG.

  In the image generation process 2, first, the sub-image extraction unit 721 extracts k2 as the target sub-image of the LFI using k2 as a counter variable (step S301).

  Next, the deviation value calculation unit 722 indicates how many pixels the subject in the predetermined area (center 10 × 10 pixels) of the target sub-image is deviated from the center in the right-side sub-image. The image shift coefficient is calculated by the method described above (step S302).

  Next, the partial area definition unit 723 defines a partial area of the target sub-image (step S303). In step S303, the size L of the area corresponding to the image shift coefficient obtained in step S302 is acquired with reference to the area definition table. Then, an area of that size (a square area of L × L pixels) is defined as a partial area at the center of the target sub-image.

Then, the arrangement image generation unit 724 resizes the image of the partial area of L × L pixels so as to match the area (corresponding area) corresponding to the target sub-image in the second reconstructed image to be an arrangement image (step) S304). The size of the corresponding area is a size obtained by distributing the second reconstructed image (320 × 240 pixels in this case) determined in the setting to each sub-image, and follows a preset stored value.

Then, the arrangement image is arranged at a site corresponding to the target sub-image of the second reconstructed image (generated image) (step S305). Here, they are arranged so as to be reversed left and right and up and down.
A specific example of the processing from step S301 to step S305 will be described with reference to FIG.

In the LFI, as shown in FIG. 12A, substantially circular sub-images S _{11 to} S _MN are arranged in a lattice pattern. Extracted as a sub image of interest in order from S ₁₁ (step S 301), and acquires a position shift coefficient with the sub image on the right (step S 302). Figure 12 (a), the image shift coefficients of the _{S 12} of _{S 11} indicated by a thick arrow is _large, the image shift factors and _{S 24} of _{S 23} is small.

The definition is made so that the partial area becomes larger as the image shift coefficient is larger (step S303).
As shown in FIG. 12 (b) in the present _example, a 10 × 10 pixel square portions of the central portion of _{S 11} as a partial region of the _{S 11,} the 3 × 3 pixels square portion of the central portion of _{S 23} as partial region of the S _23, are defined respectively.

  An example of the second reconstructed image is shown in FIG. In this example, the second reconstructed image is an image of 320 × 2_40 pixels, and for each of 80 × 60 sub-images, a corresponding region of 4 × 4 pixels (region indicated by a bold line) corresponding to the coordinates is defined. Is done.

  In step S304, the image of the partial area is resized to a 4 × 4 arrangement image, and in the subsequent step S305, the arrangement image is inverted vertically and horizontally (point object) and arranged in the corresponding area. This is because the image of the subject OB is inverted for each sub-image because the main lens imaging surface MA exists in front of the image sensor 313. Resizing is only reduced when the second reconstructed image corresponding to the number of pixels on the liquid crystal monitor screen 13 is small, and the reduction rate changes according to the image shift coefficient. On the other hand, if the number of pixels on the liquid crystal monitor screen 13 is large and the corresponding area is large, the image is enlarged or reduced according to the image shift coefficient.

  Returning to FIG. 11, when an image is arranged in the corresponding area for the current target sub-image in step S305, it is determined whether or not the above processing has been executed for all sub-images (step S306). If there is an unprocessed sub-image (step S306; NO), the counter variable k2 is incremented (step S307), and the processing from step S301 is repeated for the next target sub-image.

  On the other hand, when all the sub-images have been processed (step S306; YES), the image generation process 2 ends with the completed second reconstructed image as a confirmation image.

Here, the confirmation image has the same number of pixels (320 × 240 pixels) as the live view image corresponding to the number of pixels on the liquid crystal monitor screen 13. However, the confirmation image may be larger than the live view image.
Further, although the arrangement image is generated by resizing the partial area, the arrangement image may be generated by extracting the pixels in the partial area according to the size of the corresponding area.

Returning to FIG. 6, when the generation of the confirmation image is completed, the display unit 70 displays the generated confirmation image (step S107).
Then, the input unit 50 acquires the reconstruction setting for generating the main image from the storage unit 34, the input unit 32, and the like (step S108). The reconstruction setting includes information such as the distance between the main lens 311 and the reconstruction surface, the portion occupied by the reconstruction image, whether or not the filter is used, the filter setting, and the number of pixels of the reconstruction image.

  When the reconstruction setting is acquired, the main image generation unit 640 executes the main image generation process using the reconstruction setting (step S109). In the present embodiment, the main image generation processing is image generation processing 3 described later.

  A functional configuration of the main image generation unit 640 that executes the main image generation processing will be described with reference to FIG. In the present embodiment, the main image generation unit 640 is the third reconstructed image generation unit 730 shown in FIG.

  The third reconstructed image generation unit 730 includes a sub image extraction unit 731, a pixel shift coefficient calculation unit 732, a filter processing unit 733, an image shift coefficient calculation unit 734, a partial region definition unit 735, and an image arrangement unit 736. And an output unit 737. The third reconstructed image generation unit 730 generates a third reconstructed image (the main image of the present application) as an output image from the LFI using these units.

  The sub-image extraction unit 731 sequentially extracts the sub-images constituting the LFI as the attention sub-image and transmits the extracted sub-images to the pixel shift coefficient calculation unit 732.

  The pixel shift coefficient calculation unit 732 sequentially calculates each pixel of the target sub-image as the target pixel, and calculates a coefficient (pixel shift coefficient of the target pixel) indicating how much the pixel corresponding to the target pixel is shifted in the peripheral sub-image. To do. The pixel shift coefficient corresponds to the distance of the target pixel from the subject. The pixel shift coefficient may be calculated using any method for estimating the distance of the subject of the LFI pixel, but here it is calculated by the following method.

  The sub-images on the right side of the target sub-image (the left side when there is no sub-image, in this case, the left and right are reversed in this case) are set as SR1, SR2,. k is a natural number determined in setting. Further, the pixel value of the target pixel is assumed to be a pixel value a, and the coordinates on the target sub-image are set to (x, y). Taking the current pixel shift as d, the difference between the pixel value a and the pixel value of (x + d, y) of SR1 (the pixel value of the pixel shifted to the right by d) is taken. Similarly, the difference between SR2 and (x + 2d, y),..., The difference between SRk and (x + kd, y) is obtained, and the sum of absolute differences is obtained. This is repeated for the range of possible parallax. The obtained positional deviation with the smallest sum of absolute difference values is defined as a pixel deviation coefficient.

  The pixel shift coefficient calculation unit 732 calculates the pixel shift coefficients of all the pixels included in the target sub-image and transmits them to the filter processing unit 733. The filter processing unit 733 compares the distance to the subject corresponding to each pixel indicated by the pixel shift coefficient with the distance to the reconstruction plane included in the reconstruction setting acquired from the input unit 50. Then, filter processing corresponding to the difference is executed, and blur is added to each pixel of the light field image. If the difference between the refocus distance and the distance to the subject is greater than or equal to a predetermined value, blur addition processing is performed according to the set blur intensity. For example, a Gaussian filter is used for the blur addition process.

  The pixel shift coefficient (corresponding to the amount of misalignment) is small when a certain pixel is taken from a subject that is very far away, and large when the subject is taken from a nearby subject. The relationship between the positional deviation and the distance between the subject and the camera in the real world is determined by the focal length of the main lens 311 and the microlens 312-i in the microlens array 312, and the position and size of the image sensor 313. . The correspondence between the positional deviation and the subject distance is obtained in advance by experiments and stored in the ROM 22 as a deviation distance table.

  In the present embodiment, filter processing is executed using a deviation distance table. The deviation distance table is obtained in advance by experiments and stored in the ROM 22. The deviation distance table associates a level (LV) indicating a distance within a predetermined range, a deviation coefficient (image deviation coefficient and pixel deviation coefficient) belonging to that level, and a distance to a reconstruction plane belonging to that level. I remember it.

  The filter processing unit 733 calculates the difference between the level to which the pixel deviation coefficient belongs and the level to which the distance to the reconstruction plane belongs using a deviation distance table. The filter processing is not executed based on the assumption that the pixel having a difference of 0 corresponds to a subject that is sufficiently close to the reconstruction plane. That is, no blur is added. On the other hand, if the difference is 1 or more, the subject of the pixel is deviated from the reconstruction plane, so blur is added. As the difference becomes larger, the filtering process is executed so as to add stronger blur.

The image deviation coefficient calculation unit 734 calculates the image deviation coefficient of the target sub-image in the same manner as the deviation value calculation unit 722 in FIG. However, the image deviation coefficient calculation unit 734 calculates the image deviation coefficient in both the horizontal direction and the vertical direction, and then averages both to obtain the image deviation coefficient of the target image. In addition, the interval at which the partial area image is arranged in the reconstructed image is determined according to the image shift coefficient. The arrangement interval includes the image shift coefficient, the focal length of the main lens 311 included in the imaging setting information, the focal length of the microlens 312-i in the microlens array 312, the position where the imaging element 313 is placed, It is determined by the distance to the reconstruction surface included in the configuration settings.
The farther away the subject, the smaller the expected angle difference for each microlens, so the movement within the sub-image is also small. Accordingly, the arrangement interval is reduced. Conversely, the closer the subject, the greater the difference in the expected angle for each microphone_lens, so the movement within the sub-image is also greater. Accordingly, the arrangement interval is increased.

In the present embodiment, a determination is made using a table (arrangement interval table) stored in the ROM 22 in which the arrangement interval is determined using the focal length of the main lens 311, the distance to the reconstruction surface, and the image shift coefficient as indexes. And
The arrangement interval table has an image shift coefficient and a distance to the reconstruction surface according to the focal length of the microlens 312-i in the microlens array 312, which is a fixed parameter, and the position where the image sensor 313 is placed. And a suitable arrangement interval value obtained by experiment for each value of the focal length of the main lens 311 is defined.
The arrangement interval may be obtained from an equation determined in advance stored in the ROM 22 using the above parameters as variables.

The partial area definition unit 735 acquires the size of the partial area that matches the image deviation coefficient from the area definition table of FIG. 10B, and corrects the size based on the magnitude of the blur intensity of the sub image of interest.
The blur intensity of the target image is a coefficient that determines how much the entire target sub-image is deviated from the reconstruction plane and that portion should be displayed in a blurred manner. Here, the difference between the level of the image shift coefficient determined by the shift distance table described above and the level of the distance to the reconstruction surface is defined as the blur intensity.

  One side of the corrected partial region is larger than the arrangement interval. Therefore, when the partial region images are arranged in the third reconstructed image, the arrangement images overlap each other. Since the pixel values of the overlapped parts are averaged, the blurring becomes stronger as the number of arranged images to be overlapped increases. For this reason, if the blur intensity is strong, the correction is executed so that the size of the partial area becomes large.

The image arrangement unit 736 arranges the partial region image in the generated image (third reconstructed image) without resizing the image. At this time, the interval between the previously arranged partial regions and the image follows the arrangement interval obtained by the image deviation coefficient calculation unit 734. At this time, the arrangement interval and the size of the partial area are determined so that the arranged images overlap. For the overlapped portion, the pixel values of the pixels arranged in an overlapping manner are added and averaged to obtain the pixel value of the generated image.
The sub image extraction unit 731 to the image arrangement unit 736 perform the above process on all the sub images included in the LFI to generate a third reconstructed image.

Next, processing (image generation processing 3) executed by the third reconstructed image generation unit 730 will be described with reference to FIGS.
The main image generation unit 640 (third reconstructed image generation unit 730) starts image generation processing 3 shown in FIG. 14 when step S109 of the image output processing 1 (FIG. 6) is reached.

  In the image generation process 3, first, the sub-image extraction unit 731 extracts the k3th sub-image of LFI as a target sub-image using k3 as a counter variable (step S401).

  Next, for each pixel of the target sub-image, the pixel shift coefficient calculation unit 732 calculates the pixel shift coefficient as described above (step S402).

  Then, for each pixel of the target sub-image, the filter processing unit 733 calculates a difference (level difference) between the pixel shift coefficient and the reconstruction distance using the shift distance table (step S403).

  Next, a filtering process corresponding to the level difference is executed under the conditions stored in the ROM 22 for the pixels having a difference from the reconstruction distance of a predetermined value or more (the level difference is not 0) (step S404).

Next, the image shift coefficient calculation unit 734 acquires the image shift coefficient of the target sub-image (step S405). In step S405, in the same manner as in step S302 of FIG. 11, the image deviation coefficient is acquired for the right neighbor, and the image deviation coefficient is also obtained for the lower neighbor (upper when there is no image). The deviation coefficient.
Since the image deviation coefficient is calculated based on the positional deviation in both the vertical direction and the horizontal direction in exchange for the increase in the calculation amount, the image deviation coefficient that reflects the subject distance of the sub-image more accurately than step S302 in FIG. The image quality of the generated image is improved. As another method for improving the accuracy of the image shift coefficient, it is conceivable to calculate the image shift coefficient for more sub-images in one direction.
Note that the image deviation coefficient may be obtained by averaging the pixel deviation coefficients of each pixel of the sub-image obtained in step S402.

  Next, in step S <b> 406, the image deviation coefficient calculation unit 734 arranges numerical values of arrangement intervals corresponding to the image deviation coefficient, the focal length of the main lens 311, and the distance to the reconstruction surface, stored in the ROM 22. Read from the interval table and set as the arrangement interval.

Next, the partial area definition unit 735 acquires the size (L2 × L2) of the partial area that matches the image shift coefficient from the area definition table. Further, the correction is made so that the partial area increases as the blur intensity increases, and the size of the partial area (L2 × L2) is determined. Then, an L2 × L2 square area at the center of the target sub-image is defined as a partial area (step S407). Note that L2 is larger than the arrangement interval.
In the examples of FIGS. 15A and 15B, S ₁₁ and S ₁₂ have a large partial area because the image shift coefficient is large. On the other hand, S ₂₃ is the image shift factors are small, partial region is small.

Then, the partial region images are arranged in the third reconstructed image that is the generated image at the arrangement interval determined in step S406 (FIG. 15C, step S408). Here, they are arranged so as to be reversed left and right and up and down. Since the images overlap, the pixel values are determined by averaging the overlapped portions by the overlap. Figure 15 _(c), the image of the partial region of the _{S 12} as the image of the partial region of the _{S 11} (region of horizontal lines) (region of the vertical line) is overlapped.

  If an image is arranged in the corresponding region for the current sub image of interest in step S408, it is then determined whether the above processing has been executed for all sub images (step S409). If there is an unprocessed sub-image (step S409; NO), the counter variable k3 is incremented (step S410), and the processing from step S401 is repeated for the next target sub-image.

  On the other hand, when all the sub-images have been processed (step S409; YES), the image generation process 3 ends with the completed third reconstructed image as the main image.

  Returning to FIG. 6, when the generation of the main image is completed, the generated main image is stored in the storage unit 80 (step S110), and the image output process 1 ends.

  In the image generation process 3, blur processing for each sub image and blur processing for each pixel are executed. For this reason, even if an error is included in the subject distance estimation for each pixel, it is averaged by blur processing (for overwriting) for each sub-image, and the influence of the error is mitigated. Therefore, a main image with high image quality can be generated.

  As described above, the digital camera 1 according to the present embodiment includes a plurality of reconstructed image generation units that can be secured in accordance with calculation time and usage. Therefore, it is possible to generate and display a reconstructed image suitable for various uses such as live view, confirmation after shooting, and ornamental use at a preferable speed. As described above, the digital camera 1 according to the present embodiment can display a reconstructed image having a necessary and sufficient image quality at a sufficient speed according to the situation, and thus is highly convenient for the user.

Specifically, the first reconstructed image generation process can generate a reconstructed image (first reconstructed image) including the contents of all the sub-images included in the LFI with a small amount of calculation.
Therefore, a reconstructed image that can be used for shooting preparation even when a sufficient amount of calculation cannot be ensured for normal reconstructed image generation processing (eg, as exemplified in Patent Document 1) such as use in live view. Can be generated and displayed particularly fast. Also, it is possible to generate an image with higher image quality than when using an image in which pixels are extracted and arranged one by one from the sub-image.

  In addition, by the second reconstructed image generation process, a reconstructed image (second reconstructed image) having higher image quality than the first reconstructed image is generated in accordance with the coefficient corresponding to the subject distance in each sub-image unit. I can do it. The necessary calculation amount of this process is larger than that of the first reconstructed image generation process, but is smaller than that of a normal reconstructed image generation process (for example, as exemplified in Patent Document 1). Therefore, a reconstructed image having higher image quality than the first reconstructed image can be displayed at high speed for confirmation after shooting.

Further, the third reconstructed image generation process can generate a reconstructed image (third reconstructed image) with high image quality obtained by performing the blur addition process in units of sub-images and the blur addition process in units of pixels. The required calculation amount of this process is larger than that of the second reconstructed image generation process, but is smaller than that of a normal reconstructed image generation process (for example, as exemplified in Patent Document 1). Therefore, a reconstructed image with high image quality can be generated on a terminal such as a digital camera that limits the processing speed of the CPU.
Also, in the first to third reconstructed image generation processing, the image is inverted vertically and horizontally for each sub image, so that the image is correctly arranged from the image captured by being inverted for each sub image by the configuration of the optical system. The reconstructed image can be generated with a small amount of calculation.

Further, the digital camera 1 of the present embodiment is set so that a larger partial area is defined for a sub-image having a large image deviation coefficient, and the entire information of the partial area appears in the reconstructed image.
A large image shift coefficient indicates that the subject of the target sub-image is close to the main lens, and that the position of the adjacent sub-image that is visible due to the parallax of the corresponding microlens is greatly shifted. Here, the subject in the center is a subject representing the sub-image.
When the position of the corresponding subject is greatly shifted between the sub-images, the size of the partial area required to prevent missing information when connecting the partial areas of adjacent sub-images to form the entire image is large. Means. Therefore, in the present embodiment, the larger the image deviation coefficient, the larger the partial region is arranged in the corresponding region of the generated image so that no information is lost in the generated image. In addition, for an image with a small image shift coefficient, the partial area is made small so that duplication of information appearing between adjacent sub-images does not appear excessively on the reconstructed image.
In the digital camera 1 of the present embodiment, even a reconstructed image generated at high speed with such a configuration can reduce the degree of information omission and information duplication in consideration of positional deviation due to parallax.

(Embodiment 2)
A second embodiment of the present invention will be described.
The digital camera 1 of this embodiment has a physical configuration shown in FIG. In the digital camera 1 of the present embodiment, the CPU 21 is faster and the display unit 35 has a higher resolution, for example, VGA (640 × 480 pixels), compared to the corresponding part of the first embodiment. 2 are the same as those of the digital camera 1 of the first embodiment.
In the digital camera 1 of the present embodiment, since the display unit 35 is large, the method described in the first embodiment makes the image roughness noticeable. Therefore, by utilizing the high speed of the CPU 21, a reconstructed image to be displayed on the display unit 35 of the digital camera 1 with higher definition and higher image quality is created.

The digital camera 1 of this embodiment has a functional configuration shown in FIG.
In the present embodiment, the live view image generation unit 620 is the second reconstructed image generation unit 720, the confirmation image generation unit 630 is the third reconstructed image generation unit 730, and the main image generation unit 640 is the fourth reconstruction image. An image generation unit. The other units in FIG. 5 are the same as the corresponding configurations of the digital camera 1 of the first embodiment.

The second reconstructed image generation unit 720 and the third reconstructed image generation unit 730 have the same configuration as the part having the same name in the first embodiment.
The fourth reconstructed image generation unit executes a reconstructed image generation process, which will be described later, generates a reconstructed image from the LFI, and transmits the generated image to the storage unit 80.

  Processing executed by the digital camera 1 of the present embodiment will be described with reference to a flowchart.

  When the digital camera 1 is turned on and the input unit 50 receives an operation to prepare for shooting, the digital camera 1 starts an image output process 2 shown in FIG.

  In the image output process 2, steps S501 to S502 are executed in the same manner as steps S101 to S102 of the image output process in FIG.

  In the present embodiment, in step S503, the image generation process 2 (FIG. 11) is executed in the same manner as in the first embodiment to generate a live view image.

  When the generation of the live view image is completed in step S503, the digital camera 1 executes steps S504 to S505 in the same manner as steps S104 to S105 of the image output process 1 in FIG.

  In step S506, the confirmation image generation unit 630 that is the third reconstructed image generation unit 730 in FIG. 13 executes the image generation processing 3 shown in FIG. 14 in the same manner as in the first embodiment to generate a confirmation image.

  Returning to FIG. 16, when the generation of the confirmation image is completed in step S506, the digital camera 1 executes step S507 in the same manner as step S107 of the image output process of FIG.

In step S508, the main image generation unit 640 that is the fourth reconstructed image generation unit executes the image generation processing 4 to generate a main image.
The image generation process 4 performs image reconstruction by weighting and adding the pixel values of the light field image for each microlens through which the light beam has passed.

The image generation process 4 generates a reconstructed image in the following procedure.
(1) The position where the light beam from the pixel to be reconstructed passes through the main point of the main lens and reaches the microlens array is specified.
(2) A main lens blur area having a radius based on the refocus distance corresponding to the reconstruction plane with the specified position as the center (area on the microlens array where the light beam from the pixel to be reconstructed reaches) is calculated.
(3) A microlens in which some or all of the microlenses included in the microlens array are included in the main lens blur region is specified.
(4) Select one of the specified microlenses.
(5) The area where the selected microlens and the main lens blur region overlap is calculated and divided by the area of the microlens to obtain a weighting coefficient.
(6) A pixel value is acquired on a sub-image at a position where a light beam from a pixel to be reconstructed is imaged by the selected microlens.
(7) The corrected pixel value is obtained by multiplying the acquired pixel value by the above weighting coefficient.
(8) The correction pixel value is calculated over the entire microlens part or all of which is included in the main lens blur region, and the sum is obtained.
(9) Divide the sum of the corrected pixel values by the sum of the overlapping areas to obtain the pixel value of the image to be reconstructed.

  When the generation of the main image is completed in step S508, the digital camera 1 executes step S509 in the same manner as step S110 of the image output process in FIG. 6, and ends the image output process 2.

  As described above, the digital camera 1 of the present embodiment can display a live view image with the same image quality as the confirmation image of the first embodiment. In addition, the confirmation image can be displayed with the same image quality as the main image of the first embodiment.

As a modification of the present embodiment, a configuration in which a live view image is generated using the image generation process 1 instead of the image generation process 2 is conceivable.
According to this configuration, a live view image that is desired to have the highest generation speed can be generated with the same amount of calculation as in the first embodiment, while a confirmation image and a main image have higher image quality than those in the first embodiment. It can be.
Note that the fourth reconstructed image generation unit of the present embodiment may generate a reconstructed image using any known process for generating a reconstructed image from the LFI (for example, the process exemplified in Patent Document 1). .

(Third embodiment)
Embodiment 3 of the present invention will be described.
The digital camera 1 of this embodiment has a physical configuration shown in FIG. In the digital camera 1 of the present embodiment, the CPU 21 is faster than the corresponding part of the second embodiment. 2 are the same as those of the digital camera 1 according to the second embodiment.

  In the first and second embodiments, an image that does not include blur is displayed as a live view image. The third embodiment is characterized in that an image to which blur is added from the live view image stage is displayed using the fact that the CPU 21 is faster.

The digital camera 1 of this embodiment has a functional configuration shown in FIG.
In the present embodiment, the live view image generation unit 620 is the fifth reconstructed image generation unit 740 shown in FIG. Other parts in FIG. 5 are the same as the corresponding configurations of the digital camera 1 of the second embodiment.

  Compared with the third reconstructed image generating unit 730 shown in FIG. 13, the fifth reconstructed image generating unit 740 has (1) no part corresponding to the pixel shift coefficient calculating unit 732 and the filter processing unit 733, (2 ) The difference is that the image deviation coefficient calculation unit 734a acquires the pixel deviation coefficient of the sub-image of the predetermined part, and determines the arrangement area size and arrangement interval using the average value as the deviation coefficient of all the sub-images. Other parts are the same as the parts having the same names in the third reconstructed image generation unit 730 of the first embodiment.

  Processing executed by the digital camera 1 of the present embodiment will be described with reference to a flowchart.

  When the digital camera 1 is turned on and the input unit 50 receives an operation to prepare for shooting, the digital camera 1 starts an image output process 3 shown in FIG.

  In the image output process 3, steps S601 to S602 are executed in the same manner as steps S101 to S102 of the image output process of FIG. 6 according to the first embodiment.

  In this embodiment, the image generation process 5 (FIG. 19) is executed in step S603 to generate a live view image.

The image generation process 5 will be described with reference to FIG.
In the image generation process 5, first, the image deviation coefficient calculation unit 734a acquires image deviation coefficients for some of the sub-images determined by setting among the sub-images included in the LFI (step S701). Here, “part” is set so that the image shift coefficient obtained from “partial sub-image” reflects the degree of deviation of the entire LFI. Specifically, a predetermined portion at the center, four sub-images at the corner, and the like are conceivable.
The method for acquiring the image shift coefficient for each sub-image is the same as that in step S302 of the image generation process 2 (FIG. 11) according to the first embodiment.

  Next, based on the image shift coefficients of some of the sub-images calculated in step S701, the image shift coefficients of the entire LFI are set. Here, the calculated plurality of image deviation coefficients are averaged and set as the deviation coefficient of the entire sub-image included in the LFI (step S702).

  Then, based on the deviation coefficient of the entire sub-image, the arrangement interval and the size of the partial area applied to all the sub-images are calculated according to the conditions stored in advance in the ROM 22 or the storage unit 34 (step S703). As the deviation coefficient increases, the partial areas and the arrangement interval also increase. The size of the partial area is larger than the arrangement interval. Here, for example, when the deviation coefficient of the entire sub-image is 10, the arrangement interval is 10 pixels, and the size of the partial area is 20 pixels × 20 pixels.

  Next, using k4 as a counter variable, the sub-image extraction unit 731 extracts the k-th sub-image of the LFI as a target sub-image (step S704).

  Then, the partial region defining unit 735 defines the partial region having the size obtained in step S703 at the center of the target sub-image, similarly to step S407 in FIG. 14 (step S705).

  Further, the image placement unit 736 places the images of the partial areas at the placement interval obtained in step S703, similarly to step S408 in FIG. 14 (step S706). At this time, the image is arranged upside down and horizontally.

Then, in step S707, it is determined whether or not the processing for arranging images for all sub-images has been executed.
If an unprocessed sub-image remains (step S707; NO), k4 is incremented (step S708), and the processing is repeated from step S704.
On the other hand, when the processing has been completed for all the sub-images (step S707; YES), the generated image is output as a live view image, and the image generation processing 5 ends.
Returning to FIG. 17, steps S604 to S609 are executed in the same manner as steps S504 to S509 of FIG. 16 in the second embodiment, and the process is terminated.

As described above, according to the digital camera 1 of the present embodiment, it is possible to display an image with blur added from the live view image generation stage. As a result, the user can easily predict the completed image.
In addition, since it is not necessary to calculate the shift coefficient for each pixel at the live view image generation stage, the amount of calculation required for adding blur is small. Therefore, an image with blur can be generated at high speed.

(Fourth embodiment)
Embodiment 4 of the present invention will be described.
The digital camera 1 of this embodiment has a physical configuration shown in FIG. The function of each part of the digital camera 1 is the same as that of the digital camera 1 of the first embodiment.

  In the first embodiment, in the image generation process 2 and the image generation process 3, image shift coefficients are calculated for all sub-images. In the third embodiment, in the image generation process 5, image shift coefficients of some sub-images (four sub-images at a predetermined portion in the center or corners, etc.) are calculated, and the addition average is calculated as the total image shift coefficient. Set as. On the other hand, in the present embodiment, when generating a live view image, image shift coefficients are calculated for some sub-images, and image shift coefficients of other sub-images (not subject to image shift coefficient calculation) are calculated from the calculation results. presume. Further, the present embodiment is characterized in that a sub-image to be calculated is selected cyclically.

The digital camera 1 of this embodiment has a functional configuration shown in FIG.
In the present embodiment, the live view image generation unit 620 is the sixth reconstructed image generation unit 750 (FIG. 20), and the confirmation image generation unit 630 is the third reconstructed image generation unit 730 shown in FIG. The main image generation unit 640 is the fourth reconstructed image generation unit described in the second embodiment. The other units in FIG. 5 are the same as the corresponding configurations of the digital camera 1 of the first embodiment.

  The sixth reconstructed image generation unit 750 in FIG. 20 includes an image deviation coefficient calculation unit 751, an image deviation coefficient estimation unit 752, a sub image extraction unit 721, a partial region definition unit 723, a layout image generation unit 724, An image arrangement unit 725 and an output unit 726 are included. The sixth reconstructed image generation unit 750 is different from the second reconstructed image generation unit 720 in FIG. 10A in that an image shift coefficient calculation unit 751 and an image shift coefficient estimation unit 752 instead of the shift value calculation unit 722. And including. Other configurations are the same as those of the second reconstructed image generation unit 720.

The image deviation coefficient calculation unit 751 cyclically selects a sub-image (calculated image) for which a deviation coefficient is to be calculated every time a live view image is generated. Then, an image shift coefficient is calculated for the selected calculated image. Specifically, the sub-images are classified into n groups (for example, n = 2). Then, when the live view images are sequentially generated, the groups are sequentially selected for each loop as the group of interest (the group including the calculated image). A sub-image belonging to the group of interest is a calculated image. The value of n is preset and stored in the storage unit 34.
The calculation method of the image deviation coefficient is the same as that of the deviation value calculation unit 722 of the first embodiment.

  The image deviation coefficient estimation unit 752 estimates the image deviation coefficient of a sub-image (estimated image) other than the calculated image, using the image deviation coefficient of the calculated image calculated by the image deviation coefficient calculation unit 751. A specific method of estimation will be described later.

  Processing performed by the digital camera 1 of the present embodiment will be described with reference to FIG. When the digital camera 1 of the present embodiment is turned on and receives an operation for the input unit 50 to prepare for shooting, the image output process 4 shown in FIG. 21 is started.

  In the image output process 4, steps S801 and S802 are executed in the same manner as in steps S101 and S102 of the image output process 1 (FIG. 6) executed in the first embodiment.

  Then, the image shift coefficient calculation unit 751 of the live view image generation unit 620 (sixth reconstructed image generation unit 750) updates the frame flag (step S803). The frame flag is a flag for selecting a group for calculating a calculated image. The frame flag is initially set to 1, and is incremented by 1 each time the loop (steps S801 to S807) for generating a live view image is performed once. When the set number of groups n is exceeded, the number returns to 1. As a result, for example, when n = 4, the frame flags are cyclically set to numerical values 1 to 4 such as 1, 2, 3, 4, 1, 2, 3, 4, 1,.

  When the frame flag is updated in step S803, the image deviation coefficient calculation unit 751 calculates an image deviation coefficient of the calculated image, and starts processing for estimating the image deviation coefficient of the estimated image (deviation coefficient estimation processing).

  The deviation coefficient estimation process executed in step S804 will be described with reference to FIG. In the shift coefficient estimation process, first, the image shift coefficient calculation unit 751 selects a calculated image corresponding to the current frame flag (step S901).

  A calculation image selection method selected here will be described with reference to FIGS. In FIGS. 23 and 24, each sub-image is represented by a square. For example, when the sub-images are divided into two groups (n = 2), one of the sub-image shown in black and the sub-image shown in white in FIG. 23 is selected as the calculated image. For example, a black sub-image is selected when the frame flag is 1, and a white sub-image is selected when the frame flag is 2.

  Alternatively, the LFI may be divided into regions including n sub-images, and the calculated image may be selected cyclically within the divided regions. For example, when n = 9, as shown in FIG. 24, the image is divided into regions (thick lines) including three vertical and three horizontal sub-images. Then, the sub-images in the area are numbered from 1 to 9. For example, the sub images a1 to a9 in FIG. 24 mean the first to ninth sub images of the first region (region a), respectively. Then, one sub-image having a number that matches the current frame flag is selected as a calculated image from each region (region a to region l in the example of FIG. 24).

  When the calculated image is selected in step S901, the image deviation coefficient calculating unit 751 next selects one target sub-image from the calculated image (step S902). Then, the image shift coefficient calculation unit 751 calculates an image shift coefficient for the target sub-image (step S903). The method for calculating the image shift coefficient is the same as that in the first embodiment.

  Then, the image deviation coefficient calculation unit 751 determines whether or not the amount of change in the image deviation coefficient of the target sub-image is greater than or equal to a threshold value (step S904). Here, the image deviation coefficient calculation unit 751 compares the difference between the image deviation coefficient estimated in the previous deviation coefficient estimation process and the image deviation coefficient calculated in the current process with a predetermined threshold for the target sub-image. To do. This threshold value is obtained in advance by experiments and stored in the storage unit 34. When the numerical value of n is relatively small and the time difference from the previously calculated time is small (for example, when n = 2), the difference with respect to the last calculated image shift coefficient is compared with a predetermined threshold value. May be.

  If the change amount is less than the threshold value as a result of the comparison (step S904; NO), the image deviation coefficient calculation unit 751 turns off the change flag of the sub image of interest (step S905). On the other hand, when the amount of change is equal to or greater than the threshold (step S904; YES), the image deviation coefficient calculation unit 751 turns on the change flag of the target sub-image (step S906). The change flag is a binary variable associated with each sub-image.

  When the setting of the change flag is completed, the image deviation coefficient calculation unit 751 next determines whether image deviation coefficients have been calculated for all the calculated images selected in step S901 (step S907). If there is an unprocessed calculated image (step S907; NO), the process is repeated from step S902 on the next calculated image. On the other hand, if all the calculated images have been processed (step S907; YES), the process proceeds to step S908.

  In step S908, one sub-image that has not been subjected to estimation processing among images other than the calculated image (estimated image) is selected as a target image (step S908).

  Then, the change flag of the peripheral calculated image (peripheral image) is checked to determine whether or not the image shift coefficient has changed (step S909). For example, in the case of n = 2, it is determined that there is a change in the peripheral image when there is a predetermined number (for example, two or more) of images whose change flag is ON among calculated images adjacent to the target image in the upper, lower, left, and right directions ( Step S909; YES). At this time, if all the images whose change flag is ON have changed in the same direction (for example, the direction in which the deviation coefficient increases), it is determined that the change is reliable and it is determined that there is a change. If it has changed, it may be determined that it is not reliable and it may be determined that there is no change. When n is larger than 2, m (for example, m = 4) calculated images are selected in order from the target image, and the change flag is similarly referred to.

  When it is determined that there is a change (step S909; YES), the image deviation coefficient of the closest calculated image is estimated as the image deviation coefficient of the target image (step S910). At this time, a numerical value obtained by adding and averaging the image shift coefficients of the calculated image referred to in step S909 may be used as the estimated value.

  On the other hand, if it is determined that there is no change in the peripheral image (step S909; NO), step S910 is skipped. As a result, the image shift coefficient set in the previous loop is used in subsequent processing without being updated. Then, it is determined whether or not the above processing has been completed for all estimated images (step S911). If there is an unprocessed estimated image (step S911; NO), the processing is repeated from step S908 on the next estimated image. On the other hand, if all estimated images have been processed (step S911; YES), the deviation coefficient estimation process ends.

Here, the process of estimating the image deviation coefficient of the surrounding calculated image as the image deviation coefficient of the estimated image when it is determined that the surrounding image has changed has been described. As another estimation method, a method of distributing the amount of change in the surrounding calculated image according to the distance from the calculated image may be applied. A specific example of distribution is shown in FIG. FIG. 25 shows an example in which the latest calculated value is different from the previous calculated value (or the previous estimated value) (+3 for the left calculated image and +1 for the right calculated image) for the calculated image indicated by the bold line. . The difference between the image misalignment coefficients is multiplied by a weight that decreases with distance from the calculated image and added to the image misalignment coefficient set in the previous loop processing of the estimated image (dotted square), and the addition result is used as the estimation result.
In FIG. 25, the arrow indicates the direction of distribution (addition), and the number of numerical values on the arrow is allocated. Here, 2/3 of the difference between the calculated images is assigned to estimated images adjacent to the calculated image vertically and horizontally and diagonally, and 3/1 is assigned to the estimated image adjacent thereto.

  As another estimation method, the image deviation coefficient of the estimated image may be set by three-dimensional interpolation processing using the image deviation coefficient of the calculated image.

  Returning to FIG. 21, when image shift coefficients are set for all sub-images in step S804, the image generation process 2 shown in FIG. 11 is started using the image shift coefficients set by the sub-image extraction unit 721 (step S805). In the image generation process 2, each step is executed in the same manner as in the first embodiment except that the image coefficient set in the image shift coefficient estimation process in step S302 is acquired.

  When the live view image is generated by the image generation process 2, the display unit 70 displays the generated image (step S806). And it is discriminate | determined whether the input part 50 detected imaging operation (step S807).

  As a result of the determination, if an imaging operation is not detected (step S807; NO), the process returns to step S801 and continues to generate a live view image. On the other hand, when an imaging operation is detected (step S807; YES), the confirmation image generation unit 630 (third reconstructed image generation unit 730) generates a confirmation image by the image generation processing 3 of FIG. 14 (step S808). . In this case, in the image generation process 3, the image deviation coefficient may be calculated for all the sub-images, or the image deviation coefficient set in the last deviation coefficient estimation process (step S804) may be used. Or similarly to the process which produces | generates a live view image, you may divide a sub image into a calculation image and an estimated image, and may perform an estimation process. At this time, it is possible to increase the estimation accuracy by increasing the ratio of the calculated image in the entire sub image as compared with the live view image.

  Subsequently, the generated confirmation image is displayed (step S809), and the main image is generated by the image generation process 4 as in the third embodiment (step S810). Then, the generated main image is recorded (step S811), and the image output process 4 ends.

As described above, according to the digital camera 1 of the present embodiment, it is necessary to calculate the image shift coefficient for all images in the process that requires the image shift coefficient of each sub-image, such as the image generation process 2. There is no. Since the calculation of the image shift coefficient having a large necessary calculation amount can be limited, the calculation amount necessary for generating a high-quality image is reduced.
As a result, the power required for the calculation process is also reduced. Therefore, the driving time of the digital camera 1 and the number of shots can be increased.

  In addition, when generating a live view image, a calculated image is selected cyclically. When the calculated image is fixed, the defect always appears on the image when the subject of the calculated image is different from other subjects or when noise is generated in the calculated image. In the present embodiment, since the calculated image is changed for each generation process, even if some of the sub-images are defective, the influence can be reduced. In addition, since the image deviation count is calculated for all the sub-images in the long term, the live view image is an image that takes into account the tendency of the entire subject.

  In this embodiment, the live view image generation unit 620 generates a live view image by the image generation process 2 using the image shift coefficient calculated or estimated by the shift coefficient estimation process of FIG. However, the present invention is not limited to this, and the shift coefficient estimation process can be applied to any process (image generation process 3, image generation process 5, etc.) that generates a reconstructed image using an image shift coefficient. For example, the live view image generation unit 620 may execute image generation by the deviation coefficient estimation processing and the image generation processing 3. Further, not only the live view image generation unit 620 but also the confirmation image generation unit 630 or the main image generation unit 640 adopts a configuration in which a reconstructed image is generated using an image shift coefficient. A coefficient estimation process can be used.

  As mentioned above, although Embodiment 1 thru | or 4 of this invention was demonstrated, embodiment of this invention is not restricted to this, A various deformation | transformation is possible.

  For example, in the above embodiment, the image is described as a grayscale image, but the image to be processed according to the present invention is not limited to a grayscale image. For example, the image may be an RGB image in which three pixel values of R (red), G (green), and B (blue) are defined for each pixel. In this case, the pixel value is similarly processed as an RGB vector value. Alternatively, the above processing may be performed using each of R, G, and B values as independent grayscale images. According to this configuration, a reconstructed image that is a color image can be generated from a light field image that is a color image.

  The refocus distance (corresponding to the distance to the reconstruction surface) may be set manually or by measuring the distance to the subject to be focused. Furthermore, after the reconstructed image is displayed, the subject to be focused on may be selected again using a touch panel or the like, and the refocus distance may be reset.

Further, although the case where the main lens imaging surface MA exists on the main lens side from the micro lens array has been described, the configuration of the optical system of the present invention is not limited to this.
For example, the other designs are the same, and the main lens imaging surface MA may be behind the microlens array. In this case, when the arrangement image is arranged in the generated image, the arrangement image is arranged as it is without being inverted vertically and horizontally.
In addition, there may be a case where the imaging surface of the microlens is on the microlens side of the imaging element. In this case, on the basis of the case other than that, the arrangement image is arranged in the generated image by being further inverted vertically and horizontally.

Further, the live view image generation process, the confirmation image generation process, and the main image generation process are not limited to the above combinations. In the present invention, the combination of the above processes is arbitrarily selected under the condition that the necessary calculation amount of the live view image generation process is equal to or less than the confirmation image generation process, and the main image generation process requires a larger calculation amount than the confirmation image generation process. I can do it.
For example, the live view image generation process and the confirmation image generation process may be the image generation process 1, and the main image generation process may be the image generation process 3 or 4. Further, the live view image generation process and the confirmation image generation process are the image generation process 2, and the main image generation process can be combined with the image generation process 3 or 4.

Further, the same type of image generation processing may be adopted in the live view image generation processing and the confirmation image generation processing, and the calculation amount may be adjusted in the processing. For example, when the live view image generation process and the confirmation image generation process are the image generation process 2, the live view image generation process calculates the image deviation coefficient in only one direction, and the confirmation image generation process sets the image deviation coefficient to be vertical. By calculating in the two horizontal directions, it is possible to improve the accuracy of the image shift coefficient for the confirmation image.
Alternatively, it is also possible to improve the accuracy of the image shift coefficient by taking a larger area for calculating the image shift coefficient than the live view generation process for the confirmation image generation process.

  With such a configuration, it is possible to display a live view image to be displayed at the highest speed in accordance with the speed of the CPU, using a process with the least amount of calculation, at a high speed. In addition, the confirmation image can be displayed sufficiently quickly with at least the same image quality as the live view image and with higher image quality depending on the setting. In addition, the main image can be a higher quality image. As described above, a digital camera that is highly convenient for the user can be provided by properly using the image generation processing according to the required display speed.

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  A central part that performs processing for image generation including the CPU 21, the ROM 22, the RAM 23, and the like can be realized by using a normal computer system without depending on a dedicated system. For example, a computer program for executing the above operation is stored and distributed in a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. You may comprise the part which performs the said image generation process. Alternatively, the computer program may be stored in a storage device included in a server device on a communication network such as the Internet, and a part that executes the image generation processing by downloading or the like using a normal computer system may be configured.

  When the function of the part that executes the image generation processing is realized by sharing of an OS (operating system) and an application program, or by cooperation of the OS and the application program, only the application program part is recorded on a recording medium or a storage device. May be stored.

  It is also possible to distribute the computer program via a communication network. For example, the computer program may be posted on a bulletin board (BBS: Bulletin Board System) on a communication network, and the computer program may be distributed via the network. The computer program may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to the specific embodiment which concerns, This invention includes the invention described in the claim, and its equivalent range It is. Hereinafter, the invention described in the scope of claims of the present application will be appended.

(Appendix 1)
An acquisition unit for acquiring a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned;
An extraction unit that extracts a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
A generation unit configured to generate the reconstructed image by arranging the partial images in the arrangement order of the sub-images corresponding to the partial images;
An image processing apparatus comprising:

(Appendix 2)
The generation unit generates the reconstructed image by inverting and arranging the partial images as point targets,
The image processing apparatus according to appendix 1, wherein:

(Appendix 3)
For each of the plurality of sub-images, for a predetermined portion of the sub-image, the position of the predetermined portion in the sub-image, and the position where the subject of the predetermined portion appears in another sub-image within a predetermined range from the sub-image, A displacement acquisition unit for acquiring an image displacement coefficient indicating the displacement of the position of
The extraction unit extracts a partial image from a larger range of the sub-image as the image deviation coefficient acquired by the deviation acquisition unit is larger.
The image processing apparatus according to appendix 1 or 2, characterized in that:

(Appendix 4)
The shift acquisition unit calculates the image shift coefficient for a calculation target sub-image selected from the plurality of sub-images, and calculates the image shift coefficient for sub-images other than the calculation target based on the calculated image shift coefficient. presume,
The image processing apparatus according to appendix 3, characterized in that:

(Appendix 5)
The acquisition unit sequentially acquires the multi-viewpoint images,
The shift acquisition unit cyclically selects the calculation target sub-image for each of the sequentially acquired multi-viewpoint images, sequentially acquires image shift coefficients of the plurality of sub-images,
The generation unit sequentially generates the reconstructed image from the multi-view images sequentially acquired by the acquisition unit, using the image shift coefficient sequentially acquired by the shift acquisition unit.
The image processing apparatus according to appendix 4, wherein

(Appendix 6)
The generation unit divides the reconstructed image into a plurality of arrangement regions corresponding to any of the sub-images, and subpixels from which the partial images are extracted from pixels included in the partial images extracted by the extraction unit Extract the number of pixels that match the arrangement area corresponding to, and arrange the extracted pixels in the corresponding arrangement area,
The image processing apparatus according to any one of appendices 1 to 5, wherein:

(Appendix 7)
The generation unit divides the reconstructed image into a plurality of arrangement regions corresponding to any of the sub-images, and converts the partial images extracted by the extraction unit to each of the arrangement regions corresponding to the extraction-source sub-images. To reduce or enlarge to fit the size,
The image processing apparatus according to any one of appendices 1 to 5, wherein:

(Appendix 8)
An image processing unit, the image processing device according to any one of supplementary notes 1 to 7 that acquires a multi-viewpoint image captured by the image capturing unit, a display unit that displays a reconstructed image generated by the image processing device, An imaging apparatus comprising:

(Appendix 9)
An imaging unit that captures a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned;
A display for displaying an image;
A first generation unit that generates a first reconstructed image to be displayed in a shooting preparation state by a first image generation method using a plurality of sub-images of the multi-viewpoint image;
A second generation unit that generates a second reconstructed image for reproduction display after shooting by a second image generation method using a plurality of sub-images of the multi-viewpoint image;
A control unit that causes the display unit to display the first reconstructed image in a shooting preparation state and to display the second reconstructed image in a reproduction display after shooting;
With
The second image generation method has a larger calculation amount than the first image generation method in order to improve image quality.
An imaging apparatus characterized by that.

(Appendix 10)
An extraction unit that extracts a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
An arrangement unit that arranges the partial images in the arrangement order of the sub-images corresponding to the partial images;
Further comprising
The first image generation method is a method of generating the first reconstructed image using the extraction unit and the arrangement unit,
The second image generation method is a method of generating the second reconstructed image using the extraction unit and the arrangement unit,
The second image generation method is a method in which processing for improving image quality is added based on the first image generation method, or alternative processing with a larger calculation amount is executed.
The imaging apparatus according to appendix 9, characterized in that:

(Appendix 11)
For each of the plurality of sub-images, for a predetermined portion of the sub-image, the position of the predetermined portion in the sub-image, and the position where the subject of the predetermined portion appears in another sub-image within a predetermined range from the sub-image, An image shift acquisition unit for acquiring an image shift coefficient indicating the shift of the position of
The imaging apparatus according to appendix 10, wherein the imaging apparatus is characterized.

(Appendix 12)
The image shift acquisition unit calculates the image shift coefficient for a calculation target sub-image selected from the plurality of sub-images, and the image shift coefficient for sub-images other than the calculation target based on the calculated image shift coefficient. Estimate
The imaging apparatus according to appendix 11, wherein:

(Appendix 13)
The imaging unit sequentially captures the multi-viewpoint images,
The image shift acquisition unit cyclically selects the sub-images to be calculated for each of the multi-viewpoint images taken sequentially, sequentially acquires the image shift coefficients of the plurality of sub-images,
The first generation unit sequentially generates the first reconstructed image from the multi-viewpoint images sequentially captured by the imaging unit using the image shift coefficient sequentially acquired by the image shift acquisition unit.
The imaging apparatus according to attachment 12, wherein the imaging apparatus is characterized in that

(Appendix 14)
The first generation unit, as part of the first image generation method, causes the extraction unit to extract partial images from a predetermined range of the same size from each of the sub-images,
The second generation unit, as part of the second image generation method, starts from a large range from a sub-image having a large image shift coefficient acquired by the image shift acquisition unit to the extraction unit, and is small from a small sub-image. Extract each partial image from the range,
The imaging device according to appendix 11 or 12, characterized in that.

(Appendix 15)
For each pixel of the sub-image, a pixel indicating a positional deviation between a position occupied by the pixel in the sub-image and a position where a subject appearing in the pixel appears in another sub-image within a predetermined range from the sub-image. A pixel shift acquisition unit that acquires a shift coefficient for each pixel;
The first generation unit, as part of the first image generation method, causes the extraction unit to extract partial images from a predetermined range of the same size from each of the sub-images,
The second generation unit performs filtering on the pixels of the sub-image based on the size of the pixel shift coefficient acquired by the pixel shift acquisition unit as part of the second image generation method, The extraction unit causes the partial image to be extracted from a large range from a sub-image having a large image shift coefficient acquired by the image shift acquisition unit and from a small range from a small sub-image, respectively.
The imaging device according to appendix 11 or 12, characterized in that.

(Appendix 16)
The first generation unit, as part of the first image generation method, starts from a large range from a sub-image with a large image shift coefficient acquired by the image shift acquisition unit to the extraction unit, and is small from a small sub-image. Each partial image is extracted from the range,
As a part of the second image generation method, the second generation unit causes the image shift acquisition unit to acquire the image shift coefficient for a part larger than the first image generation method, and causes the extraction unit to Partial images are extracted from a large range from a sub-image with a large image shift coefficient, and from a small range from a small sub-image, respectively.
The imaging apparatus according to any one of appendices 11 to 13, wherein the imaging apparatus is characterized in that

(Appendix 17)
For each pixel of the sub-image, a pixel indicating a positional deviation between a position occupied by the pixel in the sub-image and a position where a subject appearing in the pixel appears in another sub-image within a predetermined range from the sub-image. A pixel shift acquisition unit that acquires a shift coefficient for each pixel;
The first generation unit, as part of the first image generation method, starts from a large range from a sub-image with a large image shift coefficient acquired by the image shift acquisition unit to the extraction unit, and is small from a small sub-image. Each partial image is extracted from the range,
The second generation unit performs filtering on the pixels of the sub-image based on the size of the pixel shift coefficient acquired by the pixel shift acquisition unit as part of the second image generation method, The extraction unit causes the partial image to be extracted from a large range from a sub-image having a large image shift coefficient acquired by the image shift acquisition unit and from a small range from a small sub-image, respectively.
The imaging apparatus according to any one of appendices 11 to 13, wherein the imaging apparatus is characterized in that

(Appendix 18)
A third generation unit for generating a third reconstructed image by the third image generation method;
A storage unit for storing a third reconstructed image generated by the third generation unit;
Further comprising
The third image generation method has a larger calculation amount than the second image generation method in order to improve image quality.
The imaging apparatus according to any one of appendices 9 to 17, characterized in that:

(Appendix 19)
An acquisition step of acquiring a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned;
An extraction step of extracting a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
Generating the reconstructed image by arranging the partial images in the order of arrangement of the sub-images corresponding to the partial images;
An image generation method including:

(Appendix 20)
On the computer,
An acquisition function for acquiring a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned,
An extraction function for extracting a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
A generation function for generating the reconstructed image by arranging the partial images in the order of arrangement of the sub-images corresponding to the partial images;
A program to realize

  The present invention can be applied to an imaging apparatus that can capture a multi-viewpoint image.

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 10 ... Shutter key, 11 ... Power button, 12 ... Imaging lens, 13 ... Liquid crystal monitor screen, 14 ... Operation part, 20 ... Internal bus, 21 ... CPU, 22 ... ROM, 23 ... RAM, 30 ... Input / output interface, 31 ... imaging unit, 311 ... main lens, 312 ... micro lens array, 313 ... imaging device, 312-1 to 312-NxM ... micro lens, 32 ... input unit, 33 ... output unit, 34 ... Storage unit 35 ... Display unit 36 ... Communication unit 37 ... Media drive 38 ... Removable media 40 ... Image shooting unit 50 ... Input unit 60 ... Image generation processing unit 610 ... LFI generation unit 620 ... Live View image generation unit, 630 ... confirmation image generation unit, 640 ... main image generation unit, 70 ... display unit, 710 ... first reconstructed image generation unit, 711 ... sub image Output unit, 712 ... Partial region definition unit, 713 ... Pixel extraction unit, 714 ... Pixel arrangement unit, 715 ... Output unit, 720 ... Second reconstructed image generation unit, 721 ... Sub image extraction unit, 722 ... Deviation value calculation unit , 723 ... Partial region definition section, 724 ... Arrangement image generation section, 725 ... Image arrangement section, 726 ... Output section, 730 ... Third reconstructed image generation section, 731 ... Sub-image extraction section, 732 ... Pixel shift coefficient calculation section 733 ... Filter processing unit, 734 ... Image deviation coefficient calculation unit, 734a ... Image deviation coefficient calculation unit, 735 ... Partial region definition unit, 736 ... Image arrangement unit, 737 ... Output unit, 740 ... Fifth reconstructed image generation unit 750 ... Sixth reconstructed image generating unit 751 Image misalignment coefficient calculating unit 752 Image misalignment coefficient estimating unit 80 80 Storage unit OB Subject MA Main lens imaging plane S _{11 to} S _MN Sub picture , LFI ... light field image, RI ... reconstructed image

In order to achieve the above object, an imaging apparatus according to the present invention provides:
An imaging unit that captures a multi-view image in which a plurality of sub-images obtained from a plurality of viewpoints are aligned;
A display for displaying an image;
A first generation unit that generates a first reconstructed image to be displayed in a shooting preparation state by a first image generation process using a plurality of sub-images of the multi-viewpoint image;
Second image generation that uses a plurality of sub-images of the multi-viewpoint image and has a larger calculation amount than the first image generation processing for improving image quality, using the second reconstructed image for reproduction display after shooting. A second generation unit generated by processing;
A control unit that causes the display unit to display the first reconstructed image in the shooting preparation state and to display the second reconstructed image in the reproduction display after the shooting;
It is characterized by providing .

In order to achieve the above object, an imaging apparatus according to the present invention provides:
An imaging unit that captures a multi-view image in which a plurality of sub-images obtained from a plurality of viewpoints are aligned;
A display for displaying an image;
A cutout unit that cuts out a partial image from a predetermined range of the sub-image included in the multi-viewpoint image;
An arrangement unit that arranges the partial images in the arrangement order of the sub-images corresponding to the partial images;
A first generation unit that generates a first reconstructed image to be displayed in a shooting preparation state by a first image generation process using the cutout unit and the arrangement unit ;
Second image generation with a larger amount of calculation than the first image generation processing for improving the image quality of the second reconstructed image for reproduction display after shooting using the cutout unit and the arrangement unit A second generation unit generated by processing;
A control unit that causes the display unit to display the first reconstructed image in the shooting preparation state and to display the second reconstructed image in the reproduction display after the shooting;
For each of the plurality of sub-images, for a predetermined part of the sub-image, a position of the predetermined part in the sub-image and a position in another sub-image in which the subject of the predetermined part is within a predetermined range from the sub-image An image misalignment acquisition unit that acquires an image misalignment coefficient indicating the misalignment of the position;
It is characterized by providing.

Claims (20)

An acquisition unit for acquiring a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned;
An extraction unit that extracts a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
A generation unit configured to generate the reconstructed image by arranging the partial images in the arrangement order of the sub-images corresponding to the partial images;
An image processing apparatus comprising: The generation unit generates the reconstructed image by inverting and arranging the partial images as point targets,
The image processing apparatus according to claim 1. For each of the plurality of sub-images, for a predetermined portion of the sub-image, the position of the predetermined portion in the sub-image, and the position where the subject of the predetermined portion appears in another sub-image within a predetermined range from the sub-image, A displacement acquisition unit for acquiring an image displacement coefficient indicating the displacement of the position of
The extraction unit extracts a partial image from a larger range of the sub-image as the image deviation coefficient acquired by the deviation acquisition unit is larger.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The shift acquisition unit calculates the image shift coefficient for a calculation target sub-image selected from the plurality of sub-images, and calculates the image shift coefficient for sub-images other than the calculation target based on the calculated image shift coefficient. presume,
The image processing apparatus according to claim 3. The acquisition unit sequentially acquires the multi-viewpoint images,
The shift acquisition unit cyclically selects the calculation target sub-image for each of the sequentially acquired multi-viewpoint images, sequentially acquires image shift coefficients of the plurality of sub-images,
The generation unit sequentially generates the reconstructed image from the multi-view images sequentially acquired by the acquisition unit, using the image shift coefficient sequentially acquired by the shift acquisition unit.
The image processing apparatus according to claim 4. The generation unit divides the reconstructed image into a plurality of arrangement regions corresponding to any of the sub-images, and subpixels from which the partial images are extracted from pixels included in the partial images extracted by the extraction unit Extract the number of pixels that match the arrangement area corresponding to, and arrange the extracted pixels in the corresponding arrangement area,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The generation unit divides the reconstructed image into a plurality of arrangement regions corresponding to any of the sub-images, and converts the partial images extracted by the extraction unit to each of the arrangement regions corresponding to the extraction-source sub-images. To reduce or enlarge to fit the size,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   An image processing unit, the image processing device according to any one of claims 1 to 7 that acquires a multi-viewpoint image captured by the image capturing unit, and a display unit that displays a reconstructed image generated by the image processing device. An imaging apparatus comprising: An imaging unit that captures a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned;
A display for displaying an image;
A first generation unit that generates a first reconstructed image to be displayed in a shooting preparation state by a first image generation method using a plurality of sub-images of the multi-viewpoint image;
A second generation unit that generates a second reconstructed image for reproduction display after shooting by a second image generation method using a plurality of sub-images of the multi-viewpoint image;
A control unit that causes the display unit to display the first reconstructed image in a shooting preparation state and to display the second reconstructed image in a reproduction display after shooting;
With
The second image generation method has a larger calculation amount than the first image generation method in order to improve image quality.
An imaging apparatus characterized by that. An extraction unit that extracts a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
An arrangement unit that arranges the partial images in the arrangement order of the sub-images corresponding to the partial images;
Further comprising
The first image generation method is a method of generating the first reconstructed image using the extraction unit and the arrangement unit,
The second image generation method is a method of generating the second reconstructed image using the extraction unit and the arrangement unit,
The second image generation method is a method in which processing for improving image quality is added based on the first image generation method, or alternative processing with a larger calculation amount is executed.
The imaging apparatus according to claim 9. For each of the plurality of sub-images, for a predetermined portion of the sub-image, the position of the predetermined portion in the sub-image, and the position where the subject of the predetermined portion appears in another sub-image within a predetermined range from the sub-image, An image shift acquisition unit for acquiring an image shift coefficient indicating the shift of the position of
The imaging apparatus according to claim 10. The image shift acquisition unit calculates the image shift coefficient for a calculation target sub-image selected from the plurality of sub-images, and the image shift coefficient for sub-images other than the calculation target based on the calculated image shift coefficient. Estimate
The imaging apparatus according to claim 11. The imaging unit sequentially captures the multi-viewpoint images,
The image shift acquisition unit cyclically selects the sub-images to be calculated for each of the multi-viewpoint images taken sequentially, sequentially acquires the image shift coefficients of the plurality of sub-images,
The first generation unit sequentially generates the first reconstructed image from the multi-viewpoint images sequentially captured by the imaging unit using the image shift coefficient sequentially acquired by the image shift acquisition unit.
The imaging apparatus according to claim 12. The first generation unit, as part of the first image generation method, causes the extraction unit to extract partial images from a predetermined range of the same size from each of the sub-images,
The second generation unit, as part of the second image generation method, starts from a large range from a sub-image having a large image shift coefficient acquired by the image shift acquisition unit to the extraction unit, and is small from a small sub-image. Extract each partial image from the range,
The imaging apparatus according to claim 11 or 12, characterized in that: For each pixel of the sub-image, a pixel indicating a positional deviation between a position occupied by the pixel in the sub-image and a position where a subject appearing in the pixel appears in another sub-image within a predetermined range from the sub-image. A pixel shift acquisition unit that acquires a shift coefficient for each pixel;
The first generation unit, as part of the first image generation method, causes the extraction unit to extract partial images from a predetermined range of the same size from each of the sub-images,
The second generation unit performs filtering on the pixels of the sub-image based on the size of the pixel shift coefficient acquired by the pixel shift acquisition unit as part of the second image generation method, The extraction unit causes the partial image to be extracted from a large range from a sub-image having a large image shift coefficient acquired by the image shift acquisition unit and from a small range from a small sub-image, respectively.
The imaging apparatus according to claim 11 or 12, characterized in that: The first generation unit, as part of the first image generation method, starts from a large range from a sub-image with a large image shift coefficient acquired by the image shift acquisition unit to the extraction unit, and is small from a small sub-image. Each partial image is extracted from the range,
As a part of the second image generation method, the second generation unit causes the image shift acquisition unit to acquire the image shift coefficient for a part larger than the first image generation method, and causes the extraction unit to Partial images are extracted from a large range from a sub-image with a large image shift coefficient, and from a small range from a small sub-image, respectively.
The image pickup apparatus according to claim 11, wherein the image pickup apparatus is an image pickup apparatus. For each pixel of the sub-image, a pixel indicating a positional deviation between a position occupied by the pixel in the sub-image and a position where a subject appearing in the pixel appears in another sub-image within a predetermined range from the sub-image. A pixel shift acquisition unit that acquires a shift coefficient for each pixel;
The first generation unit, as part of the first image generation method, starts from a large range from a sub-image with a large image shift coefficient acquired by the image shift acquisition unit to the extraction unit, and is small from a small sub-image. Each partial image is extracted from the range,
The second generation unit performs filtering on the pixels of the sub-image based on the size of the pixel shift coefficient acquired by the pixel shift acquisition unit as part of the second image generation method, The extraction unit causes the partial image to be extracted from a large range from a sub-image having a large image shift coefficient acquired by the image shift acquisition unit and from a small range from a small sub-image, respectively.
The image pickup apparatus according to claim 11, wherein the image pickup apparatus is an image pickup apparatus. A third generation unit for generating a third reconstructed image by the third image generation method;
A storage unit for storing a third reconstructed image generated by the third generation unit;
Further comprising
The third image generation method has a larger calculation amount than the second image generation method in order to improve image quality.
The image pickup apparatus according to claim 9, wherein the image pickup apparatus is an image pickup apparatus. An acquisition step of acquiring a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned;
An extraction step of extracting a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
Generating the reconstructed image by arranging the partial images in the order of arrangement of the sub-images corresponding to the partial images;
An image generation method including: On the computer,
An acquisition function for acquiring a multi-viewpoint image in which a plurality of sub-images obtained by viewing a subject from a plurality of viewpoints are aligned,
An extraction function for extracting a partial image from a predetermined range of sub-images included in the multi-viewpoint image;
A generation function for generating the reconstructed image by arranging the partial images in the order of arrangement of the sub-images corresponding to the partial images;
A program to realize