JP2015046687A - Image processing apparatus, image processing method, program, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method which, when a user desires to reproduce a wide-angle image focused on a subject selected after photographing, reduces an amount of data for reconstitution.SOLUTION: An image processing apparatus comprises: acquisition means for acquiring plural pieces of re-focusable image data of different field angles; generation means for generating, from each of the pieces of image data, a reconstituted image on a refocus plane based on subject information obtained from the plural pieces of image data; and synthesis means for synthesizing, for each refocus plane, the reconstituted image generated from the pieces of image data of different field angles.

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus having means for acquiring information on the incident direction of light using a microlens array and generating a reconstructed image.

  2. Description of the Related Art In recent years, an imaging device called a light field camera that can acquire not only light intensity distribution but also information on the incident direction of light in an imaging device such as an electronic camera has been proposed. For example, according to Non-Patent Document 1, a microlens array (hereinafter referred to as MLA) is arranged between a photographing lens and an image sensor, and one microlens (hereinafter ML) is provided for a plurality of pixels of the image sensor. ). With this configuration, light that has passed through the ML is acquired by a plurality of pixels for each incident direction.

  A normal captured image can be generated for the pixel signal (light field) acquired in this way. In addition, by applying a method called “Light Field Photography”, an image focused on an arbitrary image plane (refocus plane) can be reconstructed after shooting.

Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02 Todor Georgiev, et al., “Superresolution with Plenoptic 2.0 Camera”, 2009 Optical Society of America

JP 2011-199565 A

  The range of the angle of view of the light field camera described in Non-Patent Document 1 depends on the performance of hardware included in the apparatus main body, such as the focal length of the photographing lens and the size of the image sensor. Patent Document 1 proposes a technique for continuously shooting while moving a shooting range of an imaging device and synthesizing a plurality of obtained images when acquiring a wide-angle image that exceeds the performance of hardware.

  However, when it is desired to display a wide-angle image in which the subject selected after shooting by the technique proposed in Patent Document 1 is focused, all of the plurality of captured images shot by the light field camera are held, and the amount of data is enormous. turn into. Therefore, an object of the present invention is to provide an image processing apparatus capable of displaying a wide-angle image focused on a subject selected after shooting while reducing the amount of data for reconstruction.

  According to the present invention, an image processing apparatus obtains reconstructed images on an image plane based on subject information obtained from a plurality of image data, an acquisition unit that acquires image data having a plurality of different angles of view that can be refocused, A generating unit configured to generate the image data; and a combining unit configured to combine the reconstructed image generated from the image data having different angles of view for each image plane.

  According to the present invention, it is possible to provide an image processing apparatus capable of generating a wide-angle image focused on a subject selected after shooting while reducing the amount of data for reconstruction.

1 is a block diagram of an imaging apparatus to which an image processing apparatus according to an embodiment of the present invention is applied. The figure for demonstrating the correspondence of the image pick-up element and MLA in the imaging device of FIG. The figure for demonstrating the arrangement configuration of the imaging lens, MLA, and image sensor in the imaging device of FIG. The figure for demonstrating the correspondence of the pupil region of a taking lens and the light reception pixel in the imaging device of FIG. The figure for demonstrating the passage area | region of the refocus image generation light ray The figure which shows the flowchart of the image composition recording operation | movement in the imaging device of FIG. The figure for demonstrating the imaging data processed by the image composition recording operation | movement of FIG. 6, and its imaging | photography condition The figure for demonstrating correction | amendment of the distance list | wrist in the image composition recording operation | movement of FIG. The figure which shows the example of a production | generation of the reconstructed image in the image composition recording operation | movement of FIG. The figure which shows the example of a synthesis | combination of the wide angle image in the image composition recording operation | movement of FIG. The figure which shows the structure of the file recorded in the image composition recording operation | movement of FIG. The figure which shows the flowchart of the reproduction | regeneration operation | movement of the file recorded by the image composition recording operation | movement of FIG. The figure which shows the other example of a structure of the imaging lens in the imaging device of FIG. 1, MLA, and an image pick-up element. 1 is a block diagram illustrating another configuration example of an imaging apparatus to which an image processing apparatus according to an embodiment of the present invention can be applied.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  In this embodiment, by combining a plurality of wide-angle images from a plurality of reconfigurable image data obtained by an imaging device capable of acquiring light field data, a wide-angle image focused on a selected subject can be displayed. An embodiment of an image processing apparatus that can be made will be described. However, the image processing operation according to the present embodiment may be performed during continuous shooting by the imaging device, or may be performed when image data is played back by another playback device. That is, the image processing according to the present invention is not necessarily realized in the illustrated imaging apparatus, and the image processing operation according to the present invention is performed by acquiring the image data acquired by the illustrated imaging apparatus as input image data. You may implement | achieve with the apparatus which can perform.

  FIG. 1 is a block diagram of an imaging apparatus 100 that can acquire light field data to which an image processing apparatus according to an embodiment of the present invention is applied. In the figure, 101 is a taking lens, 102 is an MLA, and 103 is an image sensor. The light that has passed through the photographic lens 101 forms an object image in the vicinity of the focal position of the photographic lens 101. The MLA 102 includes a plurality of MLs 1020, and is arranged in the vicinity of the focal position of the photographic lens 101 so that light that has passed through different pupil regions of the photographic lens 101 is divided and emitted for each pupil region. It is. The image sensor 103 is a photoelectric conversion element represented by a CMOS image sensor or a CCD image sensor. Each ML 1020 is arranged so that a plurality of pixels correspond to each other, so that the image sensor 103 receives the light emitted by the ML 1020 divided for each pupil region of the photographing lens 101 while maintaining the division information. Convert to an image signal that can be processed. The image signal generated by the image sensor 103 is output to the A / D converter 104. The A / D converter 104 digitizes the image signal and outputs it to the image processor 105 as image data.

  The image processing unit 105 performs various image forming processing such as white balance and color conversion, compression processing for compressing the formed image, and image recognition processing for recognizing who the person's face is. The main image processing unit 105 performs a reconstruction process for generating a reconstructed image by a refocusing operation, which is a feature of the present embodiment, and an image composition process for generating a wide-angle image by synthesizing the reconstructed image. Generate a file recorded by.

  The image data processed by the image processing unit 105 and the generated reconstructed image are temporarily stored in the main memory 114 via the bus 112. The stored image data and the reconstructed image are recorded as files on an external recording medium (not shown) via the recording unit 115 or via an external device such as a personal computer (see FIG. (Not shown).

  The switch group 117 outputs various information related to shooting, such as setting of the shooting mode, to the CPU 111. A signal output from the release switch serves as an AE or AF operation start trigger or a shooting start trigger. In response to these start triggers, the CPU 111 controls each part of the imaging apparatus including the image sensor 103 and the display unit 106. The ROM 113 stores a program and data for the CPU 111, and the CPU 111 executes the program to control each unit of the image pickup apparatus and perform an image processing operation (image composition recording operation) according to the present embodiment. Realize.

  The display unit 106 displays a user interface screen of the photographing apparatus, uses it as an electronic viewfinder, or displays a wide-angle image after photographing. The display unit 106 includes a touch panel, and outputs touch operation information to the CPU 111.

  As described above, this embodiment is characterized in that the image processing unit 105 operates the image processing apparatus according to the present invention. In other words, the image processing unit 105 reconstructs an image (reconstructed image) set at an arbitrary focal position (refocus plane) from image data (photographed image) by performing arithmetic processing (reconstruction processing). Configured to synthesize. The calculation process in this case is performed using a technique called “Light Field Photography”.

  Next, the configuration of the taking lens 101, the MLA 102, and the image sensor 103 in the image pickup apparatus according to the present embodiment will be described.

  FIG. 2 is a diagram of the image sensor 103 and the MLA 102 as viewed from the optical axis Z direction in FIG. The MLA is arranged so that one ML 1020 corresponds to a plurality of unit pixels 201. A plurality of unit pixels 201 behind one ML are collectively defined as a pixel array 20. In this embodiment, the pixel array 20 includes a total of 25 unit pixels 201 in 5 rows and 5 columns.

  FIG. 3 is a diagram in which the light emitted from the photographing lens 101 passes through one ML 1020 and is received by the image sensor 103 as viewed from the direction perpendicular to the optical axis Z. Light emitted from each of the pupil regions a1 to a5 of the photographic lens 101 and passed through the ML 1020 forms an image on the corresponding unit pixels p1 to p5 at the rear, respectively.

  FIG. 4A is a view of the aperture of the taking lens 101 as seen from the optical axis Z direction. FIG. 4B is a diagram of one ML 1020 and the pixel array 20 disposed behind the ML 1020 as viewed from the optical axis Z direction. As shown in FIG. 4A, when the pupil region of the photographing lens 101 is divided into the same number of regions as the pixels behind one ML, one pixel is emitted from one pupil division region of the photographing lens 101. The formed light is imaged. However, here, it is assumed that the F-numbers of the photographing lens 101 and the ML 1020 substantially coincide.

  The correspondence between the pupil division areas a11 to a55 of the photographing lens 101 shown in FIG. 4A and the pixels p11 to p55 shown in FIG. 4B is point-symmetric when viewed from the optical axis Z direction. Therefore, the light emitted from the pupil division area a11 of the photographing lens 101 forms an image on the pixel p11 in the pixel array 20 behind the ML. Similarly, the light emitted from the pupil division area a11 and passing through another ML 1020 forms an image on the pixel p11 in the pixel array 20 behind the ML.

  Here, a method for calculating an image plane (refocus plane) corresponding to an arbitrary subject position in the screen will be described. As described with reference to FIG. 4, each pixel of the pixel array 20 receives light that has passed through different pupil regions with respect to the photographing lens 101. By combining a plurality of pixel signals from these pixels, a pair of signals divided into pupils in the horizontal direction is generated.

式（１）は、ある画素配列２０の各単位画素について、撮影レンズ１０１の射出瞳の左側領域（瞳領域ａ１〜ａ２）を通過した光を積分する式である。これを水平方向に並ぶ各画素配列２０に適用し、これらの出力信号群で構成した被写体像をＡ像とする。
また、式（２）は、ある画素配列２０の各画素について、撮影レンズ１０１の射出瞳の右側領域（瞳領域ａ４〜ａ５）を通過した光を積分する式である。これを水平方向に並ぶ各画素配列２０に適用し、これらの出力信号群で構成した被写体像をＢ像とする。

Expression (1) is an expression for integrating light that has passed through the left area (pupil areas a1 to a2) of the exit pupil of the photographing lens 101 for each unit pixel of a certain pixel array 20. This is applied to each pixel array 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as an A image.
Expression (2) is an expression for integrating the light passing through the right area (pupil areas a4 to a5) of the exit pupil of the photographing lens 101 with respect to each pixel of a certain pixel array 20. This is applied to each pixel array 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as a B image.

  A correlation operation is performed on the A image and the B image, and an image shift amount (pupil division phase difference) is detected. Furthermore, the focal position corresponding to an arbitrary subject position in the screen can be calculated by multiplying the image shift amount by the focal position of the photographing lens 101 and a conversion coefficient determined by the optical system.

  Next, processing for reconstructing an image on an arbitrarily set image plane (refocus plane) with respect to image data acquired by the configurations of the photographing lens 101, the MLA 102, and the image sensor 103 will be described.

  FIG. 5 shows, from a direction perpendicular to the optical axis Z, from which pupil division region of the photographing lens light that passes through a pixel on an arbitrarily set refocus plane is emitted and which ML is incident. FIG.

  In FIG. 5, the position of the pupil division area of the taking lens is coordinates (u, v), the position of the unit pixel on the refocus plane is coordinates (x, y), and the ML position on the MLA is coordinates (x ′, y). '), The distance from the taking lens to the MLA is F, and the distance from the taking lens to the refocus plane is αF. Here, α is a refocus coefficient for determining the position of the refocus plane and can be arbitrarily set by the user. In FIG. 5, only the directions of u, x, and x ′ are shown, and v, y, and y ′ are omitted.

As shown in FIG. 5, the light passing through the coordinates (u, v) and the coordinates (x, y) reaches the coordinates (x ′, y ′) on the MLA. The coordinates (x ′, y ′) can be expressed as in Expression (3).

When the output of the pixel receiving this light is L (x ′, y ′, u, v), the output E (x, y) obtained with the coordinates (x, y) on the refocus plane is L Since (x ′, y ′, u, v) is integrated with respect to the pupil region of the photographing lens, Equation (4) is obtained.

  In Expression (4), since the refocus coefficient α is determined by the user, if (x, y) and (u, v) are given, the ML position (x ′, y) where light enters from Expression (3) is given. ') I understand.

  Then, the pixel corresponding to the position (u, v) is known from the pixel array 20 corresponding to the ML. The output of this pixel is L (x ', y', u, v). E (x, y) can be calculated by performing this operation for all pupil division regions and integrating the obtained pixel outputs. If (u, v) is the representative coordinates of the pupil division area of the taking lens, the integral of equation (4) can be calculated by simple addition.

  Next, reconstruction processing and image composition recording processing in the image processing unit 105, which are the features of this embodiment, will be described with reference to a flowchart of image composition recording operation in FIG.

  In the figure, in step S601, each image data held in the main memory 114 is read out in the photographing order. In step S602, the read image data is divided into predetermined distance information acquisition areas (hereinafter referred to as D areas). Here, since the division into the D area is reflected in the focusing accuracy at the time of reconstruction, it is desirable to divide each pixel array 20 having the minimum resolution. However, the size of the divided area (D area) may be appropriately determined in accordance with restrictions on the calculation capability of the image processing unit 105, the frame rate, and the like, such as forming a single D area by combining a plurality of pixel arrays 20. That is, the number of D areas may be reduced as the frame rate is higher or the calculation capability of the image processing unit 105 is lower.

  In step S603, the above-described equations (1) and (2) are applied to the image data divided into the D areas, and the subject distance is calculated as the distance information representing each D area. In the distance acquisition here, when a plurality of pixel arrays 20 are combined to form one D area (for example, a 5 × 5 pixel array), the result of only the center pixel array 20 may be the subject distance. The average value of the 5 × 5 pixel array 20 may be used as the subject distance. Also, the subject distance for each D area obtained in this way is used as a distance information map.

  In step S604, a subject is extracted from the read image data, and a distance list associated with the corresponding subject distance in the distance information map is generated as subject information. Here, the subject extraction method uses an arbitrary algorithm such as edge detection. The generated distance list is updated for each image data.

  In step S605, it is determined whether reading of all the image data is completed. If completed, the process proceeds to S606, and if not, the process returns to S601. In step S606, in order to match the image data, the distance list is corrected by adding a subject distance that is insufficient in the above-described distance list. A correction example of the distance list in this embodiment will be described later.

  In step S607, the image data held in the main memory 114 is read in the order of shooting. In step S608, the above-described reconstruction process is performed using the read image data and the distance list, and a reconstructed image focused on each subject distance and a pan-focus reconstructed image are generated. At this time, along with the generation of these reconstructed images, link information that associates the image data read from the main memory 114 with the reconstructed image generated therefrom is generated. This information is referred to when image data is deleted in step S615. Here, the pan-focus reconstructed image is a reconstructed image generated by performing reconstruction processing so that each D area of the distance information map is focused. An example of reconstruction processing in this embodiment will be described later.

  In step S609, it is determined whether the wide-angle image to be combined with the reconstructed image generated in S608 is held in the main memory 114. If held, the process proceeds to S612. If not, the process proceeds to S610. Here, the wide-angle image is an image that exceeds a normal angle of view by connecting a plurality of reconstructed images.

  In step S610, the pan-focus reconstructed image generated in step S608 is stored in the main memory 114 as an initial pan-focus wide-angle image. In step S611, the reconstructed image focused at each subject distance generated in step S608 is stored in the main memory 114 as an initial image of a wide-angle image for each subject distance.

  In step S612, it is determined whether or not it is possible to combine the generated pan-focus reconstructed image and the pan-focus wide-angle image, and the process proceeds to step S613. Proceed to Here, the pan-focus wide-angle image is an image having an angle of view that exceeds a normal angle of view by connecting a plurality of pan-focus reconstructed images. In addition, the determination of positional deviation is based on a method of dividing a picture into small blocks of an arbitrary size and calculating a motion vector from corresponding points at which the luminance SAD (Sum of Absolute Difference) is minimized for each small block. This is performed by a well-known technique.

  In step S613, the pan-focus reconstructed image generated in S608 and the pan-focus wide-angle image held in the main memory 114 are combined, and the pan-focus wide-angle image is updated with the combined image. The updated pan-focus wide-angle image is stored in the main memory 114 in preparation for synthesis with the next pan-focus reconstructed image.

  In step S614, the reconstructed image generated in S608 and the wide-angle image held in the main memory 114 are subjected to image composition processing for each subject distance, and the synthesized image is updated as the wide-angle image for each subject distance. The updated wide-angle image is stored in the main memory 114 in preparation for the synthesis with the next reconstructed image. An example of these image composition processes in this embodiment will be described later. In step S615, the image data read in S607 is deleted from the main memory 114 based on the link information described above.

  In step S616, it is determined whether or not all image data has been read. If complete, the process proceeds to S617. If not, the process returns to S607. By repeating steps S607 to S616, it is possible to generate a wide-angle image focused on each subject distance.

  In step S617, a wide-angle image focused for each subject distance is recorded in the data portion, and a file with the head address of each wide-angle image added to the header information portion is recorded, and the recording operation is terminated. A configuration example of the file recorded in this embodiment will be described later.

  Next, the imaging situation and imaging data of the present embodiment will be described with reference to FIG.

  FIG. 7A is a diagram schematically showing a shooting situation when the subjects B1, B2, and B3 are shot three times in the order of the shooting positions S1, S2, and S3. Here, D1 is a subject distance from the imaging apparatus 100 to the subject B1. D2 is a subject distance from the imaging apparatus 100 to the subject B2. Further, D3 is a subject distance from the imaging apparatus 100 to the subject B3. FIG. 7B shows the angle of view of the image data L1 photographed at the photographing position S1. FIG. 7C shows the angle of view of the image data L2 photographed at the photographing position S2. FIG. 7D shows the angle of view of the image data L3 photographed at the photographing position S3.

  Next, the correction of the distance list in step S606 will be described with reference to FIG.

  FIG. 8A shows a pre-correction distance list that associates the detected subjects with the horizontal positions of the photographing positions L1 to L3 and the vertical axis of the subjects B1 to B3 with the calculated subject distance. In the distance list before correction, the subject distance is insufficient for the subject B3 of the image data L3. FIG. 8B shows a corrected distance list representing the subject distance as in FIG. In this embodiment, for example, the subject distance of the subject B3 in the imaging data L3 that is insufficient is corrected by the subject distance of the subject B3 in the imaging data L2, for example.

  Next, the reconstruction process in step S608 will be described with reference to FIG.

  FIG. 9 shows a reconstructed image in which each object distance is generated by performing reconstruction processing on the image data L2 based on the distance list. Here, FIG. 9A shows the image data L2 before reconstruction. FIG. 9B shows a reconstructed image R21 that is focused on the subject distance D1. FIG. 9C shows a reconstructed image R22 focused on the subject distance D2, and FIG. 9D shows a reconstructed image R23 focused on the subject distance D3. FIG. 9E shows a pan-focus reconstructed image R24.

  Further, although not shown in FIG. 9, the image data L2 and L3 are similarly reconstructed based on the distance list to generate a reconstructed image focused on each subject distance.

  Next, the image composition processing in steps S610 and S611 will be described with reference to FIG.

  FIG. 10 shows a wide-angle image generated by combining the reconstructed image generated from the imaging data L1 and the reconstructed image generated from the image data L2 for each subject distance. FIG. 10A shows a composite image generated by combining the reconstructed image R11 focused on the subject distance D1 and the reconstructed image R21 focused on the subject distance D1. FIG. 10B shows a composite image generated by combining the reconstructed image R12 focused on the subject distance D2 and the reconstructed image R22 focused on the subject distance D2. FIG. 10C shows a composite image generated by combining the reconstructed image R13 focused on the subject distance D3 and the reconstructed image R23 focused on the subject distance D3. FIG. 10D shows a composite image generated by combining the pan-focus reconstructed image R14 and the pan-focus reconstructed image R24.

  In FIG. 10, although not shown, the reconstructed image generated in the same manner for the image data L3 and the wide-angle image obtained by combining the image data L1 and L2 are combined to update the wide-angle image. To do.

  Next, file generation and recording in step S613 will be described with reference to FIG.

  FIG. 11 shows a configuration example of a file 1100 including four types of wide-angle images: a wide-angle image focused on each of the subject distance D1, the subject distance D2, and the subject distance D3, and a pan-focus wide-angle image. The file 1100 includes a header information part 1101 and a data part 1102. In the header information section 1101, the data size, initial reproduction image, creation time, and the like are written. Furthermore, address information (start addresses) 1112 to 1115 of four types of wide-angle images, that is, a wide-angle image focused on the subject distance D1, a subject distance D2, and a subject distance D3, and a pan-focus wide-angle image are also written. Here, the initial reproduction image may be a head address of a pan-focus wide-angle image as a default image at the time of reproduction, or may be an object focused on an arbitrary subject distance, and is not particularly limited.

  The data unit 1102 holds four types of wide-angle images 1121 to 1115, which are a wide-angle image and a pan-focus wide-angle image focused on the subject distance D1, the subject distance D2, and the subject distance D3. That is, an arbitrary wide-angle image can be acquired from the top addresses 1112 to 1115 of each wide-angle image written in the header information part 1101.

  Next, the operation of reproducing the file recorded in step S613 will be described with reference to the reproduction flowchart of FIG.

  In FIG. 12, in step S1201, a file held in the main memory 114 is read. In step S1202, a wide-angle image to be reproduced is set with reference to the initial reproduction image from the read file. In step S1203, the set wide-angle image is displayed. In step S1204, it is determined whether or not there is an input such as a touch operation on the switch group 117 or the display unit 106 by the user, and if not, the process returns to S1204.

  In step S1205, it is determined whether there is an input for selecting an area in the image. Here, an input for selecting an area in the screen is determined from touch operation information on the display unit 106 or the like. If there is an input for selecting an area in the image, the process proceeds to S1206. Further, when there is no input for selecting an area in the image, the reproduction operation is terminated to execute other processing.

  In step S1206, in order to determine the optimum wide-angle image for reproduction, the wide-angle image that maximizes the contrast of the selected region in the image is selected from the wide-angle images held in the read file. In step S1207, the head address of the selected wide-angle image is set as the initial reproduction image of the read file, and the process returns to S1203.

  With the above operation, a wide-angle image focused on each subject can be reproduced according to the user's selection.

  According to the present invention described above, it is possible to provide an image processing apparatus capable of generating a wide-angle image focused on a subject selected after shooting while reducing the amount of data for reconstruction. .

  Here, a configuration example of another imaging apparatus to which the above-described image processing apparatus of the present embodiment can be applied will be described.

  The image processing apparatus according to the above-described embodiment targets data acquired by the configuration of the photographing lens 101, the MLA 102, and the image sensor 103 illustrated in FIG. 3 as refocusable light field data. However, light field data acquired with the configuration shown in FIG. 13 may be used. In FIG. 13, the same components as those in FIG. 3 are denoted by the same reference numerals.

  The details of FIG. 13 are described in Non-Patent Document 2 and will be briefly described here. 3 is different from FIG. 3 in that the photographing lens 101 of the camera is adjusted so that the plane of the MLA 102 is in focus, but in FIG. 13, the image plane IP1 in front of the plane of the MLA 102 is focused. Has been adjusted.

  In this embodiment, the light field data acquired by the imaging apparatus having the configuration shown in FIG. 14 is used as the refocusable light field data. Data may be used. In the figure, the same components as those in FIG. 1 are denoted by the same reference numerals.

  In the case of FIG. 14, the light rays refracted by the optical systems 101a to 101c are received by the imaging elements 103a to 103c corresponding to the respective light beams. The plurality of images acquired by the image sensors 103a to 103c are parallax images obtained by observing the subject space from different viewpoints. By combining these plural images, it is possible to obtain a two-dimensional intensity distribution and angle information of light in the subject space, that is, a light field. A method for obtaining a light field by combining a plurality of images in a multi-lens configuration camera as shown in FIG. 14 is described in Patent Document 1 and the like, and is omitted here.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, it goes without saying that the object of the present invention can also be achieved when the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the functions of the above-described embodiment can also be realized when an OS (basic system or operating system) operating on the computer performs part or all of the actual processing based on the instruction of the program code read by the computer. Realized. Needless to say, this case is also included in the present invention.

  Furthermore, after the program code read from the storage medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the processing based on the instruction of the program code is also performed. Included in the invention. That is, it goes without saying that the present invention also includes the case where the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code to realize the functions of the above-described embodiment. Yes.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (17)

Acquisition means for acquiring image data of a plurality of different angles of view that can be refocused;
Generating means for generating a reconstructed image on an image plane based on subject information obtained from the plurality of image data from each image data of the plurality of image data;
An image processing apparatus comprising: a combining unit configured to combine the reconstructed image generated from the image data having different angles of view for each image plane.   The generating unit divides each image data into a plurality of divided areas, calculates a subject distance from the image data of each divided area, detects a subject from each image data, and detects the detected subject and the calculated subject distance. The image plane is set on the basis of a subject distance associated with the detected subject, and a distance acquisition unit that generates the subject information for each of the image data. The image processing apparatus described.   When there is image data in which the subject distance associated with the detected subject is insufficient in the generated subject information, the distance acquisition unit calculates the subject distance associated with the other image data. The image processing apparatus according to claim 2, wherein the generation unit generates a reconstructed image on an image plane set based on the corrected subject information.   When generating the reconstructed image on the image plane based on the subject information, the generating means generates link information that associates the generated reconstructed image with the image data used for generation, and the combining means The image processing apparatus according to claim 1, wherein after the reconstructed image is synthesized for each image data, the image data is deleted according to the link information.   File generating means for generating a file having a data part for storing a reconstructed image synthesized for each image plane and an information part for storing address information of the synthesized reconstructed image stored in the data part The image processing apparatus according to claim 1, further comprising:   Reproduction processing for acquiring the file reproduced from a recording medium and setting one of the reconstructed images synthesized for each image plane stored in the data portion of the acquired file as an initial reproduction image of the file The image processing apparatus according to claim 1, further comprising a unit. Imaging means for imaging a subject image formed by a photographic lens and generating reconfigurable image data;
The image processing apparatus according to any one of claims 1 to 7, which acquires and processes image data generated by imaging the subject image at different angles of view by the imaging unit;
Recording means for recording a reconstructed image synthesized for each image plane on a recording medium;
Reproducing means for reproducing a reconstructed image synthesized for each image plane recorded on the recording medium.   8. The imaging apparatus according to claim 7, wherein the recording unit records the file generated by the file generation unit on the recording medium, and the reproduction unit reproduces the file from the recording medium. .   9. The selection unit according to claim 8, further comprising a selection unit that selects the initial reproduction image set by the reproduction processing unit from a synthesized reconstructed image stored in a data portion of the reproduced file. Imaging device.   The imaging unit includes a pupil dividing unit of the photographing lens, and the imaging unit is arranged so that the pupil dividing unit is positioned in the vicinity of a focal position of the photographing lens. The imaging device according to any one of 7 to 9.   The imaging unit includes a pupil dividing unit of the photographing lens, and the imaging unit is arranged so that the pupil dividing unit is located behind a focal position of the photographing lens. The imaging device according to any one of 7 to 9.   8. The image pickup means includes a plurality of image pickup elements that pick up a subject image formed by a corresponding photographing lens and generate the image data, and generate image data having different angles of view. The imaging device according to any one of Items 1 to 9. An acquisition step of acquiring image data of a plurality of different angles of view that can be refocused;
Generating a reconstructed image on an image plane based on subject information obtained from the plurality of image data from each image data;
An image processing method comprising: a combining step of combining the reconstructed image generated from the image data having the different angles of view for each image plane.   A program for causing a computer to execute the image processing method according to claim 13.   A storage medium storing a program for causing a computer to execute the image processing method according to claim 13.   A program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 6.   A storage medium storing a program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 6.

Images